This paper presents an implicit Runge-Kutta (IRK) based convolution quadrature time-domain fast multipole boundary element method (CQ-FMBEM). Application of a convolution quadrature method (CQM) to a time-domain boundary element method (BEM), which is called CQ-BEM, can improve numerical stability of time-stepping procedure. In recent researches, the IRK based CQ-BEM showed better performance than the conventional linear multistep based one regarding accuracy. However, the IRK based CQ-BEM requires more computational time and memory. Therefore, in this paper, the fast multipole method (FMM) is applied to the IRK based CQ-BEM for 3-D scalar wave propagation problems. The formulation of the IRK based CQ-BEM and the application of the FMM are described. The accuracy and computational efficiency of the proposal method are compared with the linear multistep based CQ-FMBEM by solving some numerical examples.