In analyzing mutual synchronization of coupled self-excited oscillators, a systematic dimension reduction method, called the phase reduction method, has played an important role in applied mathematics. In our previous work, we proposed a generalized phase reduction method, which is applicable to strongly driven oscillators [W. Kurebayashi, S. Shirasaka, and H. Nakao, Physical Review Letters 111, 214101 (2013)], by extending the conventional phase reduction method that is valid only for weakly driven oscillators. In this paper, we apply the generalized phase reduction method to strongly coupled self-excited oscillators. Using our method, we can theoretically analyze mutual synchronization by strong coupling. Furthermore, our method enables us to analyze hysteresis phenomena, in which the synchronization property of the oscillators depends on their initial states. By numerical simulations, we illustrate that our method works well even for the strong-coupling case in which the orbits of the oscillators are significantly deformed.