This paper focuses on the constrained control of high-order linear-time invariant (LTI) swarm systems. In particular, the system is first locally pre-stabilized using a proportional-integral control law and the stability property of the controlled system is proved using swarm stability arguments. Then, we show that, under certain conditions, the pre-stabilized system systematically admits a candidate Lyapunov function and an associate upper-bound function which is related to the constraints. Then, we propose a set invariance-based distributed Reference Governor (RG) to modify the reference such that the trajectories of the agents do not violate the constraints at each time instant. Furthermore, we show that the performance of the proposed RG scheme can be improved using a phase-lead compensator. Numerical examples are provided to demonstrate the effectiveness of the proposed method.