How do you effectively speed up a completed video?

I've completed an animated video at 24 FPS but realised that it's rather slow. The quality is fine, the story just drags out a little so I need to quicken the pace. 

I could remove frames to shorten the duration but I don't want to alter the music I made, which is designed to hit certain beats in the video. 

I tried accelerating the pace using fast-forward features in After Effects and Premiere but some of the animations look a bit jittery as a result. 

Is there any way the video speed can be increased while maintaining its smoothness? 

Thanks in advance