Basic Question about FPS when speeding up footage

120 fps playing at 24 fps is slow-motion that should be super smooth.  The "over-cranking" (the higher frame rate, or higher temporal information) allows the motion in the footage to play slowly without having to invoke Optical Flow for smooth slow-motion.  That said, Premiere Pro does a pretty impressive job with speed changes from normal speed to slow speed even if the footage was shot at the same frame rate as the Sequence (which is usually, but not always, one of the standard frame rates established by analog film and video formats - so 24, 25 or 29.97).

If you're increasing playback speed (fast-motion), you're just using temporal information that is already there.  Unless there's something wrong with how the file is decompressing, going from normal speed to faster speed should always look pretty good.

When you shoot at 120fps, then your shutter setting is dialed for 120 and motion blur looks natural when played back in slow motion. When you ramp it up to realtime, now all your motion blur is way less than it should be at realtime and it looks more choppy. You can fix this by shooting at slower shutter speeds or adding motion blur back in with Pixel Motion Blur in AE or RSMB or some tool like that. A lot of people just leave it though for action / sports scenes cause they like that choppy / super crisp feel. It's personal preference.

Did you modify the 120 fps footage to 24 fps, and then insert slo-mo footage into your timeline? Or did you use insert the 120 fps footage as is into your 24 fps timeline?

Normally speeding up footage isn't a problem since the frames are all there. It's just when your end game is slow motion where you want to make sure that your footage has a higher frame rate than your sequence settings.