Issues editing large/long h.264 files (multiple smaller h.264 files are fine however)

@Wesley S. In case you still looking for a solution, check this info, looks like it may help: 

https://www.reddit.com/r/premiere/comments/md2qwo/weird_trick_that_fixes_mp4h264_files_stuttering/

1- Storage bandwidth CAN be an issue, please provide us with your storage configuratgion, and where are you storing everything related to your project.

2- Resolution does not have a role on the "heaviness" of h.264 files. Bit rate does. Especially if its variable. Please share some more detailed info about the files.

3- Proxies are your best friend, no matter the codec, no matter the performance of your system.

4- Larger files tend to take longer the conform. Make sure you are giving enaugh time for the conforming to finish

If your sequence is set to show continues thumbnails for each frame that is to be expected. Your CPU has to work hard to redraw or refresh the individual thumbsized frames when using H.264 as opposed to Pro Res or R3D files.

So thanks everyone here who tried to help my idiot ass - I've figured it out.  Here's my solution for anyone having issues of their own:

I couldn't figure out why, when I had the video file in my selects timeline, it was fine, but as soon as I took it to the master sequence, everything stopped working.  I did a little bit of thinking (something I'm not too good at) and realized that I had applied a scale effect to the footage in my main sequence.  With it removed, everything played back fine.  In the end, I transcoded all of my footage so that it was the same size, negating the need for said effect.  The issue is not present with any other effects (lumetri color, RG Universe, time remapping, etc.)

TL;DR - motion effects (scale, position, etc.) were the issue.  Transcode footage to all be the same size.  No other effects cause the issue.

I would encorage adobe to figure out why this one effect is causing such a huge performance cost.

Could be some sort of a memory/buffer management bug. I'd try to switch priority from 'performance' to 'memory'. Or try to borrow additional 32GB RAM and test. Since you have 16c/32t CPU, more RAM is actually recommended: https://videoguys.com/blogs/news-and-sales/avoiding-ram-starvation-getting-optimum-performance-in-premiere-pro
Another option is to run the file in question through my 'vfr-to-cfr' tool from this package: https://1drv.ms/u/s!Andr88T403SpgRL5G8nDA5pXK_Vf?e=dwpceJ
Even if you pass CFR-file as input, it still checks for errors in fps metadata, so may help still. It works with a file copy, the source remains unchanged.