Premiere project being extremely sluggish

I'm editing an ad for a client but Premiere is being extremely slow and sluggish making editing extremely awkward and taking a lot longer than it needs to.

I'm working with a lot of 4k footage and expected this to be the issue but it has manifested in a weird way. If I click play, after a few seconds it will play with 0 issues, no lag, no stuttering nothing, I can watch the whole video. However if I mouse over an area on the timeline and click to skip there the whole of premiere freezes up for around 5-10 seconds. This doesn't happen with making edits or cutting, it ONLY happens with scrubbing, and it only happens in this specific project. I have tried clearing caches, I have tried restarting my PC, I have tried deleting each clip to see if a specific one is causing the issues, setting the playback quality to 1/4. I have even tried creating and enabling proxies, all to no success, the proxies didn't even make a slight difference (and yes they're applied). 

What is especially weird is that if I hold the mouse click down after the 5-10 second freeze I can scrub with 0 lag, it's only the initial click that is causing any issues, but still makes it really hard to jump between sections to make edits. 

All the things that cause this lag to occur I've noticed so far are:

The initial click when scrubbing (but not holding it down after)

Skipping forward or backward a frame.

Clicking play (but then flawless playback)

Quick spec rundown:
RTX 3070

i7-10700k 

32gb ram

2tb SSD