Choppy playback/ laggy play pause

So I started to edit in 1080p 60fps last week and ever since I've had issues with playback in that res/fps. This happens before I even get to editing, just simply playing through the source clips. Very choppy playback, video totally freezes and only get audio. Even at 1x speed. My Pause/play is very delayed, easily 5 seconds before it responds. Playing the same footage outside of Premiere gives me no issues what so ever.

Specs:
GPU: Nvidia GTX 1660 (not overclocked)
CPU: Ryzen 5 1600 (not overclocked)
RAM: 16 GB DDR4 3000
Hard drives: 500GB Samsung 860 evo SSD (OS, Premeire, Footage)
2TB Seagate barracuda 7200 rpm (backup and games)
OS: Windows 10 64 bit

The size of the footage I'm using is about 3 hours long. Around 8 GB

Premiere is updated to 14.5

Now let me go through what Ive tried so far to no avail:
- Lowering res to 1/2 and 1/4
- Lowering paused res to 1/2 and 1/4
- Turning off High-Quality Playback
- Trying both Asio and MME
- Changing the renderer to GPU Acceleration
- Giving Premiere 14 GB of RAM
- Rendering out small areas once I do edit
- Rolling back Premiere to previous versions
- Clearing Cache
- Changing Cache to both SSD and HDD

Also, monitoring my task manager. During playback, my CPU stays around 20%, Memory about 50%, GPU about 10%, Disk never goes above 1%. My computer performance is fine outside of Premiere.

I tried proxies but seeing that it takes 4 hours to ingest this file... I was hoping to find another way.

Maybe I need a second SSD for my cache or more RAM. Just want to go through everything before I spend money on an upgrade that won't help.