Playback barely makes use of gpu render - lag issue

[Jokuc]

I'm sure you get a thousand of these a day but despite looking at suggestions to solve it nothing seems to help.

I am having issues with laggy playback in Premiere for windows. Currently I have two 1440p clips (from the same video file source) that have the crop effect applied to them and are being played side-by-side. No fancy effects, only the crop.

As soon as I start to play it I get massive lag spikes and it can barely play at all. The clips play fine one at a time however I think the issue might have to do with Premiere not utilizing my graphics card properly:

This is my cpu integrated graphics and gpu during playback of these clips. As you can see the 3080 is being used, but not very much, and the integrated graphics takes a huge load and immediately spikes to 100%. 

• It's not a ram issue, I have tons of ram and only 18% is being utilized during playback.
• I have GPU CUDA Acceleration enabled in project settings.
• GPU drivers are up to date.
• It's a brand new premiere install, no previous media cache.
• Everything runs on high speed SSDs.
• I get the exact same lag spikes even if I turn off "high quality playback" and set the playback quality to 1/8.
• I tried go into windows graphic settings, added premiere and changed graphic option to use rtx3080.

I did some testing and put 5 different clips - some 1440 and one 4K, in big pile and tried to play them at the same time.

Somehow this played just fine, even when I set playback quality to high and resolution to full! This is how cpu/gpu worked during playback:

So I thought maybe it's an issue with the video file, but then I tried the above config again with the 5 clips but started playback like 10% into the clip - and I got the exact same lag issues again with cpu integrated graphics spiking and gpu not being utilized properly:

This seems extremely inconsistent. What the heck is going on here?

[Jokuc]

I did some more testing and it's behaving very weird, I disabled the integrated graphics so premiere had to use the 3080, but the lag is still happening. I tried to play like 10 different 1440p clips at the same time and sometimes it plays it just fine at full resulution with gpu at like 40% load, but other times it starts lagging even at 1/8 quality and gpu spikes to 90% even though I didn't change anything. This is only happening in premiere.