Premiere pro works ultra fast once I start working but over time it becomes slower and slower

My PC is powerful ok! Like rtx4090, 64gb ram and i913900, nvme's all around with more than 3gb/s speeds.

Once I boot up my PC and start working in premiere everything flows like butter! Even scrubbing the timeline with selection follows playhead which is intense.  (And I don't really know why, I get it, it activates the lumetri panel but still)

After I continue working for a couple of hours everything start becoming slow and laggy and it's really painful. The only solution that seems to work is to restart my computer and to a lesser extent restart the premiere to be able to work a another few hours. Feels like a trial!  

Pls do help me understand why this is happening so I at least find a solution that doesn't involve a restart of the PC or premiere.

Memory is getting filled when I scrub the timeline but when I minimize and maximize the premiere it kinds of empty it

Which makes things a bit better in those cases but the laggy performance being build over time is not fixed by that.

I can have like 25GB used out of 64GB RAM and like 8GB VRAM out of 24GB VRAM and it still is painfully slow, it's like it's not a memory thing.  

Clearing the cache from the preferences, doesn't seem to do anything for my running project either when everything is laggy

And it all leaves me head scratching, what makes premiere becomes painfully slow over time?

I have windows 11 with latest drivers and everything. But this behavior has been like that for at least 2 or 3 years now or maybe more