Removal Tool takes 5+ seconds after each click

Hello everyone, and a Happy New Year 🙂

I am experiencing a serious issue in Photoshop (latest update but have had it for years) - whenever I work on a raw image from my Sony A7 IV and I start using the Removal Tool to clean up, every single click takes considerable amount of time .. I am literally standing and staring at my monitor for like 5+ seconds unable to do anything and imagine how on every image I use remove tool 100+ times.

What could be the possible reasons and solutions for this? I've turned over the Internet and can't find anything useful that I haven't already done..

FYI, I always edit my photos first in Lighroom (where performance is kinda okay, some lag here in there after the 10th Mask and remove click which I expect to be very normal) and then I right click - edit in PS.

And that's where it goes downhill. I've also tried closing Lightroom - no impact at all.

Is it simply a CPU problem and should I upgrade it to a 14900K or 7900X?

Is it the god-awful Windows and the only solution is to join the expensive Mac world?

Harware Spec:
CPU: Ryzen 5700X
RAM: 96 GB DDR4 3200MHz
GPU: RTX 3060TI

Photoshop Settings:
Scratch Disk - KC3000 SSD
Image processing:
- Select Subject and Remove Background: Device
- Selections Processing, Remove Tool Processing, Enhance detail: all on "Faster"

Removal Tool settings:
Mode: Generative AI off / on / Auto (doesn't make a difference)
Sample all layers and Remove after each stroke ticked on (ticking those off doesn't seem to make a difference either expect remove after each stroke but my work depends on that so I can't really use it)

Performance: 90% Ram usage (70GB)

I swear in God, nothing works and I am about to throw my PC from the balcony.