Photoshop brush lag in Windows Server 2022

I configured Photoshop CC in Windows Server 2022 with Nvidia A40 graphics card and serving the application to multiple users via Parallels RAS. But the issue is all the operations are working fine in Photoshop like Zooming In and Out but there is lag when using the brush tool.

The brush lag is something like this, the # defines the lag between the brush and drawing line.

The graphics card is getting detected in Photoshop and I am accessing Photoshop via RDP. Whenever I use the brush tool the GPU is spiking up, that means the GPU is getting utilized by Photoshop. But I am not sure why there is a lag when using brush tool. The Server which I am using is very powerful, the server has almost 50 Cores of CPU and 512 GB of RAM.

Would really appreciate if anyone can help me out on this. Thanks.