After Effects can't render if hardware acceleration is enabled on Windows 11

[Marko22433262er9w]

I've been having trouble rendering the entire week at this point and the weirdest issues. I'm doing compositing with larger files ~4.5K and for the most part, it is all working and previewing until I hit render. 

At one point it took about 30 minutes to render out a png sequence of 63 frames of just the footage without any effects. I put the same footage in media encoder and the entire clip of ~170 frames rendered in about 5min...

When I hit render a few potential things happen:

1. It starts to render, but after a few frames, it hangs forever, and I have to kill AE process. 

2. It loops indefinitely on preparing render.

3. AE crash or blue screen

The bluescreen codes were always VIDEO_MEMORY_MANAGEMENT_INTERNAL and it was always dxgmms2.sys indicating a gpu/driver related issue. But what ultimately ended up working was disabling hardware accelerated GPU scheduling & reinstalling AE and all my plugins.

My system: 

Ryzen 7 2700x
32gb ram ddr4

RTX 2080 Ti

2x nvme & 1 sata SSD drives
Windows 11 latest update as of 08/03/2023

Latest AE version

All the things I tried so far:

- Reinstalled GPU drivers 2x with DDU (both latest Nvidia studio & game drivers)

- Disabled OC on my CPU

- Reseated my Ram because for some reason my PC would sometimes indicate a memory issue when booting after blue screen. Ran memtest to confirm it's not ram.

- Disabling CUDA rendering in AE

- Disabling hardware acceleration in AE

- Disabling/limiting multi core rendering

- Limiting ram usage

- Reinstalled AE & Plugins

- Disabled hardware acceleration in windows

- CPU stress test

- GPU stress test

The main plugins used were Red Giant VFX suite (primarily Supercomp) & BorisFX BCC, but like I said even without using those plugins AE would have problems rendering. 

If it's some issue with my system I honestly can't tell what. I can do a full cycles render in Blender and my only issue is going over the gpu vram... And these comp projects they are kinda heavy but if there are no issues with frames rendering in preview panel then I don't see why all of a sudden it can't do a full render.

[Marko22433262er9w]

I had a wild idea to plug in my old SSD and install windows 10 on it to see if it's windows related.

On windows 10 the hardware accelerated GPU scheduling is off by default and when I turned it on I had the same issues. This setting is ON by default on windows 11 so if someone has similar issues I suggest turning it off.