Low memory warning with 3D layers

I'm getting frustrated. I have no issues with any other 3D rendering applications such as blender or unreal engine, however the moment I enable 3D layers in after effects (using just 2D graphics and text i should add) I immediately get low memory warnings and AE slows to a sluggish pace. I can't get any work done like this.

I've tried searching for a fix and all I can find is people recommending to clear cache and make sure memory allocation is done correctly. I've done all this already, it doesn't change anything. The only workaround I can find is to not do any 3D work in after effects, and that's just not a very feasible workaround.

Specs:

GPU - RTX 3090

RAM - 32gb DDR4 3600

CPU - Ryzen 7 5800x