Major issue with Multi Frame rendering on larger projects

[1suky1]

There seems to be a big issue with multi frame rendering when working on larger projects and files.

My system specs:

Ryzen 7950x
64GB ddr5 ram

RTX 4090
3x NVME ssds (one dedicated to AE cache)
Windows 11 latest updates

AE version 23.3.0 (Build 53)

I have a 6K comp project that I need to render. If multi frame rendering is enabled, AE will eat basically 99% of 64gb ram even though I have it set to reserve 16gb for other applications. And I have it set to reserve 10% of CPU.

When I hit render I see the blue dashed bar spinning, preparing for render. I have task manager open and all I see is ram usage going to 99%, with AE using +50gb of ram (max should be ~45gb). The render hasn't even started yet and after a while the ram usage drops down and AE crashes without any error. Nothing else is running in the background and pretty much System is the next process that uses ~600mb of ram.

I have attached the last crash file (all are the same) and the weirdest thing is it says that it's related to Nvidia drivers. I don't know how it can be related to gpu drivers, it is a new system (2 weeks old) and I have latest studio drivers 531.61. I did a clean uninstall with DDU and Windows at that point installed an older version but the same issue still happened. 

And I can't share the project since I'm under NDA but it was mostly just cleanup with a few comped elements. I used 2 plugins, Lockdown & Red Giant's Spot Clone Tracker. But even projects that use none of these plugins were crashing. 

So my theory is that because theoretically with a 32 thread cpu I can render 20+ frames, AE will try to store all those 20 frames in memory even though they don't fit, without respecting the memory limits? Or it also tries to fill the gpu memory for all those frames and it doesn't fit either so that's what crashes it in the end? Very weird bug.

[jenkmeister]

MFR will start with a small amount of concurrent frames but is looking regularly as to whether more or less concurrent frames will result in faster rendering time so you shouldn't see 20+ frames rendering unless MFR really believes that will be doable and faster. From the crash stack it looks like there are 8 concurrent frames being rendered when the crash occurs. 

But you're on the right track with the NVIDIA drivers causing the issue. There have been a number of issues with the recent NVIDIA drivers causing AE and Premiere Pro apps to crash. For this case, the frames being rendered appear to have Lockdown on them, and Lockdown appears to be calling into the NVIDIA drivers. I would suspect then that something to do with Lockdown interacting with the GPU is causing the crash.

That said, I know you said that you've seen crashes without Lockdown, so if you're able to reproduce this and get a .crash, it'd be great to look at that further to see if there is something else going on. 

[1suky1]

I have uploaded yesterday's crashes. One is from the same projects but others are not and do not use Lockdown at all.
https://drive.google.com/file/d/1fU6yE3-ebw9tUD7dDke_3LQV2Nv6V-Lj/view?usp=share_link

I'll try to do a few more tests but now when I turned back MFR I can't get it to crash on any other project... But the one mentioned in this post still does crash. What I also don't understand is if it's nvidia drivers issue, why does AE eat up 99% of ram? 

And one thing I didn't mention as I didn't think it was related at first, but some of yesterday's crashes also happened at random when scrubbing through timeline or enabling/disabling layers. Today once I disabled MFR I haven't had a single crash. 

And since we're on topic, when rendering with MFR I also get these weird hangs and pauses in rendering. I haven't tested it properly but MFR rendering is just very inconsistent. Sometimes I would get a super fast render in 1-2 minutes and another time it's not even halfway through after 5 minutes.

[jenkmeister]

AE will attempt to use all the RAM it is allocated. A lot of that is just holding cached versions of layers and frames, but it will release or reuse it as is needed by AE as it is rendering. There isn't any relation to the RAM usage and the NVIDIA drivers. The GPU VRAM could be running out, and AE will watch for GPU VRAM usage but plugins can also use allocate GPU VRAM without informing After Effects and that can cause out of memory for the GPU. However the stack looks less like that and more like a bug in the driver itself, though perhaps all related. 

One thing to know about MFR - it's not in use unless you are actually previewing a set of frames. Scrubbing around the comp or enabling/disabling layers, anything that's to do with just 1 frame, won't actually engage MFR. So any crashes you saw doing that are likely unrelated to MFR. 

In terms of MFR rendering, if you have the disk cache enabled, you may find that when rendering or exporting, that some frames are being exported from the cache, and sometimes they need to be rendered. If you find yourself getting unexpected export times, you could do a test by purging RAM and Disk Cache and then export, and then repeat those steps to see if you get similar times. 

[1suky1]

That was a lot of helpful insight. I do know that AE will use everything I give it but it shouldn't use more than that, right? At some point I would have screens go blank and entire PC freezing up. And to be honest I get faster render times after I clean cache & memory. Even though I'm on nvme ssds any time AE has to pull from cache it seems to be way slower.

Also I did a bit more snooping and I found out that windows installed an update yesterday called:
2023-04 Update for Windows 11 Version 22H2 for x64-based Systems (KB4023057)
Is it possible that this update started causing the other crashes (and possibly other issues)? Because I really had a smooth ride right until yesterday and that is the only significant change that happened. 

[jenkmeister]

KB4023057 appears to be related to Windows Health Tools, apparently to make updates smoother. So I don't think it should have had any effect but I guess you could uninstall it and see? Details on that update: https://support.microsoft.com/en-us/topic/kb4023057-update-health-tools-windows-update-service-compo...

[jenkmeister]

And yes, our goal is that AE doesn't use more memory than you've assigned it. There is some slack in that limit to allow for frames to finish rendering, or if another Adobe app is open using some of the memory, or if a plugin is holding memory AE can't release. This is something we are continuing to work on though, in fact, the next Beta build drop has a few more improvements around this.

[1suky1]

That's good to hear. It doesn't show me an option to uninstall the update but things are working better so far.