Memory Leak in MFR during Render

[egustal]

Same issue occurs in ME and AE queue. I will get "out of memory" issues and it calls out some masks sometimes. However, I'm running on a system with 128 GB RAM, and simply restarting the render from the failed frames works flawlessly. So I end up babysitting a render and restart it from the failed frames every 8,000 frames or so. I think there is a fundamental memory leakage problem because renders become slower each subsequent time, requiring a reboot (simple restart of AE or ME doesn't fix it). 

Other than this flaw, good work on MFR, this project would be prohibitively slow to render SFR. 

Details:

- PNG sequence, 1920x1080 (source comps are 3840x2160, this is a preview render before I pay big bucks for a farm)

- nested comps

- AMD 3990x / Windows 10

- 128 GB RAM

- limited CPU to reserve 70% for other apps

- maxed out the RAM usage, reserving smallest amount for other apps

- total render time estimate is ~80 hours (but it fails every 5 hours or so)

- cache drive size of 2TB SSD

[egustal]

...or maybe the CPU/RAM throttle is pegging a min/max threshold not anticipated by so many CPUs midway through allocating memory. 

[Mylenium]

Long and short: We can't really know. Your info is at best incomplete and does not contain any details about actual effects being used or the specific processing steps. Start by enabling per frame logging in the render queue and if possible monitor your task manager/ resource monitor. Quite possible that after a while simply a driver crahses or you run out of disk space on the system level. that and of course you could simply be using an effect that doesn't work well with MFR or for that matter GPU acceleration. Either way, impossible to know without more detailed info.

Mylenium

[egustal]

Thanks. I've attached the per frame output. Cache is set to read-only and cache drive has 1.7TB free, OS drive has 200+ GB free and output drive has a 1TB free. 

But like I said, simply restarting AE / occasionally the machine lets the render continue just fine from the frames that supposedly take too much memory. I'm just wondering if it's a bug since all the effects are built-in (not plugins) and none show the MFR compatability warning icon. 

[egustal]

effects used
- turbulent displace

- directional blur / gaussian blur
- levels / HSL / curves / maybe some other color correction
- linear key

- echo

also several masks with feathering, and layers with time remapping

[Mylenium]

The log is inconclusive, but since all effects seem to run low on memory, there's indeed a deeper issue. I would suspect, though, that it rather has to do with the combination of effects you use than being per se a generic memory issue. There's a good chance that e.g. Echo is simply reserving so much memory with every new frame that eventually it will run out. that may also explain the inconsistent behavior in different scenarios. Aside from of course turning off MFR you may want to debug this by turning off effects selectively and doing test renders.

Mylenium

[egustal]

Thanks for taking a look.

Due to time constraints I've even upped the system to 192 GB RAM, however it failed again. MFR will just consume as much as it can. The render is well past the sections with echo and blur, looks like it failed again last night. I think if Adobe cannot address the bug immediately, I would suggest to them they allow more fine-tuned throttling of MFR, such as limiting total thread count. I guess their unit test projects probably don't have such a long render on a 32+ core machine.  I think at some point the memory management can't handle it when it decides to add another thread. 

After the next render fails, i'm going to try the latest Beta and update (looks like 2022 saw an update just now). 

[egustal]

Ok, so the leak still persists, even with effects turned off. My solution for anyone else having trouble, is to open the Secret settings. Hold SHIFT while clicking on preferences-> general and keep SHIFT down. Then set the cache to purge every nth frame. In my case, clearing the layer cache every 5000 or so frames works. I can see the memory drop down to idle and then work itself back up. This makes it so I don't have to babysit the render!

[Ali Jaber]

I would try to set the work area to something like 4000 frames, and add it to the render queue, then setting the work area to the next 4000 frames, and adding it to the queue, and so on... I wonder if it works for you this way

[egustal]

All fun and games until you have a top-level comp that's 300,000 frames. The Secret settings > purge nth frame is a lifesaver, and essentially accomplishes the same thing as rendering in small batches. It's a super-detailed 3 hour piece that stitches together many sub-comps.