AE 24.0 Roto Brush Crash, Memory & Won't complete render overnight

[GotChaDigi]

I've tried to use the AE Beta versions, and the new AE 24.0 to use the Roto Brush to mask a high-res video interview(s), and as these inserted screenshot show the rendering runs forever and either never finishes or crash the system.  I really want to be able to replace a simple background with different imagery.  The dynamic link option from Premier Pro doesn't work either.

What would you suggest I do to resolve these problems?

Previous ASC Bug 09-Oct-2023 post [no responses]

System Info:
Processor Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz 3.00 GHz
Installed RAM 96.0 GB (95.7 GB usable)
System type 64-bit operating system, x64-based processor
Adapter Description NVIDIA RTX A4000
Driver Version 31.0.15.3758

[Kartika Rawat]

Hi GotChaDigi!

Sorry about the issue. Do you see any error codes or messages when it crashes? Could you check if crash reports are saved? https://adobe.ly/3PWqdhP

Do you have any third party plugins? 

Let us know. 

Thanks,

KR

[GotChaDigi]

Yes, I have a pdf file of the log files attached.  I've tried the beta 24.1 too.  Freezes on Freeze propagation.

No, I do not have any plugins.

[GotChaDigi]

Heard nothing back about this yet 😞

[jenkmeister]

That log file is for the home screen. If you have an "After Effects Log.txt" file, that would be helpful. But the one we really need is when AE crashes, a dialog should appear with a place to fill in information and your email address. If you see that dialog, please fill it in and submit the dialog. That will tell us where the crash is happening. 

[jenkmeister]

I'm also confused why this is an AME screen shot. Did you get the rotobrush work completed in AE first? 

[GotChaDigi]

The Roto Brush ran overnight and didn't complete.  It took the resources down to less than 10% and froze.  Had to reboot each try.  Tried to Freeze small portions.  The video in the background moved slowly, but the process pop-up never got past 2%.  Froze and had to reboot.  The attached PDF is from all the logs.  I liked the dump file that I provided another TS.

[jenkmeister]

Are you able to share the video you were trying to roto? 

[GotChaDigi]

Yes, I'll move it to a dropbox link and share the link.  Do you need the .aep file also?

Another related question:  I need to get these videos edited and replace the backgrounds.  What are some other alternatives methods you might recommend vs. Roto Brush.  I've used the Premier Pro Ultra Key + Crop option, but the background layer with masking bleeds through the subject, and the techiques to create an all white foreground mask don't work well.

[jenkmeister]

Anything you can share that helps us repro would be great. Thanks.

Let me get back to you on the alternative options. It'd be good to see the video fully to inform that.

[GotChaDigi]

Is there a way to securely share it with you instead of on the community board?

[jenkmeister]

Yes - you can DM me here on the forums with the link. Thanks.

[jenkmeister]

Thanks - I've got the files now. Yeah, that's going to be a tough scenario for any keying tool. Sort of white background when the subject is white with white in their shirt and the shadow hitting that white wall too. This would have been a good case for a green screen which would make a keying job much easier (but you've probably already come to that conclusion)

As to a roto: Trying to roto a 60fps 3+ minute 4K scene is going to use a lot of memory which is likely why your renders never finish. If you going to do that, I would recommend doing the edit first to get the clip down to exactly what you need, then doing the roto clip by clip on the edited footage. If you don't need 60fps, I'd also suggest you drop that to your final output FPS before the roto. I think you'd need maybe 128GB RAM to even take a shot at that sort of original clip being able to roto and freeze in some sort of reasonable time.

Given the subject, at least in the sample video doesn't move to much, you could also do a rough mask (see the image below) and then render that footage out. Bring that footage back in and then run the roto on that adjusted footage, that might speed things up but honestly you may need to hand mask this out. 

[GotChaDigi]

Thanks, I wish your phone support team would've mentioned that reducing the overall resolution would improve the AE Roto Brush propagation and rendering challenges.  I asked one of them that very question at one point.  UGH

I will reduce the res way down and try it and let you know.  Thanks

[GotChaDigi]

I re-rendered the original video MP4 file and saved it as a 1980x1080 29.97fps file.  Opened and created a new Comp in AE.  Re-did the Roto Brush.  Only progress the timeline to 40x frames and then tried to freeze it.  That started at 3:14 PM and it's now 4:00 PM PM, and it progressed only 1% (116 of 694) frames.  I won't complete; I'm sure of it, as it's doing the same thing as before with the bigger mp4 file.

[jenkmeister]

I just started the roto freeze on the 4K comp. I'll see how far I can get on my machine.

[jenkmeister]

13 minutes in and frames are still freezing ok for me on the 4K 60 fps version. 

What are you system specs? 

[jenkmeister]

Strangely - just after I took this screen image, the freezing dialog says things have stopped but I can see AE is still working. I'll dig into this some more tomorrow to see what's going on.

[GotChaDigi]

My system specs are in the folder I shared with you too.

[GotChaDigi]

Weirdly, the status popup never went past 1% as far as I could tell.  I walked away from the computer and came back and the freeze seems to have completed.  I added it to the ME render queue and it ran and exported (export is in the temp folder under the dropbox folder).  When I tried to preview it in AE the audio was way off; even after I queued it.  However, the export was fine.  It completed the whole thing.  Are are some masking overlay problems with the export.  Tried to go back to fix, but had to unfreeze and the whole file started to propagate all over again UGH.  Is there a way (cntl key or something) to stop the propagation?  

Just trying to scrub up the timeline to remove small sections of the mask.  Thanks.

[jenkmeister]

Yeah I've now seen the same thing. Unclear why the progress bar has stopped for me either. 

If you click into the areas the propagation has already completed it should stop. However, this may be a case where you want to split the clip into small, say 30 second segments, and then roto each one. 

[GotChaDigi]

Anything yet.  I was able to export to ME, but it ran for 7 hours and stopped at 17 secs or mins.  See screenshot:

[jenkmeister]

My roto freeze actually completed overnight. It resulted in a 300MB AEP though (since all the roto data ends up in the project). I can send you that or an exported version of that video if it helps. 

[GotChaDigi]

Sure, thanks.  Still takes too long and something is up with the propagation and freeze processes.

[jenkmeister]

I've got a 1080p at 29.976 export uploading now along with the aep file that contains the roto'd data. Hopefully that helps. If not, then again, I'd suggest breaking the clip up into smaller sections so that each one can be roto's independently so that it completes on your system. I'll send the link to you in a direct message here on the forums once everything has uploaded.