Ae Rotobrush keeps propagating over and over again?

Hi, jumping on this as I have the same problem. Select object with rotobrush, spacebar to let it play, stop when it goes wrong and maybe got back a frame or 2. Sometimes it's fine, but othertimes every frame change starts it propogating over hundreds of frames and nothing I do seems to stop it. Also sometimes when letting it play it will suddenly jump back say a dozen frames and start propogating again.

Before someone gets brutally murdered over this, here's how I do it. JUST READ IT.

Firstly, I've been in your situation too. It is infuriating, and frankly, this should be addressed seriously by Adobe. Just dropping name of big companies who use AE doesn't mean nobody never got the same issues too.

YES, AE WILL GETS YOU IN A UNRECOVERABLE AND INFINITE LOOP OF PROPAGATION. SO HERE'S THE WORKAROUND I USE.

Firstly, "Automatic Propagation" is stupid. In AE2023, there should be a way to stop and constrain the propagation in time, other than Freezing the complete job. Once you're happy with a portion, you should be able to just lock it, so AE don't come after you and just mess with it. Just this, not having being made to work this way is puzzling.  I know you can reduce the span or have multiple ones, but that doesn't change anything.

So, here the way I do it on real life situation with complex rotoscoping jobs. 

If you use this, I guaranty you it will not be another waste day in front of AE.

AE has numerous limitations(let's call them that way).

You won't be able to roto brush a 5 minutes video with a subject jumping and moving around with black hair over a black pixelated background, from start to finish, UNLESS, you split and prep your video before. Unfortunately.

For instance, suppose you don't have a MP4 with no alpha channel (such a YouTube video), and you can't color key the background without loosing too much details that you want to keep.

Learn about #pre-multiplied vs #straight alpha. Maybe you can get away by just adding an alpha channel. (Just to let you know it is possible or someone suggest this, but I never got this lucky myself.)

Otherwise, in the above situation, prep your file, (clean the file prior rotoscoping).

Here, I will create a pretty loose mask surrounding the subject so mask-tracking gets done fairly quickly. Then, I will end up with a video with an incompleted alpha channel with a matte surrounding the subject that I want to get rid of.

Since AE messes up or starts to struggle beyond the 20 seconds mark when rotoscoping, I, then split my video in 20 seconds numerated segments that I will reassemble once all done. Premiere here is the way to go for this, trust me, even if AE can do it too. 

20 seconds it the magic number for me.  I never got any issue rotoscoping with AE since, no matter the complexity of the scene. And I just have an average gaming desktop with an unsupported AMD GPU.

More specifically, this rotoscoping method (apparently called the Brute Force approach) is the only one that's gets the job done for me.

This means, it's frame by frame, using the Page UP/DN key only.

IMPORTANT - When the Layer windows loses focus and your computer idles after a predetermined amount of time set in the Preview Preferences, that's when AE starts messing with you. It will start rendering non-rendered frames and that's can be time consuming (and nerve-wracking) on a long video. You may even decide to quit and contemplate starting a new life right there, hence the 20 sec. segment.

Now, once you're done determining the background/foreground and refining your edges on the subject and you're happy with your roto brush setting, use the Pg UP/Pg DN keys to advance. - You MUST watch carefully the thin green progress bar just above the propagation ones with chevrons (that is there only to add confusion). This is the obscur Rendering Frame Bar that tells you if a frame is rendered or not. As you move, frame gets automatically rendered when going slowly (a 1-3 sec. pace between each Pg DN, depending on you computer power). Go one by one slowly, watch the RF bar, and the Layer windows. Any correction applied will affect already rendered frames and you will have to go back. That's OK. As long you go slowly, things are fine.

Using this Brute Force approach, you want to render every frame as you move forward in time. Again, this means every time you move one frame, wait that AE has done rendering the frame. If you've moved 10 frames forward in time and AE has suddenly "UNRENDERED" few frames after you made slight adjustment with the brushes, STOP and move backward using the Pg DN key. Most likely AE need your help to fix few contiguous frames here (or just one). But that's nothing and usually very quick to fix and you won't have to fix them all. When you move, they often fix themselves of if the frame was rendered, it will stand that way. As long you stop and fix and render them, you'll be back where you were in no time. 

Also, AE will display a span of approximately 8 sec. of rendered frames in the RF bar, then they will will be cached (NOT DELETED) and disappear from the RF bar. YOU DON'T HAVE TO GO BACK, they are just cached, just keep going. 

Once you have done you're 20 sec. segment this way, Freeze your roto brush job, and lock your layer.  The Cached frames will be added.

Now you can render your whole 20 sec. segment video, import it in Premiere, etc.

Sometimes the Yellow Decontamination warning is in our way because they stupidly decided to put at the lower edge of the layer window, other times a 75% zoom would be useful, or the timecode system would be as simple and user friendly as in Premiere, but that's another story. I’m not going to go through all AE "limitations" here.

The attached photo, is my setting I use involving long black hair over a black background that gives me the most success.