Feature Focus: Roto Brush 2

Report · Jun 23, 2020

Hello Beta Testers!

We are pleased to inform you that Roto Brush 2 is now available for beta testing in the 17.1.2x32 build of After Effects (Beta). This new propagation engine incorporates the technology behind Project Fast Mask from MAX 2018 to streamline rotoscoping/isolating a subject or object from a background.

The amazing notiansans from Learn How to Edit Stuff has created a great tutorial video for Roto Brush 2 in After Effects (Beta) showing the basic workflow as well as a comparison with original Roto Brush.

What We Need From You

We are looking primarily for feedback on how the new propagation engine is performing with real-world footage, though all feedback is welcome. Your testing efforts and input are highly appreciated.

If you are willing and able to share footage with the After Effects team where Roto Brush 2 is having difficulty isolating a subject when propagating, please follow these steps to submit the footage and project:

In the Project panel, select the composition that contains the footage with Roto Brush applied.
Go to File > Dependencies > Collect Files
In the Collect Files dialog:
- Set "Collect Source Files" to "For Selected Comps"
- Enabled the "Reduce Project" checkbox
- Click "Collect" and select a location to collect the project file and footage
Zip up the folder containing the newly-collected project and footage.
Add the .zip to your Creative Cloud storage ( or Dropbox, OneDrive, etc ) and email a link with a brief description of what you were experiencing to rotobeta@adobe.com. All footage submitted will be kept confidential and will not be shared with third-parties.

Getting Started with Roto Brush 2

The application of Roto Brush with the new propagation engine is largely the same as in previous After Effects versions:

Select the Roto Brush Tool in the toolbar ( Alt/Option + W ).
Double-click on an existing layer in the Composition panel. This will open up the layer in the Layer panel.
Paint a green foreground stroke on the subject, object, or region you are attempting to roto. Avoid painting across edges as this might confuse the selection. This creates a base frame ( visible as a blue highlight in the Time Ruler of the Layer panel. )
Refine the selection by painting additional foreground strokes, or paint background strokes by holding Alt/Option to remove portions of the selection.
Foreground stroke, adding to selection:

Background stroke, removing from selection:

Once you are happy with the selection on the base frame, press the Spacebar to start propagating the selection through the footage. This will trigger the propagation of the matte from the base frame to the new frame. You can also use Cmd/Ctrl + Left/Right arrows to propagate frame-by-frame either forward or backward, if you would like to go slower and evaluate the propagation more closely. This might be more ideal for footage which requires more correction strokes.
After you have propagated the matte through all of the video frames you need and you are happy with the result, you can freeze the propagation using the Freeze button at the bottom of the Layer panel. After freezing, you can still make adjustments to the matte itself in the Roto Brush effect without re-propagating. If you need to add or remove from a selection after freezing, you can unfreeze the propagation with the same button.

Correcting the Roto Brush matte:
As Roto Brush propagates the matte from frame to frame, the selection may start to include unwanted regions or "lose" portions of the original selection. To correct this, paint a background stroke ( holding Alt/Option ) on the first frame where the unwanted change occurs to remove unwanted regions, or paint a new foreground stroke to include anything that was missed. As with the base frame, avoid painting across the edge of the selection as much as possible.

Adjusting and Refining the final matte:
The result of the propagation can be further refined and improved via the controls in the "Roto Brush Matte" group in the Roto Brush effect, especially with the "Reduce Chatter" property. Additionally, the Refine Edge Tool can be used for highly-detailed edges like hair and further controlled in the "Refine Edge Matte" property group. There are also options that compensate for motion blur and decontaminating edge colors.

Use the toggles at the bottom of the Layer panel to view the matte results, or open the Composition panel to view the result in-context with other layers.

Toggles in Layer panel for viewing alpha, alpha boundary, and alpha overlay

Additional Resources

For more details on the Roto Brush workflow, below you'll find links to both Adobe documentation as well as tutorials from Roto Brush users.

Please note: Due to the enhancements in the new version, some of the tutorials may be outdated or encourage practices which may not yield the best results. For example, painting very close to an edge may result in a poor result if any pixels from the background are accidentally included in a selection.

Thanks very much for trying out After Effects (Beta) and Roto Brush 2!

Report · Oct 25, 2020

OK, some new issues I've noticed during the last update or two.

I'm noticing repeated propigation and rendering when using RotoBrush 2.

In the past, I've been able to overlay 5+ videos with no issue at all. Now, each time I go back to check the isloated frames, it propigates all the way through no matter where I start the clip. Sometimes it will go through all the frames (up to 160-170), then will continue to propigate each video layer over and over again. Its caused a normally one hour at best project (that would have up to 10-12 video layers) to a three to four hour project between the time it takes to make sure all the layers look right and to render the video.

I can elaborate or attempt to provide video to further explain.

Report · Oct 25, 2020

OK, one more note- I changed all my layers from "Best" to "Standard" on RotoBrush2 and that seems to have made the problem go away. Some of my strokes on random frames got messed up big time but I was able to go through and fix them all. Once I got that fix, it renders in a matter of seconds.

So....issues with the "Best" settings, I guess...?

Report · Oct 26, 2020

Hi PitcherGiffer,

Thank you for reporting these issues. Was the continuous propagation happening in the latest Beta build or in the latest release version ( 17.5 )? Also, were you freezing the result of each clip once the propagation was complete?

When switching between multiple clips, I don't quite see the repropagation you've described but I do see that the propagation cache markings get messed up quite frequently when switching between clips. Moving the CTI within the span range usually brings them back however. A screen recording of what you're seeing or a project file would be best to help us isolate the issue that you're seeing.

Thanks again,

- John, AE QE

Report · Oct 27, 2020

Yes, it was happening in 17.5 (and I think a version or two back?). No, I was no freezing after propagation but I should have thought to do that.

So what would happen would be, I'd roto out clip one. I'd go back to the main comp where I had clip one overlayed on. When I'd go to review the comp w the overlay, it would go through and propagate before playing. If I had to go back and fix some frames, it would propagate the whole clip again. This occured each time I needed to review and/or make changes to the overlay clips.

When I would go to render, it would take a really long time to complete as well; much longer compaired to the earlier BETA versions before the recent updates.

I see a new update is out today. I'll work with it today and see if it keeps happening.

Thanks!

Report · Nov 22, 2020

Starting to experience the repeated propagation again. Every time I move a frame forward when using the Rotobrush 2 tool, it wants to propagate all frames each time. If I make a mistake, and I have to correct something it propagates everything. I try to switch to the main compostion to see how its looking, it propagates. When I go to switch back to the clip I'm using the rotobrush on, it propagates again. I'm not finised with my project so I don't think the freezing method will work (and I don't want to have to do it each time I have to change something.

Its really annoying and a normal 15-20 minute project is already taking me over an hour because I can't move forward or go back to correct without it propagating all frames EVERY SINGLE TIME LOL!
This is the latest update that was pushed out.

Report · Nov 22, 2020

I zoom in, it propagates. I zoom out, it propagates. I pull up another window, it propagates. I switch back to AE, it propagates.

You get the idea...

Report · Nov 23, 2020

Hey PitcherGiffer,

Sounds like you may have run into an issue we recently fixed with cache markings not drawing for propagated frames, possibly forcing a re-render. More likely, Ae might be running out of memory when attempting to render a propagated frame. In the latter case, try changing Ae Memory & Performance setting: check “Reduce cache size when system is low on memory”. That might take care of the issue. Regarding the cache markings, that fix was checked into a recent build of AE 17.6 beta. Please try updating to the most recent build if you don’t already have it installed. Please let us know if either of these solutions works for you. Thanks.

Report · Nov 24, 2020

I just installed the latest update and I don't have "Reduce cache size..." enabled.

I'll see if the issue persists with this new update. If it does, I'll try to get a video of what's going on and post it here.

Report · Nov 24, 2020

Nope. Still happening.
See attached video. I can send the project file(s) if you need to investigate further.

AE Beta version 17.6.0 Build 25

MacBook Pro 2019

2.3 GHz 8- Core Intel Core i9

16 GB DDR4

AME Radeon Pro 5500M 4 GB

Intel UHD Graphics 630 1536 MB

Report · Nov 24, 2020

Thanks for posting the video. I see exactly what you're describing. You have a few seconds of propagated frames, the CTI is parked downstream from the base frame on a propagated frame, you add a foreground/background stroke, and the propagated frames are re-propagating. Expected behavior is to wait until the CTI has moved from the current position before initiating propagation. It could still be a memory issue, but more likely you found a bug. I need to investigate further and see if I can reproduce in-house. Thanks.

Report · Nov 24, 2020

OK, let me know if you need more info.

I allocate about 13 of the 16 available GB of memory to AE (beta) when in use. I also try to make sure I discard any built up cache files from time to time if I'm working on a big file/project.

Report · Nov 24, 2020

To speed things along, please go ahead and send us the project and source footage:

In the Project panel, select the composition that contains the footage with Roto Brush applied.
Go to File > Dependencies > Collect Files
In the Collect Files dialog:
- Set "Collect Source Files" to "For Selected Comps"
- Enabled the "Reduce Project" checkbox
- Click "Collect" and select a location to collect the project file and footage
Zip up the folder containing the newly-collected project and footage.
Add the .zip to your Creative Cloud storage ( or Dropbox, OneDrive, etc ) and email a link with a brief description of what you were experiencing to rotobeta@adobe.com. All footage submitted will be kept confidential and will not be shared with third-parties.

Report · Nov 25, 2020

OK, so, after I closed out of the original project I referenced in this thread, it worked fine this AM.

However, I just went to start a new project and its doing the same thing. No rhyme or reason.

Report · Nov 30, 2020

Hey PitcherGiffer.

Based on your recent findings it appears that Ae is running out of memory. Resulting in purged frames.

In a previous thread, I inadvertently gave you the wrong instructions regarding the Memory settings in AE. You need to enable “Reduce cache size when system is low on memory” in the Ae Memory & Performance setting (i.e., click the check box).

Are you running Ae on Big Sur? If so, that could be compounding the problem. No solution yet for that, but we ‘re actively working with Apple to improve performance of RB2 on macOS 11.

Ae memory prefs setting

Report · Dec 01, 2020

OK, I'll enable that option and see what happens. And yes, I am running Big Sur.
I'll work on some videos today and reply back if I experience anything else I can add to this issue.

I did seem to notice it happening more once I updated, so I bet that's what the problem is.

I appriciate you looking into this!

Report · Dec 03, 2020

OK, still continues to propigate even with the "Reduce Cache size..." option checked. So, I'm going with the assessment that its Big Sur because I make sure I have as little memory usage as possible when I work on my projects.

Report · Dec 03, 2020

Also, it seems to be getting worse with version 17.6

Report · Dec 03, 2020

Something new (unrelated to my Roto2 propagation saga); I get this popup each time I open AE BETA. Started with the 17.6 update. When I click 'OK' it goes away and doesn't seem to have any impact on what I'm working on...?

Report · Dec 03, 2020

I'm keeping a record of your results when using RB2 and adding them to our internal database for potential feature enhancements and bug fixes. So, thanks for keeping us up to date.

Regarding the Ae warning on start up, that is a known issue and is harmless. Should be fixed soon.

Report · Jan 06, 2021

I'm having issues with this. The roto brush will only propagate half of the clip and then it just stops.

I realized the end of the clip needs some touching up but I can't even get to that part because it stops propagating around the halfway mark and all I see is a black screen. I try to go frame by frame but it doesn't even load anything besides the black screen after the halfway point.

How am I supposed to edit the mask if it won't even let me get to that point in the clip? It's only a 14 second clip as well, so it's really frustrating. I've tried for hours to see if it will continue propagating so I can access that part of the mask to no avail.

I have a decent computer but this is definitely eating up all my RAM, but what's the point of it doesn't even allow me to correct the mask?

If there are any fixes or there is anything I'm doing incorrectly, please let me know.

Report · Jan 12, 2021

Cross-post here with some possible solutions: https://community.adobe.com/t5/after-effects-beta/roto-brush-2-not-propagating-full-clip/td-p/117315...

Report · Jan 12, 2021

Thanks again!

I ended up cutting the clip in half then I re-rotobrushed the end that
needed touching up and it's working fine.

--

--

*Malcolm A. Maurice | Freelance Graphic & Web Designer, Writer,
Videographer & Editor* *|*

direct: 443.355.1291 | email: m.maurice331@gmail.com

Report · Jan 12, 2021

Thanks again!

I ended up cutting the clip in half then I re-rotobrushed the end that needed touching up and it's working fine.

Adobe Community

Feature Focus: Roto Brush 2

What We Need From You

Getting Started with Roto Brush 2

Additional Resources