Rotobrush propagation fails in export

Community Beginner ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

Hi,

I have a shot with a model walking in front of a mirror, which reflection I have replaced and applied a mist or fog effect to. I then rotobrushed the foreground (or the model's back as it moves in front of the mirror) to finalize the effect. It all looks great in After Effects, but after exporting, via AME, the rotobrush edge is misaligned which of course ruins the effect:

rotobrush_export_fail.png
The layers in the composition are arranged as such:

rotobrush_export_fail_layers.png
background/foregound are instances of the same clip. The backgound layer has a motion tracker applied to track the corners of the mirror which has then been applied to the grunge2.jpg image to replace the mirror's reflection. Adjustment layer 2 has a gaussian blur effect applied, and is a luma matte for the grunge2 image to make the mist/fog effect. The foreground layer has the rotobrush effect applied.

The composition and the imported clip have the same frames per second. I have tried a number of different export settings, and the results are all the same.

Does anyone have an idea what could be causing this? I don't know if I somehow have to "lock the effects" in After Effects prior to exporting?

Thankful for any help.

/Nils_H

TOPICS
Error or problem

Views

49

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Community Expert , Sep 20, 2022 Sep 20, 2022

When I use Rotobrush I always follow this workflow. Always.

  1. Duplicate the footage layer that needs to have the background removed
  2. Name the Duplicate "Roto" to keep it straight, then trim the shot to just the frames that need to be worked on
  3. Add a garbage mask to remove anything that may interfere with Rotobrush
  4. Add color correction to improve the edge detail and contrast
  5. Pre-compose the Roto layer again, naming the new comp Roto CC, making sure to move all attributes to the new composition - bu
...

Likes

Translate

Translate
Community Expert ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

Show us the actual rotobrush window and comp window in full and tell us if you have actualyl frozen the propagation. Cropped screenshots are pretty much useless for these kinds of issues. That and of course we need to know all the details about the comp, render settings, your computer and so on.

 

Mylenium

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

Thank you for the quick reply. Yes, the propagation is frozen. I will try and post what you ask for, but I am fairly new to After Effects and completely new to rotobrush so bare with me. Also, I cannot show the model, hence the zoomed in and cropped screenshot showing the issue isolated.

/Nils_H

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

Here is a screenshot of the same frame in the layer view, showing the frozen propagation and that the rotobrush edge for that frame is following the model's outline, and it tracks very well for the duration of the part needing rotoscoping, inside After Effects, both in the layer view and in the composition view:
rotobrush_export_fail-layer.png

These are the settings I have applied to the rotobrush effect after freezing the propagation:
rotobrush_export_fail-roto.png

As far as the render setting, I used for example:
H.264 Match source - High bitrate, then did the following:
Checked: "Render at maximum depth"
VBR, 2-pass - target bitrate 40 Mbps for both passes

My computer is:
rotobrush_export_fail-system.pngrotobrush_export_fail-display.png
I hope that this is useful.
/Nils_H

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

When I use Rotobrush I always follow this workflow. Always.

  1. Duplicate the footage layer that needs to have the background removed
  2. Name the Duplicate "Roto" to keep it straight, then trim the shot to just the frames that need to be worked on
  3. Add a garbage mask to remove anything that may interfere with Rotobrush
  4. Add color correction to improve the edge detail and contrast
  5. Pre-compose the Roto layer again, naming the new comp Roto CC, making sure to move all attributes to the new composition - but don't open the new comp 
  6. Pre-compose the Pre-comp again and name the new Pre-comp Rotobrush Matte, move all attributes, and open the new Rotobrush Matte comp

    This gives you a comp with a color-corrected and garbage-matted layer that Rotobrush can work on with the least amount of problems.
  7. Run Rotobrush, refine the edge, freeze, and check the main to ensure the matte is good.
  8. If the Rotobrush Matte comp is good and only takes a few minutes to complete, I will move back to the main comp and complete the composite.

 

If I spend more than a few minutes on the Rotobrush Matte comp, I always go to the Composition Menu and Pre-Render the Rotobrush Matte comp. Always. Pre-Render will render a high quality with alpha copy of your Rotobrush Matte comp that you can use as a track matte in the composite. If you have not changed the preset, it will import the rendered footage and replace the Rotobrush Matte comp with the footage in your main composition. I almost use the rendered footage as a track matte, so I have more control over the composite.

 

If you don't want to use the layer as a track matte, open the Roto CC comp and turn off the color correction before you render.

 

After you have rendered the Rotobrush Matte comp, you can delete it to reduce the project file size and make the project more stable. Rotobrush and Warp Stabilizer are resource hogs that bloat your AE project and eat up system resources even after they have been rendered. 

 

I highly recommend that workflow. It will save you a lot of time, keep AE working better, and give you many more compositing options when you are working with the matte. Following that workflow will probably also solve your problem with the matte.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 20, 2022 Sep 20, 2022

Copy link to clipboard

Copied

Awesome, thank you Rick for taking the time to write such a detailed workflow. It might take a while for me to follow, but I will try following your recommendations and report back how it went - no matter what I will learn during the process and I like that.

Cheers, Nils_H

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 22, 2022 Sep 22, 2022

Copy link to clipboard

Copied

LATEST

Yes, that worked, thank you Rick!

I also watched some of your content on YouTube, if that indeed is you. Should you ever have the time, it would be great if you would make a video of your workflow above, and thus save it for posterity.

/Nils_H

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines