Skip to main content
N
Nils_H
Participant
September 20, 2022
Answered

Rotobrush propagation fails in export

  • September 20, 2022
  • 2 replies
  • 1491 views

Hi,

I have a shot with a model walking in front of a mirror, which reflection I have replaced and applied a mist or fog effect to. I then rotobrushed the foreground (or the model's back as it moves in front of the mirror) to finalize the effect. It all looks great in After Effects, but after exporting, via AME, the rotobrush edge is misaligned which of course ruins the effect:


The layers in the composition are arranged as such:


background/foregound are instances of the same clip. The backgound layer has a motion tracker applied to track the corners of the mirror which has then been applied to the grunge2.jpg image to replace the mirror's reflection. Adjustment layer 2 has a gaussian blur effect applied, and is a luma matte for the grunge2 image to make the mist/fog effect. The foreground layer has the rotobrush effect applied.

The composition and the imported clip have the same frames per second. I have tried a number of different export settings, and the results are all the same.

Does anyone have an idea what could be causing this? I don't know if I somehow have to "lock the effects" in After Effects prior to exporting?

Thankful for any help.

/Nils_H

This topic has been closed for replies.
Correct answer Rick Gerard

When I use Rotobrush I always follow this workflow. Always.

  1. Duplicate the footage layer that needs to have the background removed
  2. Name the Duplicate "Roto" to keep it straight, then trim the shot to just the frames that need to be worked on
  3. Add a garbage mask to remove anything that may interfere with Rotobrush
  4. Add color correction to improve the edge detail and contrast
  5. Pre-compose the Roto layer again, naming the new comp Roto CC, making sure to move all attributes to the new composition - but don't open the new comp 
  6. Pre-compose the Pre-comp again and name the new Pre-comp Rotobrush Matte, move all attributes, and open the new Rotobrush Matte comp

    This gives you a comp with a color-corrected and garbage-matted layer that Rotobrush can work on with the least amount of problems.
  7. Run Rotobrush, refine the edge, freeze, and check the main to ensure the matte is good.
  8. If the Rotobrush Matte comp is good and only takes a few minutes to complete, I will move back to the main comp and complete the composite.

 

If I spend more than a few minutes on the Rotobrush Matte comp, I always go to the Composition Menu and Pre-Render the Rotobrush Matte comp. Always. Pre-Render will render a high quality with alpha copy of your Rotobrush Matte comp that you can use as a track matte in the composite. If you have not changed the preset, it will import the rendered footage and replace the Rotobrush Matte comp with the footage in your main composition. I almost use the rendered footage as a track matte, so I have more control over the composite.

 

If you don't want to use the layer as a track matte, open the Roto CC comp and turn off the color correction before you render.

 

After you have rendered the Rotobrush Matte comp, you can delete it to reduce the project file size and make the project more stable. Rotobrush and Warp Stabilizer are resource hogs that bloat your AE project and eat up system resources even after they have been rendered. 

 

I highly recommend that workflow. It will save you a lot of time, keep AE working better, and give you many more compositing options when you are working with the matte. Following that workflow will probably also solve your problem with the matte.

2 replies

R
Community Expert
Rick GerardCommunity ExpertCorrect answer
Community Expert
September 20, 2022

When I use Rotobrush I always follow this workflow. Always.

  1. Duplicate the footage layer that needs to have the background removed
  2. Name the Duplicate "Roto" to keep it straight, then trim the shot to just the frames that need to be worked on
  3. Add a garbage mask to remove anything that may interfere with Rotobrush
  4. Add color correction to improve the edge detail and contrast
  5. Pre-compose the Roto layer again, naming the new comp Roto CC, making sure to move all attributes to the new composition - but don't open the new comp 
  6. Pre-compose the Pre-comp again and name the new Pre-comp Rotobrush Matte, move all attributes, and open the new Rotobrush Matte comp

    This gives you a comp with a color-corrected and garbage-matted layer that Rotobrush can work on with the least amount of problems.
  7. Run Rotobrush, refine the edge, freeze, and check the main to ensure the matte is good.
  8. If the Rotobrush Matte comp is good and only takes a few minutes to complete, I will move back to the main comp and complete the composite.

 

If I spend more than a few minutes on the Rotobrush Matte comp, I always go to the Composition Menu and Pre-Render the Rotobrush Matte comp. Always. Pre-Render will render a high quality with alpha copy of your Rotobrush Matte comp that you can use as a track matte in the composite. If you have not changed the preset, it will import the rendered footage and replace the Rotobrush Matte comp with the footage in your main composition. I almost use the rendered footage as a track matte, so I have more control over the composite.

 

If you don't want to use the layer as a track matte, open the Roto CC comp and turn off the color correction before you render.

 

After you have rendered the Rotobrush Matte comp, you can delete it to reduce the project file size and make the project more stable. Rotobrush and Warp Stabilizer are resource hogs that bloat your AE project and eat up system resources even after they have been rendered. 

 

I highly recommend that workflow. It will save you a lot of time, keep AE working better, and give you many more compositing options when you are working with the matte. Following that workflow will probably also solve your problem with the matte.

    N
    Nils_HAuthor
    Participant
    September 20, 2022

    Awesome, thank you Rick for taking the time to write such a detailed workflow. It might take a while for me to follow, but I will try following your recommendations and report back how it went - no matter what I will learn during the process and I like that.

    Cheers, Nils_H

    Mylenium
    Mylenium
    Legend
    September 20, 2022

    Show us the actual rotobrush window and comp window in full and tell us if you have actualyl frozen the propagation. Cropped screenshots are pretty much useless for these kinds of issues. That and of course we need to know all the details about the comp, render settings, your computer and so on.

     

    Mylenium

      N
      Nils_HAuthor
      Participant
      September 20, 2022

      Thank you for the quick reply. Yes, the propagation is frozen. I will try and post what you ask for, but I am fairly new to After Effects and completely new to rotobrush so bare with me. Also, I cannot show the model, hence the zoomed in and cropped screenshot showing the issue isolated.

      /Nils_H

      N
      Nils_HAuthor
      Participant
      September 20, 2022

      Here is a screenshot of the same frame in the layer view, showing the frozen propagation and that the rotobrush edge for that frame is following the model's outline, and it tracks very well for the duration of the part needing rotoscoping, inside After Effects, both in the layer view and in the composition view:


      These are the settings I have applied to the rotobrush effect after freezing the propagation:


      As far as the render setting, I used for example:
      H.264 Match source - High bitrate, then did the following:
      Checked: "Render at maximum depth"
      VBR, 2-pass - target bitrate 40 Mbps for both passes

      My computer is:

      I hope that this is useful.
      /Nils_H

      Terms & ConditionsAccessibility statement
      Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices