How do I add a mask to a rotoscoped layer.

[JohnnyRocket]

Hello. I rotoscoped a layer and I'm pretty happy with the result. But I want to add a mask on top of that rotoscoped layed to more/less 'frame' only that part of the rotoscope.

When I add the mask to the rotoscopped layed, nothing changes. It still showed the entire rotoscopped section. If i disable the rotoscope and only show the mask, it behaves exactlty as I expect, only the section masked shows.

How to I add a mask to a layer that is rotoscopped.

I googled this and all I find is 'how to rotoscope'

[Roland Kahlenberg · Accepted solution]

When you write "rotoscope", I am assuming you used the RotoBrush Tool. AE has pre-defined render procedures (aka Render Order) running across Masks, Effects and Transforms. These are three areas where AE dissects every layer for its render processes, for each layer. Further, this Render Order is applied; on an individual layer basis and if one or more layers are precomposed/nested, this newly composition is also run through this Render Order.

For layers that do not have their Continuously Rasterized or Collapsed Transformations Switich enabled, the Render Order dictates that Masks are rendered first; then Effects and lastly, Layer Transforms. That AE has a Render Order is vastly differently from most if not all apps you may have used previously. However, a Render Order coupled with pre-composing allows for a non-destructive workflow and circumventing the Render Order.

In your specific case, you can circumvent the default Render Order (Mask->Effect->Transform) by applying your Mask(s) after you precompose/nest the 'rotoscoped' layer. With this workflow, you will be able to compute the rotoscope and then apply your mask, accordingly.

[JohnnyRocket]

Thanks, Yeah, this is what I managed to figure out on my own after running 20 different test. I thought I was doing something wrong because the mask would work on all the layers except the rotoscopped layer.