When you write "rotoscope", I am assuming you used the RotoBrush Tool. AE has pre-defined render procedures (aka Render Order) running across Masks, Effects and Transforms. These are three areas where AE dissects every layer for its render processes, for each layer. Further, this Render Order is applied; on an individual layer basis and if one or more layers are precomposed/nested, this newly composition is also run through this Render Order.
For layers that do not have their Continuously Rasterized or Collapsed Transformations Switich enabled, the Render Order dictates that Masks are rendered first; then Effects and lastly, Layer Transforms. That AE has a Render Order is vastly differently from most if not all apps you may have used previously. However, a Render Order coupled with pre-composing allows for a non-destructive workflow and circumventing the Render Order.
In your specific case, you can circumvent the default Render Order (Mask->Effect->Transform) by applying your Mask(s) after you precompose/nest the 'rotoscoped' layer. With this workflow, you will be able to compute the rotoscope and then apply your mask, accordingly.
AdChoices