I'm working in AE v18, using Rotobrush 2 to isolate some foreground elements in footage I'm compositing a person into. I've roto-ed these elements and I'd like to add an additional mask to these layers to hide a bit of the image too time consuming to roto with the brush (working with low-contrast footage), but are easy to hide with a tracked rectangle mask. But masks don't have any effect on layers with the rotobrush effect applied.
Are roto-brushed layers incompatible with masks (if so, why?), or have I stumbled upon a bug?
Error or problem, How to, User interface or workspaces
Not a bug or any of that. RB simply bypasses/ overrides the normal effects render order due to how it operates on the footage source itme. There's some other effects that behave similarly, even if it's annoying. Simply pre-compose the RB stuff and apply your masks in the parent comp.