Roto Brush applied on frame by frame to remove background

Please double-click your video layer to open it in the Layer panel, as Roto Brush only works there. Select a base frame where your subject is clearly visible. Activate the Roto Brush Tool by pressing Alt/Option + W, then paint green strokes over your subject and use red strokes (by holding Alt/Option) to mark areas to exclude.

In the Layer panel’s timeline, adjust the span handles to limit how far the Roto Brush propagates, which improves performance. Press the Spacebar to let the tool automatically track and isolate the subject. For better edge detail, especially around hair, switch to the Refine Edge Tool and brush over soft edges. Once you're satisfied with the result, click Freeze to lock in the propagation and prevent recalculations. If the video is long, divide it into smaller clips (5–10 seconds) and repeat the process.

Make sure your comp and footage have matching frame rates to avoid errors, and close other apps to give After Effects more system resources. If certain frames don’t track well, you can manually correct the selection and let it re-propagate.

It sounds like your footage was not shot with Rotobrush in mind. If you want Rotobrush to successfully propagate you need to shoot with a higher than normal shutter speed to reduce or eliminate motion blur, be aware of background colors and edge detail, and make sure that the subject you are trying to isolate is in perfect focus. That's why there is a visual effects supervisor on set for feature films and high budget commercials. 

If you can share a screensot of your footage maybe we can make some suggestions that will improve your workflow. I often add some pretty extreme color correction to increase edge detail and fiddle with the color, then pre-compose before running Rotobrush. You can then use the layer as a track matte or turn off the color correction in the pre-comp after Roto is frozen.

Thanks for writing in, Vikas.

Are there any error messages, or does it not mask properly? What's the version of After Effects you're using? If possible, could you share a screenshot of the issue? Feel free to share it via DM if you do not want to share publicly.

We're here to help.