The trick to making Rotobrush work is to simplify as much as you can before you start by adding garbage mattes, then increasing the contrast and detail in the shot before you begin running Rotobrush. Here's my workflow.
Duplicate the footage layer
Rename the top duplicate Roto
Add a. shape layer above the Roto layer named Garbage Matte
Apply effects to the Roto layer to increase the contrast, color differences, and edge detail in the shot
Select the Garbage Matte and Roto layers, pre-compose, and name it Roto Matte
Open the Roto Matte layer (the Pre-comp) in the Layer Panel
Run, then freeze Rotobrush
If you spend more than about 15 minutes working on the Roto Matte layer or animating masks, consider pre-composing the layer again. Accept the name "Roto Matte Comp", move all attributes to the new comp, and then open the "Roto Matte Comp" and choose "Pre-render" from the Composition menu. That will open the Render Queue and allow you to render the timeline, replacing it in your main comp, thereby saving you from slowdowns or other problems caused by the huge amount of resources required to keep Rotobrush running when rendering your main comp.
I've been considering making a tutorial on this workflow for years. Prepping a shot for Roto, keying, or any matte work, especially one that was not shot and lit with separating elements in mind, is what professionals always do. Once you have a good matte generated, you can then produce a seamless and believable composite.
If you check your shot in the Color workspace and examine the scopes, you'll see that a lot of information is missing. Doing the best you can to improve that detail will help you create a decent matte. That is always the first step you should take before starting to work on the shot.
You'll now have a better chance of isolating elements in the shot and creating a matte.
... View more