It sounds like you are trying to use Rotobrush. Rotoscoping is the process of manually drawing and animating masks. It is a painstaking process that requires manually adjusting mask position every time the action changes direction and then going back and cleaning up the errors by hand.
This is the best tutorial on the Process of Rotoscoping that you will find. Scott Squires is an effects master with a bunch of credits. The tool he is using in the demo is Commotion, but the workflow is exactly the same when you use masks in AE.
If you are trying to use Rotobrush to automate the process you should be aware that Rotobrush will not work on all footage. The footage needs to be shot with the workflow in mind. We need to see the shot to give you any kind of meaningful advice.