Without seeing the footage it's really hard to say what is going on. You rotobrush in one direction only, you pause when the matte wanders off and make corrections on the frame where it goes bad. Once you are done you transfer to the refine edge tool and start from the beginning moving in the same direction that you moved before. As soon as you are done, especially if the clip is longer than a couple of seconds, you freeze the roto and then check it. If there are any problems go back to where the matte fails, unfreeze, and start in the same direction you were going in (I usually go forward in time) and then you have to go clear to the end and freeze again.
Rotobrush is NOT intended for shots that are a couple of minutes or longer. You'll run out of system resources. It is a good tool for some masking jobs, but an actor standing in front of a white wall may be a better candidate for a procedural matte of some kind rather than rotobrush. I would have to see the actual footage before making a determination.
If you are new to Rotobrush then I strongly suggest that you go through some of the training materials you will find if you type Rotobrush in the search help field at the top right corner of AE. I've been using AE for a little more than 20 years and I had to go through a bunch of tutorials before I could figure out how to use the tool properly.
I hope this helps. Show is the shot and give us some more production details if you need more help.