Rotoscoping

How long is the shot? What is the frame size? What format is the video? What about your system's OS, memory, and disk space? All of these impact the ability of Rotobrush to do a good job. 

I seldom try and use Rotobrush on a shot that is longer than a few seconds. If the original clip is longer and is trimmed in the comp, I also always Pre-compose the shot, moving all attributes and trimming the new comp to the layer length so there are no extra frames at the head and tail of the nested comp (the pre-comp) that I am going to open in the Layer panel and Rotobrush. I also almost always do some color correction on the layer before I pre-compose to improve edge contrast, and I usually add some masking (roto by hand) to minimize the search area that Rotobrush is dealing with. 

I also make a standard practice to Transcode any MP4 files to a suitable production format like PreRez because MPEG (MP4) files can be incredibly difficult for your machine to decode correctly. H.264 is not a suitable production format. MP4s can also have variable frame rates, and that can foul things up.

If I have a long shot that cannot have a cut in it for the final production, I will break it up into sections that are no more than 10 seconds, pre-compose each trimmed layer, then run Rotobruish on each nested comp. This saves time and system resources. If Rotobrush takes me more than a few minutes, I also always Precompse the nested comp again after Roto, then choose Pre-render from the Composition menu. This will replace every nested comp with transparency in your project with a production-quality video master. You can then delete all of the "For Rotobrush" comps in your project, significantly reduce the AEP project file size, and improve the stability of AE. Rotobrush is a serious resource hog, and I don't want long Roto Comps in my comp after the work is done.

Maybe this will help. 

Tell us more and show us at least a frame of the footage, and we might be able to be of a little more help. Running Rotobrush on an MP4 shot that is longer than a few seconds on a machine that barely meets the Minimum System Requirements for AE is asking for trouble.