Rotobrushed section going "out of sync"

Not quite sure how to explain this as I am very new to After Effects so hope this makes sense!

I used the rotobrush tool at the beginning of my project to remove the background of a video. It's been working really well until I removed a section of video from later in the project (not sure if this is the actual cause of the problem though) and now the rotobrushed section keeps jolting ahead of the video. Everything I've seen says this could be a problem with the frame rate but I haven't changed anything since it was working and the composition rate seems to be the same as the original.