Ae Rotobrush keeps propagating over and over again?

Hi everyone, I've run into the same problem with the Ae Roto Brush a few times now and it's more or less making it impossible to use the tool at all;

Whenever I finish rotobrushing a video clip and go back to the composition window, when I change anything in the composition that somehow triggers the Roto Brush to completely restart the propagating process, which can be very time consuming. Those changes don't even need to be something related to the rotobrush, even stuff like adding a text layer or just looking at a different part of the compisition can trigger this to happen. 

Has anyone had this same problem?