Roto Brush issue

Hello,

I am using Roto Brush on After Effect but it does not recognise the subject I am trying to highlight. Everytime a skip to the following frame, the pink border does not stay in place and I have to select it all again. I have uploaded a video below as example. I tried with green screen and different color background but I have the same problem. Anyone have any suggestion? Thanks