Track Mask With Existing Keyframes

If you are trying to smooth things out the solution isn't more keyframes, it is less. I do a lot of manual roto and I always start out with a mask keyframe every time motion changes. If the object is moving down and to the right and then changes to up and to the left I set a keyframe just where the direction changes, then I look for speed and path changes trying to visualize straight lines between the vertices. I start setting keyframes somewhere in between the first two. The fewer keyframes you have the better off you are. 

If I use Track mask and things start to drift or I get jittering movement I will start just deleting keyframes, sometimes blocks of a bunch of them to smooth things out. Long story short, I think your workflow is basically backward. Start mask tracker, when it drifts stop mask tracker, delete all of the keyframes between the last good one and the current position, make adjustments to the mask, then continue mask tracking. 

One more hint. Use a bunch of masks, and if you need to, use a bunch of masks, one for each part, and maybe try stabilizing motion before you do the roto. Something like this: