mocha tracking going wrong when scrubbing

I’m trying to do a simple screen replacement.
The tracking works perfectly fine, but when I stop the tracking to adjust the planar surface, the previously tracked frames all go haywire.

In short, the planar surface changes shape when I scrub through the timeline after adjusting it in between.

I’ve been struggling with this for days. Please help. Attaching two screenshots.