Track Camera - new track points added after deleting some

I'm trying to use the Track Camera motion tracker but every time I delete some tracking points from a foreground object NEW tracking points get added to the foreground object earlier in the timeline requiring me to go back and delete those. This happens again, and again.

How do I stop AE from adding new tracking points every time I delete some?