Tracking Golf tracer shots zoom in

Hey everybody I am creating some golf tracer shots for a golf competition. When the camera is steady making the shot tracer is pretty easy. It's when the camera zooms in I'm having difficulty. Attached is an example video (no tracer added yet) and would like to know if it's even possible to create a tracer and have it track the tracer while the camera zooms in. Also, it's an organic zoom in it's not digital. It's the way they shot it. Working on a super fast PC and everything is updated.