Stabilizing drone footage while zooming in on clip

Hey what's up everybody, I have been working with premiere pro for the last year but only as a hobby so i am fairly new to editing etc. I like to capture photos and create videos with my mavic 3 classic. Typically anything I capture is a pretty stable shot but in a shot that I want to use I am zooming in on the subject and the drone moves around a bit. It is even more noticeable when I zoom all the way into where I want to be. So far I have tried to use the warp stabilizer effect and that helped a very small amount but my issue is when I am zooming in on the subject it's zooming from left to right and right as I am about to be at the depth I want to be the drone slightly pans left then so it doesn't look very smooth.  I am using key frames to zoom in and I wasn't sure if there was anything I could do to fix this issue? Thanks in advance