Quick Zoom Effect Trouble With The Offset

I'm trying to make a quick zoom effect that you would see in a Sam Kolder video. Overall, my video works fine. Except when I use the Offset effect, when I keyframe and shift the center to the person that I want the zoom to focus on. Instead, it makes this jump across my person that I want to focus on and then goes back to the person like a boomerang. I tried to fix the Offset keyframes in the Graph Editor but even with the Easy Ease In and Out tools, I can't really edit the graph's curves.

I'm having trouble explaining this. If you have even the faintest idea what I'm talking about, then input would really help please.