Maintaining Crop Scale While "Zooming" into a clip

Being that it's Christmas I'm not expecting a flood of responses, but if anyone is out there I would appreciate the help! My issue is in regards to using the crop effect to letterbox my video. It works as it should on all of my clips except the only one where I've keyframed the scale to create a zoom effect. What happens is after a few frames the scaling moves into the image passed the letterboxing I'm left without the desired effect. By the end of the clip the letterboxing is nowhere to be found. I'm wondering if there is a way to lock the position of a crop so that I maintain the same letterbox scale even when using the zooming in effect created by the scaling. Thank you!