Answered
Some of the video is still showing up after it's moved away with position keyframes
The problem is basically in the title. Here is a recording of the problem:
Also note that the position keyframes are on the adjustment layer on top of the video clip while the speedramping is done on the video clip itself. Any help is appreciated.