Ultra Key also affects the color of the the background video

I'm trying to understand why Ultra Key also affects the color of the background video.

I have the background image in the video track below the video with the green screen. You can see in the attached image that the background video in the media library is much brighter than the image in the larger video view window. Why is this happening and how can I resolve it?

I set the Ultra Key for the primary video with a green screen on the second track, used the eyedropper to set the color to remove, and then used the effects controls to remove it completely. I don't understand why the video on the track below is affected when I've seen so many tutorials where this doesn't happen.