convolution kernel light emboss alters image outside of the keyframes

I set the keyframes for convolution kernel light emboss effect, however its made every part of the image brighter, even outside of the key frames. Ive deleted and restarted and checked over it, but still cant fix it. Help appreciated.