Using Track Mattes to Mask out Footage But Keeping the Color from the Bottom Layer

I have a layer that has shifted the channels to separate the subject from the background (shift channel, adjust levels, then shift channel again to take alpha to luminance) and I am trying to use said layer to Mask out the subject from the background (original footage inthe layer below it), however when setting the Track Matte to Luma or Alpha the clip will retain the color from the Track Matte footage! I would like to know how to allow the footage to retain its color from the footage of the orginal clip using a track matte. What am I doing wrong? What can I do? Please help me trouble shoot this problem and I hope with your help this can be solved!