Changing background without affecting skin tones?

I am shooting photos of a subject in front of a lit backdrop. The backdrop is black and I will be using different gels on the back light to change the color of the backdrop to mainly pink, purple, green, blue, or yellow.

I need to change the saturation and luminance of the background while making sure it does not affect the skin tones of my subject. I am still pretty new to this, though if I think there can be some pink and yellow in skin tones? Do I need to be concerned with the other colors? How do I change luminance and saturation of just the backdrop without affecting skin tones?

Thanks.