Layer Styles - Channels Issue?
This is such a specific thing no amount of googling seems to have come up with an answer for me. Whenever I go to uncheck RGB channels through the Layer Styles menu, it seems to do the opposite of what I'm trying to do. For example, when I uncheck the Green channel, it turns it green, instead of purple/magenta, like it should.

You can see in the preview for it as well it shows green.

However, when I uncheck Green through the Channels tab, it does what it's supposed to.

Alternatively, I put this guy's head on a separate layer to see what usually happens, and uncheck Green through Layer Styles the preview shows Green, but that layer turns purple/magenta.

Here's one last final image based off of the Head being on a different layer. If I move the top layer, wherever it overlaps with the original image it's green.

It just never seems to be consistent. Is there some very simple explanation I'm not getting here? For some projects I'm trying to edit the layers RBG separately, not the whole image (which is what happens when I do it through the Channels tab). It does this with all the RGB colors through Layer Styles: Uncheck Blue and it turns Blue instead of Yellow. Uncheck Red and it turns Red instead of Cyan.
The images are always in RBG mode. I'm using Photoshop CS5.1 running on a Mac, High Sierra 10.13.5.
