How to select every pixel that isn't grayscale/R=G=B?

An alternative method of achieving what you need is to make a saturation map and then make a selection from that map. A saturation map is black where R=G=B and gets lighter the more saturated the pixels (i.e. the more R does not equal G does not equal B)

1. To make the saturation map, add a selective color adjustment layer.

Go through each colour in the adjustment properties and for each colour turn the black slider value to -100%. For the three neutrals i.e blacks, neutrals and whites turn the black slider to +100%

Click the "absolute" button

You will now have the saturation map

2. You can add a curve above this map to increase the contrast

3. Click on any channel and make a selection from that channel

You now have all pixels where R does not equal G does not equal B selected. You can turn off visibility of the Selective colour adjustment layer and the curve and use your selection to copy the required area onto a separate layer

Dave

This should be workable in Lab, using the a and b channels while ignoring L.

Give Norman an example image and he'll show you the most efficient way 🙂

I suggest that you post the image.

Since you want to separate the color subject from the neutral background there may be masking methods that you have not considered. For example, you may have tried options in RGB mode only....or not experimented sufficiently with Calculations, for example. Post the image in a sufficient size for us to try a few things for you.