my color map isn't changing with the eye dropper tool

The brush color itself and the little forground preview square in the corner changes just fine, but the point on the big box with hue and shades doesn't move at all. It's driving me insane because I use that thing for color adjustment alot. I can't find literally anything about it.  How do I fix it?