Hue/Sat slider confusion

Hello everyone

I wanted to understand targetting adjustment using the hue/sat adjustment layer. As instructed by the Piximperfect guy, I'm using the ends to zero in on the areas I want to change while keeping the hue/sat sliders to the extreme right, but targeting a specific colour, spills out to affect other areas. Is it because that specific colour (say red) is also present in other areas? And what if I want to target the red in a particular area? How should I go about it then?

For example, in the photo below I want to target the gloves, but other red areas are also being targeted. I want to change the hue of the gloves, how should I go about it? Without affecting the other areas?