Magic wand Tolerance

I am trying to use the magic wand to change a white section to a transperant background. I have the tolerance set to 0, however whenever I clear the section it takes the border around it even though it is a different color. It will turn a dark grey into a light grey. Is it possible to make the magic wand only take one specific color or is that somehow impossible?