Adjust Hue/Saturation to a Swatch Color?

I'm working on a fairly simple project where I take a photo of a building and change the color of the walls to give an idea of what it would look like if it was painted a specific color. I've been able to adjust colors with the Hue/Saturation Adjustment in the past, but what if I want to change it to be a specific color? I have a few specific real-life paint colors that I have saved as swatches, I noticed there is an eyedropper tool icon within the Hue/Saturation dialog window, but it has a No (circle backslash) symbol when I try to click it. 

Is there a way to adjust Hue/Satutaion to a color I have saved as a swatch?

If not, is there another way I can go about this? The walls in the photo have texture, otherwise I would just outline the wall and make a quick shape with the fill set to my swatch color.