How do I use the quick select brush without it expanding too much?

[wedouglas]

So I'm trying to essentially cut out a noise background from a fence. The quickest way to do this would be to use the quick select brush and just select everything through the fence bars. 

I tried setting the brush size to 1, but as i drag it around and select stuff, it often suddenly expands to a huge area that I didn't want selected. Is there some threshold setting or something I can select to stop this from happening? I literally just want to select what I'm brushing, I don't want it trying to guess to much what I'm trying to select.

Thanks!

[c.pfaffenbichler]

Could you please post screenshots with the pertinent Panels (Toolbar, Layers, Options Bar, …) visible?