Which tool are you currently using in Select and Mask? Is that Quick Select? Refine Edge? Something else? My guess is you are simply using the wrong tool for what you are trying to accomplish.
Result is influenced by the tool and settings in the right-side panel. Please capture the entire Photoshop window so that someone can provide better guidance.