Only select one of multiple selections?

I’m not sure how well this will work with the way your document is layered, but you might consider selecting each part individually using the Quick Selection or Object Selection tool (not the old Magic Wand tool) and converting each selection to a separate path, for two reasons. You would be able to quickly select a path or any combination of paths as needed, and instantly convert them to selections. And if you need to adjust an individual part outline, it might be faster to edit it as a path.

Perhaps try Quick Mask Mode (Q):

https://helpx.adobe.com/au/photoshop/using/create-temporary-quick-mask.html

You either create a better selection or deal with the one you have. More on selection methods here:

https://helpx.adobe.com/au/photoshop/using/making-quick-selections.html

@Revenant 

There is no direct way to do what you are asking. You can hold down Alt and use any selection tool to drag around the pieces you don't want.

The layer is often hidden below many other layers so I cannot use the wand tool as it will end up making a selection based on the layers that are visible,

Uncheck "Sample All Layers" in the options bar to limit the selection to the current layer.

Jane