Prevent layer selection outside of mask boundries

So we all know masks and how they work, but what bugs me and stands in my way on an everyday basis is that when I mask an image, for example, in a smaller rectangle with different proportions, part of the image usually stays hidden outside of the mask. Nothing wrong with that, it's how masks work. The problem is that the invisible part outside of the mask is selectable while drag-selecting layers with the Move Tool - something I do quite a lot.

Is there a way I can prevent this behavior, without locking the masked layer? Ideally, only the visible part of the masked layer should be selectable while making a selection with the Move Tool.