Don't auto-select layer outside its base layer of a clipping mask

I love the move tool's auto-select feature, but it's troubling me with clipping masks. With an image clipped in a clipping mask, the auto-select is selecting the image from pixels that are hidden outside of the base layer. Is there any way to change that behaviour? It's super counter-intuitive and tedious to have work around that.