How to stop Photoshop from accidentally selecting layers overlapping or near layer I am working on

It truly drives me nuts that often when I click on a layer (in the layers panel) to then do something directly to that layer on the canvas, it many times will start working on / activate a different overlapping or adjacent layer, most times an adjustment layer. But of course it could be any layer that happens to be in the same vicinity.

Is there any way to deactivate this? Photoshop didn't used to do this. You had to click on a layer (in the panel) to change it or work on it. My workaround is to temporarily lock or hide the layer I don't want to accidentally select, but it is silly that I have to do that.

Whoever thought that clicking on a layer (to change to another layer) directly on the canvas was a good idea, probably doesn't actually USE photoshop on a regular basis. I cannot be the only person who hates this (?) Help!