Transform handles/corners shifting around when clicking on them

This is minor but has bugged me enough that I'm now searching for a fix/workaround, etc. and can't find one... Photoshop on Windows (current version). I have transform controls showing when I have a layer selected (the little white boxes around the border).

The issue is that when I click on one, say to drag and resize the layer, they all shift around slightly to some different layout so I'm no longer clicking and dragging but usually moving the whole layer around. I have to click on a corner once, wait for them to shift around, then click and drag the handle from the slightly different position where it moved to. How do I fix that?

(I can provide screenshots/vid if useful.)

Thanks!