Alt click deletes brushes

If you are using a color picker while using a brush, you probably use Alt alllllll the time.
If you just once select a new brush while doing so, forgetting the release alt (or being unaware of this 'feature'), you'll just yeet your brush out of Photoshop. And for any illustrator, you're probably using your own brushes too, so you can't simply 'restore' via the menu. 

Is there a way to remove this feature? It's extremely destructive and I have wasted HOURS of time I can never get back just because of this stupid function that nobody ever deliberately uses.