Erasers

Could you please post screenshots with the pertinent Panels (Toolbar, Brushes, Layers, Options Bar, …) visible? 

If you want to use Brushes that were created for a specific Tool with another one cmd-alt-click the Brush in the Brushes Panel. 

If you want to use Brushes with any painterly Tool create your custom brushes with »Include Tool Settings« unchecked and, if you use Brushes created by others, take the issue up with their makers and/or save them anew with the corresponding setting. 

Edit: These icons indicate that the Brushes were created for a specific Tool. 

I think there is a Feature Request regarding the ability to disable the Tool-settings globally, and this may be a valid improvement, but as it is the »problem« is the doing of the people who created the brushes.