It would be useful to see the bottom of the toolbar panel while you're experiencing this issue. I've found that I run into issues with my brush tool when I'm working on a layer mask and switch into and out of viewing the mask as a rubylith (red overlay). It tends to switch the active foreground and background selection which causes me to either apply the brush or remove it inadvertently. A quick X command switches the foreground and background selection back to what I need. You didn't state if you were working with a layer mask - but if so, with your brush selected, check your foreground/background selection on your toolbar panel. Black will remove your brush strokes (or paint off) and white will apply them (or paint on).