Adobe Illustrator - Brush Size Changing

A pet peeve I've always have with Illustrator is that when I use its brushes to draw, every so often it would always go back to 1 pt size. I'm able to fix this by just doing a random stroke and then setting the size to what I want again, but I'm getting tired of doing that since it just wastes time. Is there a way to keep the brush size to what you want and not have the program switch to 1 pt randomly?

I appreciate any help!