Vector Brush Settings - Help

I am having some issues with the vector brush settings. Everytime I tried to erase to make the edges more smoother, the eraser takes off too much. It literally looks like someone took a bite out of the edge of my illustration. The edge is not smooth and looks jagged. I tried to reduce the vector brush size and also played around with the settings, but I still can't figure this out. Am I doing something wrong? Any advice on how to overcome this problem? Thank you.