I have straight edges when using some brushes

when you define a brush, do not define it as square, select using magic wand tool for example, to select the smoke, then select> similer, you can repeat this to have more details from the smoke, then apply feather to selection, the define the brush.

please not the selection in my example.

there is many ways to do.

Please post a meaningful screenshot including all pertinent Panels (Options, Brush Settings, Brushes, …). 

Wjat does the actual Brush Tip look like?