for some reason, when I try to use the Refine Brush in mask mode, or the Content-aware feature, I often get glitchy results, with squared selections randomly appearing around hair and details and with content aware mask leaving an edge around the selected areas. Sometimes restarting my system helps but just for a few minutes/.
Sorry to hear about the trouble with the refine brush in masked mode & content replacement issue in Photoshop. We're here to help.
Could you please share the version of Photoshop (Help > About Photoshop) and the version of Windows on your computer? Would you mind sharing your workflow? Is it happening particularly with one image or any images?
Are you using this option on a smart object? Since when are you seeing this issue?
Can you post some screenshots with the Layers panel visible?