I use (very frequently so I know its not me!) the masking function in PS with my photography work. I have upgraded to the latest vesion of PS (24.01.1) and Im also using the latest OS update (Ventura 13.0.1).
Im finding that when I select a mask and use the Refine Edge tool to refine your masking selections is not working as it should. The edges are very jagged and pixelated and it is just doing a really messy job! I have found the old select and mask (select 'Select' in top menu followed by 'Select and Mask' (whilst holding shift down) - I think its the legacy Select and Mask tool? Well that works better than the 'correct' way of opening a mask and using the left hand menu tools. Drving me nuts and causing me much more work than it should. Please can someone address this issue and look into it. FYI Im using a M1 mac book pro too is that helps?