Extreme difficulty removing ojects with Generative Fill

I'm having a really hard time removing certain objects such as moustache, beard or clothes.
It usually happens when I'm trying to remove them from a certain part of the image (trim beard for example). The IA seems to go crazy and it just refuses to carry on with the prompt. It either just does minor changes to what's already there or starts adding totally bizarre content.

As I think I tried every possible prompt I could imagine I would like to know if there's something I haven't tried or a certain parameter to be tweaked.
I'm consuming enormous amounts of credit on fruitlessly regenerating.
I also think I mentioned on a previous post the lower quality of the generative fill compared to Firefly's newer algorythm.
Thanks a lot!