Generative Fill - why does it insert a new person when I ask for a change?

This happens so much that I've kind of given up on the generative fill. Am I doing something wrong? 

I open a portrait, or head shot, and I select subject and then with generative fill I'll type things like "make him balding" or "give him a receding hairline" or things like that, and every single time Adobe inserts a picture of a completely different person. 

Am I missing something? It seems so easy. I've tried a million prompts, and every time I get a new person. I want the same person I selected in the headshot. What am I doing wrong? I said "receding hairline" , I didn't say take off his clothes and give me a new person....