AI too clever perhaps

AI in the new version of PS is undoubtedly a masive step in photo editing but....

There is always a but. I am trying to use AI to modify a group of people taken in a photo shoot. The age old problem exists, one person has their eyes shut, in another shot a different person is looking down etc. Anyone who has taken a group shot knows the problem. Now removing a person on the front row of a group using AI works extreemly well. AI makes an "intelegent" guess at what the whole person behind might look like and generally does a pretty good job. It is then easy to replace the person on the front row whithout worrying too much about an exact fit to the original position.

Now for someone on the back row who is not "acceptable". Making a selection of that person and asking AI to remove that person and expecting a version of the background to be produced in their place results in AI generating a range of random images of different fictional people or for some odd reason somtimes generates a random advertising board. It seems that AI looks at the whole photo and assumes you must want a different group of people (some look like aliens so some improvement needed here Adobe!). Inverting the selection produces a similarly wierd result.

I have developed a number of workrounds which involve multiple AI generations, merging layers, more AI, etc but all are very time consuming when dealing with even a modest size group. 

Has anyone come up with a "neat" way of dealing with this age old problem  using AI?