The AI works opposite to the prompt

Are you using Generative Fill?  Generally, if you say "remove X", it does not really understand that and is liklely to add "X".  If you want to remove people, use the remove function.

Starting image from Firefly:

people standing around table with cake and tea in evening 

In Generative Fill, select "remove" on the left instead of insert:

and then I selected the woman on the right:

and click "remove":

and she is gone: