AI replaces items within masked areas also when generating new backgrounds, how do I stop this?

So I spend a long time masking all the fine details in my photos, know they are definately correct and then ask photoshop to generate a new background, most of the masked image is left untouched, but then it does replace certain areas that are masked too. I found I can put it back by going over it in paint, but this is then extremely hard work and time consuming when I have already done the masking. Does anyone know why it does this or how to stop it and only generate the image outside of the mask?