Hello. I have a unique issue using the Content Aware feature. I'll admit I'm not an avid user of AE. But in this case, I have a personal project that needed AE's help in preping my video. Please visit the link to my video explaining my issue. And thank you in advance for any help you have to offer.
That's really not a scenario CAF is meant for. No wonder it's not working consistently. That's yet another case where conventional masking will work much better and given that there realyl isn't that much going on, it would be simple enough to patch in all the sky blue areas you need based on a reconstructed wall that has been built from scratch or has been put together using clone stamps in Photoshop.
If the camera does not move then just export a frame as Photoshop layers to use as the background, Set the video footage as a guide layer so it will not render, then add your animated characters. You can then set up simple track mattes for the picture frames and use them to add offset copies of your animated characters with probably an overlay blend mode and some transparency adjustments to create the reflections on the pictures.
I'm doing this exact thing right now. I have live video as a reference, some with a locked camera, some with a handheld camera, that I am using as a background for two animated characters. There is also reference footage of two actors dancing that I am using as a guide.
A combination of motion stabilizing for both shots to eliminate camera movement in both shots and guide layers gives me the ability to perfectly time my animated characters to the scene, then I can put the movement back in the clean plate (background shot with no actors) and add it to the animated characters plate - The pre-composed copies of my animated characters. I hope that makes sense to you.
Tell us a little more about your shot. If you have shots with camera movement I can point you to tutorials on how to motion stabilize the shots and I'll share the animation preset I use to work through the process. There are about a dozen ways to make this kind of work easier to do. The most efficient workflow is never going to be removing the actors from the video or filling in holes. You always want a clean plate, a shot with no actors in it, when you are going to replace an actor with an animated character. The only time you can leave the actor in the scene is when the replacement character is bigger than the actor. Everything else really requires a clean plate to be efficient.