Can't generate reference frame (content aware fill)

[jampff]

Am I going crazy? Not an AE expert, it probably shows. But I'm trying to follow Adobe instructions to use Content Aware Fill. Only, my panel does not look like the one in the Adobe help docs. I'm supposed to click the Create Reference Frame button after choosing my parameters... but there isn't one. ??? What am I missing?

Isn't the button supposed to be under the "range" menu?

The Adobe help doc I'm trying to follow is: Use content-aware fill to remove unwanted objects from your video (adobe.com)

Not sure if it matters, but this other Adobe help page adds a step: using the Clone tool to fill in the mask... this is missing from the first article. How necessary is it?

[Mylenium]

Just showing us the panel is of no use. You have not provided any actual info about the project, the footage, your computer. Chances are there's either an issue with hardware acceleration or your footage is not suitable. You need to provide proper full screen screenshots and the pertinent info. That aside, the panel layouts change all the time and tutorials from two versions ago can quickly be outdated plus of course there could be issues with the panel layout itself. Either way, we can't resolve your issue based on the very limited details you have provided.

Mylenium

[dakuswolf]

Many many thanks, so simple but blocked for days on this 

[Rick Gerard]

You have a small mask, and we have no idea what the shot is like. CAF only works for specific types of shots. You have to have something available in the frame that can fill the hole. I have no idea why nothing is happening without seeing your shot and your settings.

[jampff]

AE version 22.5.0 (Build 53)

The footage is ProRes 422 LT, 1080x1920 (vertical), 30fps.

I didn't want to provide a full screen shot for maybe a silly reason. But since you feel you must know to solve the problem: The little pink thing I'm trying to mask out is the dog's peepee. I'm not a pro, just trying to help my wife make an IG reel for her small business. We shot a short cute promo with our dog, and he got excited and his anatomy was rather prominent. I just thought I could use CAF to remove it, that's all. We could reshoot, but this is already a reshoot because the first one didn't work for other reasons. Here's the full screenshot:

Hopefully this picture doesn't violate any forum policies... LOL kind of. I was trying to avoid the specifics, but you said they were necessary so here you go. 🙂

Again, my main issue is, why is the "Create Reference Frame" button missing from the Content Aware Fill panel? If the UI has changed, what is the current way to use the panel? Thanks!

[Mylenium]

That will likely never work for the simple fact that in the underlying AI database for CAF there's not enoug reference datas for "greenscreen with a furry pimple" scenarios. And even if there were, the selection is too small. That's just not how this works. The function needs context to determine what type of image it is dealing with, so the mask would at least need to cover a considerable area of the actual fur plus you may need extra preparation to mask out the greenscreen and floor. Since that's gonna end up being a lot of work eitehr way and there's no guarantee it will actualyl produce good results or work at all, you may wanna consider more traditional techniques liek tracking the region in question and patching it over with a feathered, masked duplicate layer or using clone stamps and all that.

Mylenium

[jampff]

Thank you! I'll try some of those techniques.

But what I really want to know is, even if CAF won't work for me in this scenario, why is the Create Reference Frame button missing from the panel? Is that normal? Does the button only show up if it detects adequate data?

[Rick Gerard]

If the Dog is moving, try using Mocha AE to track the offensive part. Track Translation only. This will give you both motion data and an animated mask (rotoscope) that you can use. A feathered edge + the tracking info + a frame or two brought into Photoshop and fixed with the fill tools there, and you're done. 

If you have never used Mocha AE, an in-depth course on the Boris FX website will give you some great ideas. 

If the Dog isn't moving much, you might try mask tracking, then use the Create Nulls from Paths script to attach nulls to the vertices. You can then use parenting or expressions to connect the repair layer to one or more of the nulls created by the script. 

You could also try mask tracking, then copy and paste the tracked mask to an adjustment layer, feather the adjustment layer's mask, and apply Hue and Saturation to remove the pink. 

[Rick Gerard · Accepted solution]

Create Reference Frame and Generate Fill is hidden below the bottom of the Panel frame. You need to click and drag down at the top of the Paint header to reveal the controls.

[jampff]

Yes!! Thank you, that's exactly what I needed. 🙂 I was seriously befuddled over the absence of that button. Thank you also for the other insights about the best use cases for CAF.

[jampff]

Thank you both, @Mylenium and @Rick Gerard for your helpful suggestions. I am sure at least one of them will help me figure out how to move forward with this project. 🙂

As I stated though, my real question isn't so much about this particular project. This is another reason why I didn't include details about this project in the beginning, because in my mind it wasn't really about "fixing this project". I just want to know, in general, about content aware fill, because although I don't know very much about AE I'm always eager to pick up another nugget about it that may serve me in the future. If I don't use it now I probably will need to someday. I just want to learn something.

So, setting aside my current project with the dog footage, what I really want to know (and I've already reiterated three times but it's been ignored???) is why the Create Reference Frame button is missing from my CAF panel? Is it a setting somewhere that needs to be turned on? In the future, in another project, if I want to use CAF how should I do it? Following the Adobe docs as referenced in my first post doesn't work because they just say, after choosing the settings, to click Create Reference Frame. But I can't because I don't have one.

So thank you both again, sincerely and kindly for your help. I mean it, I'm not trying to be sarcastic. I know your insights will help me with this project. But I also feel like raising my voice (just a little) and saying, "Will someone PLEASE answer the actual question that I asked, now for the fourth time... which was: where is the Create Reference Frame button? Thank you!" With a big smile, please and thanks. 🙂 🙂 🙂

[Rick Gerard]

Content-Aware Fill looks at the content of the adjacent pixels in the frames before and after the current frame. It is looking for similar luminance, color, and detail patterns. It then uses the pixels from the before and after frames to fill the hole. Different options for Fill use different luminance modeling algorithms to make better blends. If there are very few pixels in previous or following frames that move inside the hole, CAF has a terrible time figuring out what to do. 

You can create your own reference frames, which are basically clean plates, to help CAF with the calculations, but that requires opening Photoshop and creating frames to make the repair.

Imagine a locked-off shot of a nearly empty highway with the camera at a 45º angle to the road. A single car enters screen-eft and for the next 5-seconds crosses and exits screen-right. There are frames before and after the car moves through the frame where no car is on the road. CAF takes those frames and time shifts them to eliminate the car. It is like using a clone stamp, but the sample area is in the same place but at a different time.

The algorithm looks at different times and considers the mask's movement. It's pretty amazing that it works at all. CAF does not handle motion blur very well, perspective or parallax changes, or shots where there are never any replacement pixels in the hole. If the camera was in the middle of the road and the car was driving straight away from the camera, there would never be any suitable pixels in any frame to replace the car. You would have to create your own clean plate for CAF to work. Add in camera movement and parallax changes, and you end up with even more problems. No one tool or workflow will work with all shots. 

I think the dog's shot has problems because there is either not enough movement, too much movement, or not enough detail for CAF to generate a decent fill. Hiding the offending pink part with a rotoscoped adjustment layer is probably the simplest solution. 

I hope this helps.