Need help with using Roto Brush

[quartertwelve]

Hey everybody,

Just returned to an old project and its giving me a whole new headache. I have a clip where I'm trying to mask out a plant by using the roto brush (original, using CC 2018). No plugins, everything in system. Whenever I try to mask out the plant, the brush struggles badly to find it, making me wonder if its a contrast issue. Do I need to adjust the propogation settings? Honestly, I just need some professional eyes on this thing.

Here is a snapshot of the original footage:

Here is a snapshot of my attempted roto brush:

Here, I've used the Pen Tool and traced a mask around the plant to help confine the pixel search. It hasn't really worked any better. At first, I tried the roto brush, and it would struggle finding the plant, creating a weak and incomplete mask.

Basically, I'm trying to mask out the plant so it overlaps that picture and looks somewhat realistic. Later in the footage, a person approaches and waters it, but I haven't gotten that far. I figured first I'd find a way to build the first from at best, then move forward and see what happens.

Any advice or recommendations would be greatly appreciated. Let me know if there is any other information or other visuals that I can provide for you. Thank you always for your help.

[Mylenium]

This is nothing Rotobrush can do. The stems of the leaves are way too thin and of course there simply isn't any real contrast here. You can even see that when you turn on the scopes. There's barely any contrast on any of the components due to everything being muddy greens and greys. I'm afraid this is really a job where you have to sit down and chew thorough it with conventional masks frame by frame or use dedicated tools like Silhouette.

Mylenium

[Rick Gerard]

If there is camera movement in the shot, you could pick a frame where the plant is the sharpest, save a frame as a Photoshop file, then paint a mask on just the parts of the plant that is going to be over the replacement wall art. It will be a pain but it's about the only solution I can think of. The only blessing is that you only need to create a mask over the part of the image that is going to be over the replacement artwork.

When you have a good matte created you can motion track the shot using Mocha AE and apply the motion to the Photoshop layer in the comp. If there is not too much camera movement or parallax shift you might be able to get away with it.

The only other option would be to just insert a pcture in the single square frame with the plain white background. You could apply Levels to a copy of the footage and a garbage mask, then use the copy as a luma matte for the replacement artwork. That would take you about five minutes plus any tracking you need to do for the camera movement.

I've uploaded a project file.

[quartertwelve]

Thank you for your guidance and feedback. Apologies for the late reply, I got caught up trying a couple of things and also replicating the garbage mask that John mentioned above.

At first, I tried using the mask from my initial attempt; I then added a Black and White effect and a Find Edges Effect to see if I could increase the contrast and help the Roto Brush focus on just the plant. There was some improvement, but not by much. So, I scrapped that attempt and wrote it off as a failure.

With the garbage mask; I originally made a clean plate of a screenshot in Photoshop where I erased the orginal blank picture from the footage. I did this because the photos I want to composite in have completely different dimensions, so this felt like the best option moving forward. Still, I tried using the Luma TrkMat method, but the picture frames would appear through with a weak opacity and a green coloration. I had the Levels effect applied, but even so, I couldn't achieve a decent opacity or color preservation.

At the moment, I'm trying to use a new mask, one that only occupies the part of the plant that actually overlays the picture frame. I've isolated the portion I need, but now I'm struggling to keep it steady in the footage. I'm trying to apply a motion tracker, but I might be doing something wrong.

The two clips I've attached show what I have so far; one is the footage from a wide shot, the second is a closer look so you can see how the mask is still moving around.

If you have any other recommendations or feedback, I'm greatly interested to hear your thoughts. Overall, my objective is to get that plant to safely overlap the picture frames and have the composites look as realistic as possible. I might be overthinking everything or applying some unthinkable processes; this is all new to my AE learning experience.

Just a reminder, I'm using AE 2018 and creating everything in-house. No plugins or third-party purchases.

Thank you again for all of your help. I'll be more responsive moving forward. Thank you for your patience.

[Rick Gerard]

I would use Mocha AE to track Translation and Rotation and Scale on the Pot the plant is in. When the Motion Tracking is complete and accurate in Mocha AE apply the tracking information to a Null, then parent the still image you are overlaying on the Photos to the null and mask if necessary. 

If you are new to Mocha AE you will find excellent training here: Mocha AE Escentials.

I have a tutorial on tracking in Mocha AE that I will probably release today. I'll post it here as soon as it is available. 

[quartertwelve]

I'm not too versed in Mocha AE, but I think I did what you suggested. Still no luck in getting the mask to stay still. The mask I created is from 14:01 on the timeline, so when I was in Mocha, I made sure to line it up accordingly and then track forwards and backwards from that point. I exported the track to a null layer shaped around the pot (the track was also on the pot in Mocha as well). The attached video is what resulted.

[Rick Gerard]

Your replacement image, the masked part of the plant, must be a still image. If it is video, go to Time/Freeze Frame. The best option would be to create a Photoshop file with some really good edges. 

You'll also need a track matte to hide the existing parts of the plant that need to be hidden in the original shot.  Both the track matte and the new still image of the plant need to be parented to the null.