Automatic rotoscoping based on movement?

[Lee James]

I'd like to have After Effects automatically rotoscope anything that moves in a scene.

Can this be done?

What I'm trying to do is this: I have footage of people performing against green screen, but the green screen has already been replaced with a static background image. I want to replace the static image with a new image, so theoretically if AE can just detect any pixels that change, then it should be able to automatically rotoscope for me.

[Dave_LaRonde]

You can try using a difference key, but I can't guarantee success.  It's an extremely picky keyer.  Personally, I'd do everything I could to get the original green screen footage.

[Rick Gerard]

If the difference key doesn't fix it, and it can't if you don't have a clean frame with the replaced background, then you are out of luck. That's the only tool that AE comes with that might possibly offer a solution. If you showed us a screenshot of the video, or better yet, if we could see the video we may be able to offer some other suggestions.

[Lee James]

Thank you both. I don't have the original green screen footage, and there's no point me showing an example as I wanted to use it for all manner of different purposes.

I tried the difference key, thanks. It does a basic job of what I wanted, and I'll use it, but it's obviously not the tool I wanted.

I want a tool that simply identifies movement in the video and rotoscopes it for you. I don't see why After Effects doesn't have such a tool. It would be incredibly useful. It would mean you could do "green screen" without needing a green screen, you just shoot against a static background and After Effects would cut you out automatically.

I think I'm going to have to request this feature.

[Dave_LaRonde]

"I want a tool that simply identifies movement in the video and rotoscopes it for you. I don't see why After Effects doesn't have such a tool."

Well, that still exists only in the realm of magic.

[LiftedPro]

Dave LaR - You have constantly been proven wrong on here, tis funny

[Rick Gerard]

For that kind of a tool to work nothing in the background could move at all. No lighting changes, no wind, no video noise or compression artifacts. The processing load would be horrendous. Look back and forward one frame, examine the luminance value of every pixel, look at the blocks of color information, determine which pixels have changed, then look back another frame and forward another frame and make sure that the actor that hasn't moved their head for 2 frames is actually moving so you don't remove his head, then look back and forward another frame so you can compensate for the actor that just stopped moving, then do it again until something on the actor moves and the software can predict that the pixels that make up the actors head should stay. It's not that you couldn't build an AI system that identified common shapes and looked for them, but you would need a supercomputer to get any kind of speed out of analyzing footage, especially if the camera moves. 

The automatic background replacement you can get from your webcam doesn't have to deal with camera movement and it knows what focus distance is. Wiggle around a lot and you get a mess. 

I'm not trying to discourage you from submitting a feature request, but the reality is that the tool would only work on certain kinds of footage much the same way that Rotobrush will only work on footage that was either purposely shot or accidentally shot with Rotobrush in mind. Nothing works on every shot.

[Lee James]

Maybe I haven't made my idea clear enough? No AI is needed. No supercomputers. And no magic. The calculations are quite similar to AE's existing motion tracking. (Even basic video compression is able to identify areas of minor/major movement in the picture.)

So let me try to explain how this might work…

I drag an effect onto my clip. The effect is called "Movement Rotoscope". It has a paramater called "Performers" which is set to 1 by default, telling the computer that there's only 1 performer in this scene, and it's only looking for one continuous area.

First, the computer tracks forwards through the footage looking for portions of the screen where there is major change over time. The computer starts to determine the 'outline' around the performer, and the more the performer moves, the clearer that 'outline' becomes. At first, his t-shirt may not be included as it's not moving much, but if he leans his body over towards the end of the video, the software will recognize that it's moving with him and part of the performer.

By the end of the footage, the computer should have a pretty good idea of where the 'outline' is around the performer. It then starts tracking backwards, following the performer based on the outline.

This may not be the best way for the software to work, but I do think it should be possible, given that we live in an age where a home computer can do such wondrous things as content-aware fill in a video, and deepfakes.

[Nathan Lovell_52]

My background is writing scripts and extensions for After Effects, as well as recently learning a lot about plugins. What you are describing would be best solved by what others here have suggested, or by some sort of hybrid approach using masks or the rotobrush. You will need human interaction in order to do what you want accurately. 

In terms of what you are actually asking for, it requires machine learning, vision, and most certainly a GPU. This does not mean it is impossible, it would definitely be possible to create some sort of edge recognising plugin that could that make the pixels outside of a radius transparent. But creating a plugin like this would be an extremely costly and difficult process, which in the end would only benefit a handful or people who have incredibly powerful systems capable of handling such a processing load. 

[xxxgreg]

I second the OP's original question - the AE project team appear to have dropped the ball on this feature. Photoshop is frequently improving automatic object isolation, AE has hardly advanced its auto roto in quite a while - by now I would have expected the AE team to incorporate Photoshop's object identification AI together coupled with scene movement to be able to automatically roto moving objects in a scene. I'm sure it's not just me - there will be TONS of footage from MANY PEOPLE that need background replacement. It's a completely missed opportunity.

Regarding suggestions of creating difference mattes - these only work well under optimal conditions - I was running difference mattes about 15 years ago when necessity required it, I would have thought AI would have moved the needle in that time. Diference mattes are also difficult to impossible to properly implement after the fact.

Here is a listing of the latest updates the May 2020 release for AE - basically Adobe is only doing maintenance releases of AE nowadays:

1. Tapered shape strokes
2. Concentric shape repeaters
3. ProRes RAW import support
4. Automatically update audio devices (macOS only)
5. Copy media to shared location
6. Enable collaboration on cloud documents

Come on Adobe, you're dropping the ball with AE - give it some much-needed attention and TLC.

[xxxgreg]

This - why hasn't this happened yet in AE?

[Lee James]

Update: Just for the record, the feature I was asking about here has been implemented in DaVinci Resolve 17 - it's called the "Magic mask" tool.

So those here saying such a feature couldn't be done were wrong, frankly.

[doesntdoot]

Yeah, I was just looking for this because I saw someone do this in AE in TikTok, at least I could have sworn they did this in AE.  I would HOPE AE has it, because even basic webcams are starting to have this feature