Skip to main content
Y
yogamayamatt
Participating Frequently
May 14, 2020
Answered

Learning the right rotoscoping workflow

  • May 14, 2020
  • 3 replies
  • 9774 views

Hi,

 

I'm new (about 3 days in) to After Effects and am trying to rotoscope this person from the backgrond; to exchange everything else in the scene (chair and all else).

 

I'm happy to spend the time and learn any tools within After Effects, but I want to make sure I'm learning the right workflow for the job, to not waste time unecessarily.

 

I naively thought I could RotoBrush and Fine Tune a 7m clip and render it out over night, and my laptop is just saying, 'don't even try'.

 

So, any guidance on how I might best approach this 7m30s job. I'm starting with an h.264 1920x1080 file from a Canon 5D and need to end up with a lossless (export of the same quality) of just the man, for compositing in Premiere.

 

Even a short bullet pointed list of the process/workflow would be great, so I know that at least I'm learning the right things for the job. I'm not in this for general After Effects mastery, but to get this result so I can continue on with my Premiere Pro edit/project.

 

Image pulled from the source video is attached. The highest quality rotoscope result is the goal; time is not the issue, as long as it's the quickest route to the right result.

 

Thanks so much for guidance,

Matt

 

NOTE: Title edited by Mod.

This topic has been closed for replies.
Correct answer Rick Gerard

The procedural matte + Roto is the way to go, and, trust me, you will be done in less time if you edit the footage in Premiere Pro and then use Dynamic Link to create comps that are just the shots you need to be worked on. Even if you end up using every frame, you won't get as tired, the likelihood of fouling up a couple of frames, and having to re-render the entire thing is lowered. 

 

My workflow for this kind of thing is always, cut in Premiere Pro, select the clip I am working on, right-click and select Replace with After Effects composition, Save the AEP, but keep AE open, Return to Premiere Pro and UNDO the last action so the original footage is back in place, Change the color of the clip in the timeline to identify it, then do the work in AE, render to a lossless format, and import into a bin in Premiere Pro I name VFX.  The last step is to just drop the clips right on top of the original clips in the Premiere Pro sequence. If the work I have to do in AE is extremely simple and renders very quickly I will keep the dynamic link, but for most of my work, probably more than 90% of it, I render in AE and replace the shot in Premiere Pro.

 

I've been on a VFX project for a feature film since February. A couple of weeks ago the Producer wanted me to make some changes in a shot. There are only about 50 frames that need fixing. Here is a screenshot of the Project panel. 

The shot (Scene 3-10) that needs fixing is 0:03:11 - just over 3 seconds. The shot that it comes from is 2:17:02 - just over 2 minutes seventeen seconds.  If I take a look at the whole shot the editor sent me this is how complicated the comp is and I have only expanded half of the nested comps that make up just Scene 3 in this 6 scene project. 

Doing the repair only required me to dive into this array of layers and effects:

All I had to do was open up 3_10 Repair Comp, go to the Track Matte 2 layer and adjust a half dozen hand rotoscoped mask keyframes on two masks, move back to Layer 4 - adjust the settings and add some keyframes to curves, then open up the Scene 3 00032.psd and do some more clone and healing brushwork, then return to Layer 4 and fine-tune a half dozen puppet pin keyframes and adjust the timing of opacity keyframes to fix the glitch in the roto. 

 

The repair took about 20 minutes to render. If the editor could use an image sequence instead of a Pro-rez file I could have just sent the 50 frames and the render would have taken about 4 minutes. If I had everything in one comp the render could have easily taken several hours and the likelihood of a render failure goes way up. 

 

That project is a moderately complex completely technical job that required a lot of hand roto, a lot of procedural mattes, and a bunch of keyframes, and yes, I could have everything in Scene 3 in one main comp, but it was far simpler to break each shot down into a separate comp. 

 

3 replies

R
Community Expert
Rick GerardCommunity ExpertCorrect answer
Community Expert
May 15, 2020

The procedural matte + Roto is the way to go, and, trust me, you will be done in less time if you edit the footage in Premiere Pro and then use Dynamic Link to create comps that are just the shots you need to be worked on. Even if you end up using every frame, you won't get as tired, the likelihood of fouling up a couple of frames, and having to re-render the entire thing is lowered. 

 

My workflow for this kind of thing is always, cut in Premiere Pro, select the clip I am working on, right-click and select Replace with After Effects composition, Save the AEP, but keep AE open, Return to Premiere Pro and UNDO the last action so the original footage is back in place, Change the color of the clip in the timeline to identify it, then do the work in AE, render to a lossless format, and import into a bin in Premiere Pro I name VFX.  The last step is to just drop the clips right on top of the original clips in the Premiere Pro sequence. If the work I have to do in AE is extremely simple and renders very quickly I will keep the dynamic link, but for most of my work, probably more than 90% of it, I render in AE and replace the shot in Premiere Pro.

 

I've been on a VFX project for a feature film since February. A couple of weeks ago the Producer wanted me to make some changes in a shot. There are only about 50 frames that need fixing. Here is a screenshot of the Project panel. 

The shot (Scene 3-10) that needs fixing is 0:03:11 - just over 3 seconds. The shot that it comes from is 2:17:02 - just over 2 minutes seventeen seconds.  If I take a look at the whole shot the editor sent me this is how complicated the comp is and I have only expanded half of the nested comps that make up just Scene 3 in this 6 scene project. 

Doing the repair only required me to dive into this array of layers and effects:

All I had to do was open up 3_10 Repair Comp, go to the Track Matte 2 layer and adjust a half dozen hand rotoscoped mask keyframes on two masks, move back to Layer 4 - adjust the settings and add some keyframes to curves, then open up the Scene 3 00032.psd and do some more clone and healing brushwork, then return to Layer 4 and fine-tune a half dozen puppet pin keyframes and adjust the timing of opacity keyframes to fix the glitch in the roto. 

 

The repair took about 20 minutes to render. If the editor could use an image sequence instead of a Pro-rez file I could have just sent the 50 frames and the render would have taken about 4 minutes. If I had everything in one comp the render could have easily taken several hours and the likelihood of a render failure goes way up. 

 

That project is a moderately complex completely technical job that required a lot of hand roto, a lot of procedural mattes, and a bunch of keyframes, and yes, I could have everything in Scene 3 in one main comp, but it was far simpler to break each shot down into a separate comp. 

 

    Y
    yogamayamattAuthor
    Participating Frequently
    May 19, 2020

    This is wonderful information. Really appreciate the detail. I think the workflow to take is the one you outline at the top of this last message; because, although I will be using the entire 7m 30s, it's going to be in pieces, and who knows what I won't need, once I'm in the edit.

     

    I guess I over do it on preparation; when the work will be less time consuming if I just let go a little and do somethings in parallel.

     

    I wanted to have the background free so I didn't even need to 'act like' it's not there. I guess editors are required to edit with temp cards and shots all the time; I can get used to it!

     

    Again, Thanks.

      R
      Community Expert
      Rick GerardCommunity Expert
      Community Expert
      May 14, 2020

      The efficient way to rotoscope is to figure out ways to save time and use as few keyframes and mask points as you can. Your shot is one that is perfect for a procedural matte to pull clean edges from the chair and the actors hair with a procedural matte, then combine that with a cuple of simple masks and a track matte created with a very simple mask path on a solid to fill in the holes left by the procedural matte. Take a look at this animated gif:

      The first thing I did was duplicate the footage layer and insert a light gray solid that I can use as a track matte between the duplicates. I then selected just the top layer and tried extract + Curves, Linear Color Key, and Color Range key to remove the black background and give me some good edges. Color Range produced good detail on the hair. Rotobrush probably would also pull good detail on the hair but using an effect to create an alpha channel from the black background was far fastrer. 

       

      The last step was to set the blend mode to screen so I could easily see through the layer and draw a very quick mask around the holes I needed to fill. If there is not very much movement in the shot then you're done. The only part of the project that requires precise masking are the edges of the black book. The rest is taken care of by the procedural matte on the top layer. 

       

      By far the best tutorial series I have ever seen on Rotoscoping by hand is given by my friend Scott Squires. He did not use After Effects, but the multiple paths and simplifiation techniques he shows are exactly the same techniques you would use in AE. Pay attention to the prepration, planning and execution of the shots he used in the project. Make sure you look at both parts. Effects Corner: Rotoscoping - Part 1

        Y
        yogamayamattAuthor
        Participating Frequently
        May 14, 2020

        Thank you Rick,

         

        A lot of insight and references to learn more. I'll definitely take the time to watch/read those links.

         

        I see there's many steps from begining to end to make sure the final output is the very best it could be, without losing anything important.

         

        Only thing that may be different than your gif above, is that I need to get rid of the chair as well, so I'll be pulling just the person - and giving him a much cooler chair for the part! Actually, I've already done a decent matte for all 7m30s to the top of the shirt and across the neck.

         

        I think experiementing with effects to pull him out is worth a good go.

         

        I'm going to go through your guidance carefully. Thanks.

         

        Matt

        R
        Community Expert
        Rick GerardCommunity Expert
        Community Expert
        May 14, 2020

        Here are some screenshots showing how I would use color range to create a matte for the head and hair, then on another layer Keylight to create a matte for the sleves and neck, then animate just a couple of masks on the body and the book to roto this scene. Two procedural mattes to find the difficult edges and one solid with just a couple of animated masks would do the trick. There are litterally hundreds of ways to create procedural mattes. Most shots that require roto can be simplified if you combine techniques. Planning and executing the shot is the most important part of creating a workflow that takes hours instead of weeks or months. About 15 years ago I worked on a project with about 40 other artists. My 90-second shot took me about 200 hours to roto because the roto work was an afterthought.

         

        The screenshots with notation:

        The Project File for reference.

        Mylenium
        Mylenium
        Legend
        May 14, 2020

        There's really no quickest route here. A blck vest and a black book/ ledger set against a black background and the nto boot all the ugly things that give toto artists nightmares - low light, transparencies and then some. Sorry, but the only sensible answer here is conventional mask-based rotoscoping. Everything else isn't going to work. Time to stock up on coffee for the next few days and just chew through it.

         

        Mylenium

        Y
        yogamayamattAuthor
        Participating Frequently
        May 14, 2020

        Thanks Mylenium for the response.

         

        My idea - and how I started, was to do the body - up to the neck - using the pen tool, and render that out. I think I've managed that. Then, because of the hair I figured to do the second piece - the head - with the rotobrush tool and rotobrush fine tune. Is that what you might mean by 'conventional mask-based'?

         

        When doing the rotobrush tool, I can't figure out how much to approach at one time, and how to split the clip 7m30s into sizes manageable for this second process.

         

        Could you suggest if this is the way to go for the head: rotobrush and fine tune?

         

        Is there a way to deal with manageable pieces and then put them all back together, to then render out a single (at least for the head) 7m30s master?

         

        Lastly, if I'm bringing in a Canon 5D h.264, what format do you recommend to render out to use as a master (I can do proxys for editing), but what would be the best way out - format - of After Effects in this situation (with a body render and a head render) where I won't lose quality going back into Premiere Pro?

         

        Thank you for confirming my first thoughts, and a little more detail for the above would be much appreciated.

         

        So, taking a day or days is normal for this - it's not just me?

         

        Cool,

        Matt

        Mylenium
        Mylenium
        Legend
        May 14, 2020

        You can't muddy up the waters by slpitting the layer if you want to use Rotobrush. That's the whole point - it analyzes the motion of the clip and propagates across time. Aside from the fact that once you're back in the layer view you still essdentially work on the whole clip and thus any performance gains my go to waste, you simply create the problem of the seams at the splitting points not matching. That's the crux of using any semi-automated process/ tool. That's why you have to make up your mind and ideally only use one controllable method, not a crazy mix. That doesn't preclude refining a rotobrushed clip with masks layer, but frankly that would be like doing the work twice.

         

        So for what it's worth, just sink your teeth into masking, laborious and tedious as it can be. At least it has the potential to look good and depending on how much motion there actually is, you may only need minor tweaks every few frames plus the process can easily be optimized further by copy&pasting mask shapes. This would in fact even work when the clip is split and ensure consistent behavior across the cuts.

         

        The formats don't really matter - you're already working off compressed footage, but of course you may want to use something like TIFF sequences for output to embed your Alpha channel/ transparency and not further degrade quality with additional compression.

         

        Mylenium

        Terms & ConditionsAccessibility statement
        Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices