Tracking And Changing Background Sky in Footage

[Nickwhy90]

Hello community,

I'm currently working on a music video of mine where I'm co-editor and also decided to do some VFX where appropriate. I have shots in the open where the wheater was kind of dark and cloudy (as you'll see from the screen recording I'm uploading), so I thought I would add some more clouds and create some lightning here and there. I looked for a way to isolate the sky by  duplicating original footage, working with the tint/levels effects to make the sky very bright, use luma matte, and substitute it with other assets. 

I saw in tutorials that 'd have to camera-track the original shot, then create a null and a camera. Then, all I need to do is make the sky-replacement asset layer a 3D one, and it would move accordingly with the camera. However, in tutorials people use video assets for replacement, whilst I'm using dark cloud photos with a bit reduced opacity so they would kind of blend with the original video clouds. So when the camera is moving for example to the left, the original sky of the shot looks like it's going to the right, and the photo assets look like they're moving to the left (compying with the actual tracked camera movement), which creates a discrepancy between the two. So my question is - is there a way to make it look correct, using the static cloud photos I've already placed, or should I replace them with some sort of dark clouds footage?

My 2nd question is -When I turn on the 3D tracker camera layer, the whole precomposed comp I'm working on makes the main comp have a blank preview as you can see from one of the screenshots. Why would that happen?

Any help is appreciated!

Nik

[Rick Gerard]

I think the best approach to generating a procedural matte would be to add a significant amount of color correction to the shot to increase the contrast between the subject, the background, and the sky. A couple of copies of Brightness and Contrast might be a good place to start. You will want to improve the edge contrast of your singer and the motorcycle, and you'll just have to live with the depth of field problem with the trees in the background. When you get as much improvement as you can, you might be able to Pre-compose, moving all attributes, then run Rotobrush on the Pre-comp to generate the first matte layer. I would select the sky, then use the Refine Edge brush to bring back as much detail you can get in the out of focus background. When you have the firest matte created (the Rotobrush Comp), you will need to start stacking up addional copies of the original footage with different amounts of color correction to use as luma mattes for the background. The shot is going to requiere a bunch of hand masking to isolate the sections you are trying to turn into a luma matte. 

When you are done, you might have to create a subtle expanded and feathered mask at the horizon line so you can face in your additional sky effects. The whole thing would then need to be pre-composed and used as a track matte for a copy of the original footage and the storm elements you want to add. 

When you get the entire shot put together, it would be time for an adjustment layer with Lumetri applied to do the final color correction on the entire composite. At first glance I am guessing that you will probably need at least 3 copies of the original footage with different masks and color corrections to create the final Track Matte for the sky. 

I've dont a lot of this kind of work and I expect that it would take me about a half a day to complete this short effects shot. It could easily stretch into a day and involve Mocha Pro or Mocha AE for some of the rotoscoping work that is going to be required. 

[Nickwhy90]

Hi Rick,

Thank you for the detailed input. 

Does that mean what you're suggesting avoids using the cloud photos and working only with the original footage by isolating the sky and create copies with different color settings?

[Rick Gerard]

No, I mean the first thing to do is create a matte that will poke a hole in the sky. If you can post a full frame of the original footage, I will see if I can get you started.

Here's a short example (unfortunately with no workflow explanation) that I created for a class I taught a while ago: 

Here's a frame showing the original sky on the right and the replacement sky on the left. 

The original sky looks pretty blue, but I did not use any keying. A copy was color corrected to give me a solid white sky and a black foreground, A Solid Black layer with a mask covered all of the brighter details in the road. Then that layer and two copies of the original color corrected original shot were pre-composed and used as a procedural matte for the replacement sky. The Track Matte layer looked something like this:

Because the camera was moving, I also had to motion track the shot using Mocha AE and then add that motion to the replacement sky shot. By the time I was done, I had preserved the haze, most of the powerlines, and the moon and added a much more interesting sunset. The entire shot was then color-corrected using Lumetri on an adjustment layer at the top.

I keep trying to find the time to do a concise procedural matte tutorial. I'm asked to do a lot of this kind of work, and no one recipe works with all shots. I've been able to create the masks with nothing more than Curves and Levels, but on other shots, I have needed a stack of four or five layers with a half dozen color correction effects on each and some Rotoscoping to get a believable matte. 

Share a full Rez screenshot, and I'll show you how I would begin. You'll have to figure out the motion tracking yourself, but it should be pretty easy to do using Mocha AE because you have a fairly decent horizon plane you can track, so your replacement sky elements will follow the camera movement.

[Nickwhy90]

If you ever create tutorials on this subject, I'll be sure to watch them. I also noticed that people use different ways to achieve this result. There was one guy who used the Extractor to eliminate the original sky, but that didn't work in my case. The other method I saw and tried to use was with the usage of Levels/Tint/Exposure to create a contrast between the front objects and the sky and key out the sky, but I guess I couldn't quite do it.

Here are also 2 screen frames from the original sky:

[Rick Gerard]

I downloaded one of your low-resolution screenshots and this Adobe Stock image to create a quick procedural matte and color correction workflow.

If I had the footage, the first thing I would do would be to motion stabilize the footage using Mocha AE so that the motorcycle does not move in the frame. I did not show that in the sample comp that you can download.

After Adding the footage to a comp, the first thing I did was apply some color correction to increase the contrast and give me some edges to work with. Then a Black solid was added, and the hand Rotoscope was completed to black out the foreground and the reflections on the motorcycle. Then I did the same thing with a white layer to clean up the sky and poke a hole in the mirror. I would have used Mocha AE to do the Roto if I had a movie instead of a still to work with. Rotobrush would be useless on this shot. 

Now that I have a matte created, I drop the original footage and the replacement sky in the comp, pre-compose the layers sit took to make the Matte, and use the Pre-comp as a Luma Track Matte for the replacement sky. A bunch more fiddling with opacity, a master adjustment layer, some more masks, and I ended up with something like this.

To really sell the shot you'll need some more work on the Luma Matte, with some more layers. You will also need to have a separate roto of just the mirror and another copy of the sky that is flipped and motion tracked to put some realistic storm sky reflections in the mirror. 

If your shot is 7 or 8 seconds long, and you get fairly comfortable with the workflow i would guess that it may take up to a half a day to perfect the shot because of the cmaera movement and the required Roto work. 

If you download the project and fiddle with the controls it may give you a start on the techniques you will need to add some elements to the sky. The most time consuming part of the project will be the Rotoscoping, but Motion Stabilizing the Motorcycle will and Mocha AE will make that workflow a lot easier. 

Here's a link to a great set of Mocha AE tutorials. 

I hope this helps. 

[Nickwhy90]

Thank you! I'll try to implement what you've described. May I also ask if it's better to work on the original footage without any color correction and color grading and then color grade after I'm done with the sky and the lightning thunder, or should I use the color graded version and try to adjust the colors of the new sky and the lightning according to it?

[Rick Gerard]

It looks like you have some log footage. To create the mattes, you'll have to color correct to generate the greatest difference between color and luminance values. If you look at the sample project, you'll get the idea. 

When you make the composite, the replacement sky should also be color graded, and the source footage should have color grading that closely matches your final output. You can then render just the inserted sky by turning all of the other visible layers into guide layers, or you can render a 10-bit or better production master close to your video's final color grade.

When making any kind of movie that is longer than one or two shots, I always do my final edit, color grade, and sound mix in Premiere Pro using original footage, rendered production masters, and all of the other assets I have created. My final audio mix is almost always polished in Adobe Audition through Dynamic Link because there are more tools there. When I edit music videos, I always request the multi-track recordings from the artists as well as their production mix, drop them all in Premiere Pro, then do the final mix in Audition. That is really the only workflow you can use to produce industry-standard quality results. 

If the final edit is done in DaVinci or Final Cut, or even Avid, I still supply 10-bit or better (ProRez usually) production masters for the effects and composite shots, all of the audio resources I can, and some color grading guidelines if I cannot be present for the final mix and color grading. 

I hope this helps. 

[Nickwhy90]

We shot the video with a Black Magic camera. I have access to all the material and audio, since I'm actually the song performer. We already did the editing and color grading, but I guess after I've done what I can for the VFX, we'll have to color grade again in Premiere and do it differently, because the screenshot I shared above are totally raw, without any color correction (at least to my knowledge, don't know if they've done something on the footage before we started editing) and how it looks color corrected and color graded:

Unfortunately, my laptop isn't powerful enough to do the rotoscoping with Mocha you mentioned, so I can't isolate the foreground from the background this way. What was left as an option was to create some contrast (as you did with your example) to apply the luma inverted matte, and have one more layer with a mask of the foreground just where the horizon line is, Then I added the clouds image witha lower opacity and played a bit with its color correction and as a results there was a small difference in the sky, but still better than nothing. Here's a quick video of this result: https://we.tl/t-LBA68pH5jJ

What aslo still puzzles me is that after camera tracking, and creating a null and a camera, and after having a contrast betweeen the foreground and background, if I replace the sky with a static photo with clouds and make it a 3D layer, things just don't work when it comes to the clouds moving along with the camera. If i replace the image with a footage of clouds, it seems to be working fine. Where could the problem be? I'm attaching a video again that displays a rough version of this with an image, instead of a footage.

[Rick Gerard]

The preview looks pretty good to me. 

Here's my workflow with any raw or log footage.

• Copy the Lut if there is one available for the basic color correction and apply it to the footage in After Effects so you know where you are starting from or...
• If there is no Lut available, color-correct the footage in the Main Comp using an adjustment layer and pre-compose the two layers and name the Pre-comp "CC Master" to give yourself a decent starting point (Tracking and creating mattes from log footage is nearly impossible without color corrections)
• Duplicate the "CC Master" layer and add additional color correction so the layer has more detail, contrast, and harder edges, pre-compose, then motion track the layer to either stabilize it or get some tracking data for the critical parts of the shot, And name the layer "Tracker"
• Duplicate the "CC Master" layer again, name the copy Matte, and apply additional color correction to generate a black and  white layer you can use as a Luma Matte
• Add Black and a White solid above the Matte layer and add or animate masks to fix any holes you have in the Matte layer
• Pre-compose the Matte and the Black and White solid layers and name the Pre-comp "Luma Matte," then move it to the bottom of the timeline
• Add your replacement layer or layers to the timeline and set the Luma Matte layer as a Luma or Luma Inverted track matte to cut out your replacement layer or layers and Pre-compose and name the comp "Background"
• Add any color correction you need to the Background layer or Layers to make it look like it is part of the "CC Master" layer
• Motion Track the "Tracker" and extract the data to Nulls or Corner Pin data you can use to Motion Stabilize or copy the movement of critical pieces of the shot.
• Use Expressions or Parenting to add the motion in the shot to the "Background" layer or layers
• If you used Corner Pin/Power Pin to Motion Stabilize part of the shot, follow the workflow in the tutorial I shared

If you did everything correctly, you should have a color-graded shot and a good-looking sky or other object replacement. You now have two export or rendering options. 

The first is to render the entire comp with the Color Correction and replace the shot in your NLE. You remove the color correction in the NLE on the original shot by cutting the adjustment layer if it is used or by putting the completed composite above the other layers. 

The second option is to turn off all layers except the Background layer and Render a High Quality with Alpha production master using the Render Queue and the Hight Quality with Alpha Output module preset available under the preset rolldown (looks like this: >). You then add the Replacement sky to your original sequence on the top layer and make the color correction adjustments you need to the layer to make if match the original color-corrected layer.

The only time I would ever render a Log version of the completed shot would be when the replacement sky footage was already Log (or flat). In that case, only would I turn off all color corrections except the correction that was necessary to make the log background replacement footage match the original log footage.  This would also be an extremely rare workflow because judging Log footage without color correction is almost impossible. My AE comp would need an adjustment layer over the entire composite that I could use to give me a full range of values in the preview that I could turn off before I rendered a Log version of the composite or a log version of only the background (sky) replacement layer.

There would be no 3D layers because I would not use Camera Tracking for a Sky Replacement because there is no parallax in the sky. I'm not sure why you have 3D layers in your comp. If you used the Camera Tracker to add something else to the shot, you need a slightly different workflow.

I hope this jumble makes sense. 

If I get the time, I'll take the shot you just shared and create the comp I would use to replace the sky and setup the different rendering options I would use. 

[Nickwhy90]

Thank you for describing your approach! It really is a bit overwhelming, because it has quite some steps and details. I think I get most of the stuff, but perhaps I'll end up wondering around a bit if I start to actually implement it.