Skip to main content
N
Nickwhy90
Known Participant
July 10, 2023
Question

Tracking And Changing Background Sky in Footage

  • July 10, 2023
  • 2 replies
  • 2896 views

Hello community,

 

I'm currently working on a music video of mine where I'm co-editor and also decided to do some VFX where appropriate. I have shots in the open where the wheater was kind of dark and cloudy (as you'll see from the screen recording I'm uploading), so I thought I would add some more clouds and create some lightning here and there. I looked for a way to isolate the sky by  duplicating original footage, working with the tint/levels effects to make the sky very bright, use luma matte, and substitute it with other assets. 

 

I saw in tutorials that 'd have to camera-track the original shot, then create a null and a camera. Then, all I need to do is make the sky-replacement asset layer a 3D one, and it would move accordingly with the camera. However, in tutorials people use video assets for replacement, whilst I'm using dark cloud photos with a bit reduced opacity so they would kind of blend with the original video clouds. So when the camera is moving for example to the left, the original sky of the shot looks like it's going to the right, and the photo assets look like they're moving to the left (compying with the actual tracked camera movement), which creates a discrepancy between the two. So my question is - is there a way to make it look correct, using the static cloud photos I've already placed, or should I replace them with some sort of dark clouds footage?

 

My 2nd question is -When I turn on the 3D tracker camera layer, the whole precomposed comp I'm working on makes the main comp have a blank preview as you can see from one of the screenshots. Why would that happen?

 

Any help is appreciated!

 

Nik

 

This topic has been closed for replies.

2 replies

R
Community Expert
Rick GerardCommunity Expert
Community Expert
July 13, 2023

I downloaded one of your low-resolution screenshots and this Adobe Stock image to create a quick procedural matte and color correction workflow.

 

If I had the footage, the first thing I would do would be to motion stabilize the footage using Mocha AE so that the motorcycle does not move in the frame. I did not show that in the sample comp that you can download.

 

After Adding the footage to a comp, the first thing I did was apply some color correction to increase the contrast and give me some edges to work with. Then a Black solid was added, and the hand Rotoscope was completed to black out the foreground and the reflections on the motorcycle. Then I did the same thing with a white layer to clean up the sky and poke a hole in the mirror. I would have used Mocha AE to do the Roto if I had a movie instead of a still to work with. Rotobrush would be useless on this shot. 

 

Now that I have a matte created, I drop the original footage and the replacement sky in the comp, pre-compose the layers sit took to make the Matte, and use the Pre-comp as a Luma Track Matte for the replacement sky. A bunch more fiddling with opacity, a master adjustment layer, some more masks, and I ended up with something like this.

To really sell the shot you'll need some more work on the Luma Matte, with some more layers. You will also need to have a separate roto of just the mirror and another copy of the sky that is flipped and motion tracked to put some realistic storm sky reflections in the mirror. 

 

If your shot is 7 or 8 seconds long, and you get fairly comfortable with the workflow i would guess that it may take up to a half a day to perfect the shot because of the cmaera movement and the required Roto work. 

 

If you download the project and fiddle with the controls it may give you a start on the techniques you will need to add some elements to the sky. The most time consuming part of the project will be the Rotoscoping, but Motion Stabilizing the Motorcycle will and Mocha AE will make that workflow a lot easier. 

 

Here's a link to a great set of Mocha AE tutorials. 

 

I hope this helps. 

N
Nickwhy90Author
Known Participant
July 13, 2023

Thank you! I'll try to implement what you've described. May I also ask if it's better to work on the original footage without any color correction and color grading and then color grade after I'm done with the sky and the lightning thunder, or should I use the color graded version and try to adjust the colors of the new sky and the lightning according to it?

R
Community Expert
Rick GerardCommunity Expert
Community Expert
July 13, 2023

It looks like you have some log footage. To create the mattes, you'll have to color correct to generate the greatest difference between color and luminance values. If you look at the sample project, you'll get the idea. 

 

When you make the composite, the replacement sky should also be color graded, and the source footage should have color grading that closely matches your final output. You can then render just the inserted sky by turning all of the other visible layers into guide layers, or you can render a 10-bit or better production master close to your video's final color grade.

 

When making any kind of movie that is longer than one or two shots, I always do my final edit, color grade, and sound mix in Premiere Pro using original footage, rendered production masters, and all of the other assets I have created. My final audio mix is almost always polished in Adobe Audition through Dynamic Link because there are more tools there. When I edit music videos, I always request the multi-track recordings from the artists as well as their production mix, drop them all in Premiere Pro, then do the final mix in Audition. That is really the only workflow you can use to produce industry-standard quality results. 

 

If the final edit is done in DaVinci or Final Cut, or even Avid, I still supply 10-bit or better (ProRez usually) production masters for the effects and composite shots, all of the audio resources I can, and some color grading guidelines if I cannot be present for the final mix and color grading. 

 

I hope this helps. 

R
Community Expert
Rick GerardCommunity Expert
Community Expert
July 10, 2023

I think the best approach to generating a procedural matte would be to add a significant amount of color correction to the shot to increase the contrast between the subject, the background, and the sky. A couple of copies of Brightness and Contrast might be a good place to start. You will want to improve the edge contrast of your singer and the motorcycle, and you'll just have to live with the depth of field problem with the trees in the background. When you get as much improvement as you can, you might be able to Pre-compose, moving all attributes, then run Rotobrush on the Pre-comp to generate the first matte layer. I would select the sky, then use the Refine Edge brush to bring back as much detail you can get in the out of focus background. When you have the firest matte created (the Rotobrush Comp), you will need to start stacking up addional copies of the original footage with different amounts of color correction to use as luma mattes for the background. The shot is going to requiere a bunch of hand masking to isolate the sections you are trying to turn into a luma matte. 

 

When you are done, you might have to create a subtle expanded and feathered mask at the horizon line so you can face in your additional sky effects. The whole thing would then need to be pre-composed and used as a track matte for a copy of the original footage and the storm elements you want to add. 

 

When you get the entire shot put together, it would be time for an adjustment layer with Lumetri applied to do the final color correction on the entire composite. At first glance I am guessing that you will probably need at least 3 copies of the original footage with different masks and color corrections to create the final Track Matte for the sky. 

 

I've dont a lot of this kind of work and I expect that it would take me about a half a day to complete this short effects shot. It could easily stretch into a day and involve Mocha Pro or Mocha AE for some of the rotoscoping work that is going to be required. 

N
Nickwhy90Author
Known Participant
July 11, 2023

Hi Rick,

 

Thank you for the detailed input. 

 

Does that mean what you're suggesting avoids using the cloud photos and working only with the original footage by isolating the sky and create copies with different color settings?

 

 

R
Community Expert
Rick GerardCommunity Expert
Community Expert
July 11, 2023

No, I mean the first thing to do is create a matte that will poke a hole in the sky. If you can post a full frame of the original footage, I will see if I can get you started.

 

Here's a short example (unfortunately with no workflow explanation) that I created for a class I taught a while ago: 

Here's a frame showing the original sky on the right and the replacement sky on the left. 

The original sky looks pretty blue, but I did not use any keying. A copy was color corrected to give me a solid white sky and a black foreground, A Solid Black layer with a mask covered all of the brighter details in the road. Then that layer and two copies of the original color corrected original shot were pre-composed and used as a procedural matte for the replacement sky. The Track Matte layer looked something like this:

Because the camera was moving, I also had to motion track the shot using Mocha AE and then add that motion to the replacement sky shot. By the time I was done, I had preserved the haze, most of the powerlines, and the moon and added a much more interesting sunset. The entire shot was then color-corrected using Lumetri on an adjustment layer at the top.

 

I keep trying to find the time to do a concise procedural matte tutorial. I'm asked to do a lot of this kind of work, and no one recipe works with all shots. I've been able to create the masks with nothing more than Curves and Levels, but on other shots, I have needed a stack of four or five layers with a half dozen color correction effects on each and some Rotoscoping to get a believable matte. 

 

Share a full Rez screenshot, and I'll show you how I would begin. You'll have to figure out the motion tracking yourself, but it should be pretty easy to do using Mocha AE because you have a fairly decent horizon plane you can track, so your replacement sky elements will follow the camera movement.

 

 

Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices