Motion Tracking, Picture Frame substitution, and reflections - How?

I took a look at the footage you gave me and created a comp to try and figure out the best tracking solution. I'll give you an overview of my workflow and share a file at the end that uses the footage you supplied.

Step 1: Create a comp from the trimmed footage you supplied, added an adjustment layer and color corrected using Lumetri to give me the best color and most detail that I could get from the shot. The reason for the color correction is to provide as much information as possible in my 32-bit project.

Step 2: Pre-compose to create nested comp that I could use for Motion Tracking naming the pre-comp Nested Footage.

Step 3: To provide a comp for testing the tracking and figure out the best method I duplicated the master comp, named the copy Test comp, reset the composition frame size to 1080 X 1920, scaled the CC footage layer (nested comp) to fit horizontally, pre-composed the CC footage comp and named the new comp for Tracking Test comp. If I do not collapse transformations this gives me a much smaller frame size to experiment with the tracking methods that will analyze, track, and render faster. 

Step 4: I ran tests with Mocha AE and Camera Tracker. There is just not enough edge detail in the Monitor that is on the left to get a good track. If you had put up any kind of an image with some detail on the monitor screen. If you were using the monitor for a light source you needed some detail on the monitor frame that could be tracked. Not just tracking markers, but something like a checkerboard patterned tape on the edges that could be easily tracked and replaced.  Mocha also had some problems with the Picture frame on the right side of the frame. They were not as bad, but Mocha was not going to be a good solution to this shot because of the lack of detail in the monitor and the edges of the picture frame. 

Camera tracking worked much better if I turned on Detailed Analysis in the Advanced section. I got a tracking average error of about 1.25 pixels. Now it is time to Camera Track the full resolution copy. The total time so far is about 20 minutes.

Step 4: Camera Track the main full resolution comp. This is self-explanatory. I just turned on Advanced analysis, made sure that the process had started the track. On my basic M1 Mac Mini I'm using for testing right now the estimate for the tracking to complete was about 40 minutes. It's about what I expected from this 6K footage. I had dinner and came back to the project to set up the camera and some track solids.

Step 5: Establish an origin and ground plane. I picked a time where I had a decent number of trackers on the desktop, set a layer marker, then established an origin and ground plane when the selection I drew around the trackers looked correct.

I added a camera and solid using the same tracker selection. A little rotation and scaling of the Track Solid completed this part of the project.  I added the Grid effect and checked the track. It was good.

Step 6: I then selected track points on the monitor. There were not very many, and I could not get the target to line up with the surface very well, but I could adjust orientation later, add a grid,  scale and use the multiply blend mode to get the grid to line up with the monitor frame precisely, then check to see if I got it right. This is the most frustrating and time-consuming part of the project. On the second try, I got everything lined up and the track appeared to be good.

Step 7: Repeated the process for the clock and the picture frame. The clock was the easiest, the surface was perfectly aligned the first time. The picture frame was easier than the monitor but it still required manipulation of the orientation and some scaling to make it work. 

Step 8: Add some footage for the picture frame to the comp, make the footage layer 3D, then shift + parent the footage to the picture frame to snap it into the same position orientation and scale. The footage was then scaled to match the picture frame. 

Step 9: Enable time remapping on the replacement footage, pick a frame that I wanted to freeze and set a new time remapping keyframe, delete the first time remapping keyframe, then select Time remapping to select all time remapping keyframes and grab the first keyframe in the timeline and slide it down the comp to pick a place where the picture turns into the movie.

Step 10: Set up the push-in on the picture frame. Because perspective is controlled by camera position and framing by focal length you want to animate the zoom value (press aa with the camera selected) and scale on the replacement footage. If you need to reframe the shot as you zoom in you also can animate the orientation and rotation of the camera. I picked a point where I wanted to start the push-in and set a keyframe for scale on the replacement footage and zoom on the camera, then I moved down the timeline and adjusted both values to carefully match up the shot. I had adjusted the grid on the tracker solid for the picture frame so that the grid was perfectly lined up with the markers in the picture frame. This is what that looked like:

This is the result so far:

Sorry about the small size and low frame rate of the gif but you get the idea.

All that is left is to set the tracker solids to guide layers, replace the footage on the screen. add some reflections to the replaced footage by matching the new footage on 3D layers using Shift + Parent in the same way that the footage was matched up to the picture frame.

When all of that is done I would also set the footage layer (Layer 7 "CC for Tracking") as a guide layer, then render a Lossless with Alpha copy of this footage that you could overlay in the final edit. That way you have complete control over the color correction on the original footage and on the overlays. I might even consider rendering a computer screen overlay and Picture frame overlay separately. For most of my clients, those renders would be EXR sequences or ProRez 444. 

The total time on the project was just over an hour and a half. Here's my sample comp for you to take a look at. 

I hope this helps. This would be a good project for a tutorial. Especially good if you shot the same original footage two ways. One with more detail in the shot and one the way you did. 

I also mostly work with Raw footage. I almost always add color correction and pre-compose before tracking with the camera or using Mocha or any other tracker. I need to add as much detail as I can most of the time. When I am ready to render the CC is turned off. 

If all you are replacing is the picture frame I would use the Stabilized Power Pin method to replace the picture in the frame with footage. I would use the footage and apply time remapping to the footage so it could be frozen until you do the transition.

If the shot of the desk and picture frame does not push in to fill the frame with the picture I would just do a simple camera move or a scale on the final composite to transition to the moving shot. I would not try camera tracking on that shot. I don't think it would work very well. 

It would help me work out a workflow if I saw the original footage without color grading. I'd even be happy to show you how I would set up and track that shot. I do not need replacement footage, I have that. 

I would probably budget about an hour to do the tracking and replace the photo with an image. If you would like to share a compressed and/or scaled down copy of the original (it could easily be an MP4) with no color correction I'll try and take a look later this afternoon.

David said he tracked the shot with Blender. That's just camera tracking. The camera tracker in After Effects would be able to do the same thing. You don't really need tracker marks anymore, just edge detail.

Reflections are just a high contrast image with a blend mode and position animated. You pull the mask from the added photos. You can use Set Matte to do that. 

David's shot could have also been accurately corner pin tracked in Mocha AE. You would not have had to use Camera Tracker or use Blender. All you would end up with if you did the camera tracking in Blender is a shot with the photographs properly lined up with the picture frames. There is no need at all to render anything else for the composite. 

The Mocha Essentials link I gave you goes over a lot of the basics in tracking in Mocha. I show you how to handle a surface going out of frame in the tutorial I posted. That is probably the technique that I would use. Camera tracking would be my second option because it would probably take longer and it would be more difficult to properly line up the reflections. 

Another idea for adding the reflection is to put a black card in the picture frame, make sure the glass is clean, then light the scene so that you picked up the reflections you want in the shot. You can then do some color correction on just the black screen and set the blend mode of the replacement photograph to overlay or even screen and the reflections would come through naturally. I do that kind of stuff all the time. 

The Blender community is very powerful and there is a lot of help available right there. For help with the Blender part of your project, you can start here and we can probably point you to the resources that connect After Effects and Blender. I've been using Blender since about 1995 when I discovered it and integrating the animations and models I created there in After Effects projects ever since. If you do need that kind of help we need specific workflow details and samples of your shots.

There are three common workflows for tracking in After Effects. The most efficient workflow depends on the shot and the design requirements. I'll start with the most basic and move to the workflow that can handle the widest variety of shots and solve the widest variety of match move compositing problems.

The Motion Tracking workspace is where you find the options to use the Tracker Panel. Choosing the workspace is from the Window/Workspace menu or the Tool Bar sets up your workspace for the most efficient use of the tools available when using the Tracker Panel. 

You will find Track Camera, Track Motion, Stabilize Motion, and Warp Stabilizer. If the layer has a mask and you select the mask in the timeline The Tracker Panel will change to the mask tracking tools. Mask Tracking happens in the Composition Panel. So Does Track Camera and Warp Stabilizer. Track Motion and Stabilize Motion use the Footage Panel. If you are in the Track Motion workspace and you select the Motion Source in the Tracker Panel, the Footage Panel will open up automatically and you will be able to select the Track Type and add the appropriate tracker to your project. The right options depend entirely on the shot and the design requirements. The shot you describe has a perspective shift so the only viable option is probably Perspective Corner Pin. None of the other trackers have the ability to calculate shifts in perspective. Perspective Corner Pin requires that all four corners be visible in the shot for the entire duration of the shot. If one of the corners goes out of frame, you may be able to manually adjust one or even two corners for a frame or two, but the workflow requires a lot of guessing, it is slow, and it is difficult to keep the track accurate. That does not sound like it fits your shot.

That leaves Track Camera. Depending on the shot, this may be a good option. Track camera requires a fair amount of detail in a fair amount of fixed geometry in the shot. If the shot has significant perspective and parallax shifts you can get a good solution using Track Camera, but if the shot is a simple camera pan over a mostly flat scene, you will not get any depth information and the track may be compromised. On the right kind of shot, Camera tracking does not care if things move in or out of frame. This may be the best solution for your specific shot. I would have to see your shot of the picture frames to be sure. 

The most powerful tracking solution that comes with After Effects is the Mocha AE tracker. It is probably the solution that I would use for the kind of shot you are describing. There are dozens of possible workflows in Mocha AE, so I need to see the shot to give you my recommendation for the most efficient workflow. I can point you to a very good tutorial series on tracking in Mocha AE. The Mocha Essentials course from Boris FX is by far the best place to start learning about Mocha. As with most tutorials, going to a professional source is going to be better than searching YouTube because the web is full of poor workflows and recipes presented by well-meaning enthusiasts that often lead you down dead ends.

From what you explained I would probably use the Stabilized Power Pin workflow that I describe in the following tutorial. If there is more than one picture frame in the shot, you'll have to make a separate track for each frame you want to fill, but this is probably where I would start.

Think of the bench as the picture frame. 

The other option would be Camera Tracking. Again, this depends on your shot. I don't have a specific camera tracking tutorial I can point you to but there are a couple of tutorials in the User Guide that will get you started with the Camera Tracker. You might want to check out the Animate Hoverboard tutorial. You can also search the User Guide for Track Camera and find some more info. If I find a really good Camera Tracking tutorial, I'll post it.

Blender always tracks in 3D space, so this is likely a hybrid track with a faux 3D camera track and then the 2D tracks thrown on top. Not easily replicated in AE due to its inconsistent scene space. Other than that this would likely be a better job for mocha. The planar tracker already takes care of perspective issues and off-screen extrapolation to some degree and due to how it works the actual plane boundaries can easily be expanded after the track.

Mylenium