Best way to match rotoscoped character movements and scaling to different background

Let me start by naming your shots. The shot that you rotoscoped with the dancing elf will be called "Master."

The shot you have taken or will take of your young nieces will be called  "Insert."

The efficient workflow is to Motion Track, Camera Track or Motion Stabilize the "Master" so you have a starting point.

If the rotoscope was created on the Master layer you should turn off the mask/matte at this point. You need a Master you can work with and see so you can start lining up the "Insert."

If the "Insert" was shot with a locked-off camera against a green screen, the next step is easy. Just remove the background. If the "Insert" was shot with a moving camera and you took notes about the camera movement in the original shot and tried to match the camera move you'll have to do some tracking and some keyframing to get things to match up. If the "Insert" was shot with a moving camera without considering the camera moves in the "Master" then you are going to need to do a lot of keyframing.

With the Master tracked or stabilized and the background still visible it's time to line up the Insert. From what I have seen, motion stabilized might be the best option. If I saw the original "Master" shot, the one with the character dancing, I would have a better idea of the best workflow.  

When you have the Master and the Insert locked together it is time to start inserting the new background layers into the shot. If the "Master" has been motion stabilized then this part is easy. Just set up your layers in 3D space and arrange them so that the perspective and the ground match up. You may need a duplicate of your "Master" with the Rotoscope turned on so you can layer things. Once you have that all set up it's time to put the motion back into the scene or coordinate the camera tracking or motion tracking data with the other layers. This is the time you start running ram previews and setting any keyframes you need. The layer order would look like this:

Layer 1:  Insert

Layer 2: Rotoscoped Master renamed "Master Roto" 

Layer 3: Foreground replacement - objects closest to the camera 

Layer 4: Middleground replacement - objects a little farther from the camera

Layer 5: Background replacement - objects farthest from the camera

Layer 6: Ground Plane - a 3D layer simulating the ground the actors and the character will dance on

layer 7: The original Master layer as a reference set as a guide layer so it will not render or turned off

If you want the Master Roto and the Insert characters to go behind a foreground object move the Foreground layer above them both.

If you have motion stabilized the Master you will use the Animation preset applied to a null, rename the Master layer 'stabilized', and then parent all layers to the master to put the camera movement back into the shot.

If the "Master" was Camera Tracked, all of the layers above would be 3D layers positioned to replace planes you identified and added Track Solids to so you could verify the camera track. It is highly unlikely that you would get a good Camera Track from an animated film because the perspective and parallax changes are not usually accurate.

If the Master Layer was Motion Tracked, then the motion tracking data would have to be applied to the other layers. That is usually done by applying the tracking data to a null and then parenting the other layers to the null.

One more thing. This shot you are working on should be the only thing in your comp. Your screen recording shows a lot of extra layers extending way beyond the duration of the dancing character. After Effects is not an editing app. Each comp should only contain the elements you need to create your shot. You'll render the comp and edit the movie layer. If you have more than two or three shots in the movie you'll do your editing in Premiere Pro. Most of the movies I work on are about an hour long. Most of my AE comps are under 7 seconds because most of the shots in most of my projects are under seven seconds. I set up AE Project files for every shot in a scene that requires some kind of work in AE. I render those shots and do the editing in Premiere Pro because that workflow takes a lot less time. Break up your shots into individual comps and the whole project will be easier to manage.

I hope this gives you some pointers. If I saw the original footage of the animated character I could verify the most efficient way to put this part of the project together.

If you plan to do a lot of this kind of thing I recommend that you read a couple of books. I should prepare a reading list. I've been at this for a very long time and I read books on the subject all the time. If you properly plan almost anything is possible. 

Camera tracking will never work on the winter scene footage in your video. There is no real parallax shift. The best it could possibly do is solve for a camera pan. Moving and scaling the background footage is just adding problems. I would probably have started by trying to take any camera movement out of the trolls dancing scene and kept everything locked down. It's a lot easier to put some movement in a scene after the composite is made than to try and match the motion. 

If you want to keep what you have done pre-compose the entire shot you have created so you are only working with one layer. Motion stabilize that shot using AE's Motion Tracking workspace and Stabilize Motion. Pick two treetops and set scale and rotation. When the shot is stabilized rename the layer "stabilized." The greenscreen footage should be shot with a locked-off camera.  If it is not you'll also need to motion stabilize that shot to take out all camera movement.

Drop the greenscreen shot on top of the footage layer 'stabilize' and scale and move it to match. Then add a null to the shot and add this animation preset to the null. Now move to the first frame and parent the green screen footage layer and the stabilized winter scene footage to the null. This will put the camera movement back in the winter scene and match the movement with the winter scene shot. The animation preset ties the position of the null, anchor point, and scale values to the anchor point, rotation, and scale value of the motion stabilized footage. The only complex expression is for scale. The position is the same value as the anchor point and rotation is reversed with a minus sign. 

Here's the expression for scale:

x = value[0];
y = value[1];
tx = thisComp.layer("stabilized").transform.scale[0];
ty = thisComp.layer("stabilized").transform.scale[1];
nx = x/tx*x;
ny = y/ty*y;
[nx, ny]

The first rule when creating any kind of composite is to simplify. The more you can simplify the better your chance of success. If you already shot the greenscreen footage with a moving camera and there is any way to stabilize that, start there. Get all of the motion you can out of every shot you are trying to add to the composite. When all that is done it's pretty easy to put motion back in the shot and have everything line up.

The tracking technique you should use depends entirely on the shot. It could be as simple as the Stabilize Motion in the After Effects motion tracker, Track Transform in Mocha AE, or you might need to add rotation and scale. You might even want to use Mocha Pro or Mocha AE + CC Power Pin to corner pin stabilize the shot. All of the stabilization techniques are designed to stop a plane or surface in the shot from moving. Get that surface to stop, do your animation, then add the motion back to the shot and add the motion to the other layers (your rotoscoped actor).

You might also need to use Camera Tracking then add your new background to the comp as one or more 3D layers. This also requires matching up surfaces or planes in the original shot.

Sometimes it is even easier than that. You just put your background elements where they need to be, add a null, Parent all the background elements to the null, then animate the null so it moves realistically.  

Show us the shot and we can suggest the most efficient workflow. I match motion all the time and I use about 10 different workflows. I won't waste your time by pointing you to a solution that might be completely wrong for your shot.

You could start by investigating all sorts of tracking techniques - conventional 2D as well as 3D and then rebuild your snow scene to match the motion. Otherwise such stuff is often simply a mtter of practice, having an eye for perspective and tons of manual keyframing. Entirely depends on what it actually looks like and if in fact the footage was shot with later post work in mind already to facilitate these things...

Mylenium