First, without seeing the modified properties and keyframes of the layers giving you problems it is impossible to figure out what you have done. The most likely cause of this kind of behavior is a parenting issue. You have probably parented your new layer to something else in the scene or not properly lined it up in 3D space.
It sounds like you are ending up with a very complicated comp with many many layers and basically editing the piece in AE. This is usually a bad idea. If this were my project I would break it up into pieces. No comp would be longer than an inserted video plus a couple of frames. Comps with still shots would not be longer than a phrase in the audio track. It's pretty easy to break up a long and complicated composition into manageable sections by just duplicating the comp in the project panel, giving it a good name, then resetting the work area to the start of the new section, deleting all layers that end before the new start of the work area, trimming the duplicate comp to the work area and moving on from there. You might want to consider splitting your comp one shot before the trouble starts and starting to diagnose the problem from there.
I'm recovering from the flu so if you don't mind I'll take a moment and propose a more efficient way of working.
From what you are describing I would probably set up a "table" comp with the background shot of the table and magazine as a 2D layer. This comp and the image of the table would have to be big enough that I could drop a project sized solid in the comp and place it over one of the photos or pages in the magazine that you want to replace and it would almost perfectly fit with just a little rotation.
Say you are planning a standard HD 1920 X 1080 project so the smallest part of the magazine that you want to see full should be about 1900 pixels wide.
Did you follow that? Your background image needs to be big enough to that you do not have to scale it up or move the camera in from Its default position to create the scene you want to create.
When you have the Background comp made add a fairly large solid, apply the grid effect to the solid and make it a 3D layer. Now adjust the orientation rotation and z position of the solid until the grid lines up with the perspective in the shot of the magazine on the table. When you have got that right you have your 3D stage where you can place your new images and photos. You are now set to begin your animations. Inside this comp, the "Table Comp" you can add every element in your shot. Just drop them in, make them 3D, hold down the shift key and parent them to the locked grid layer and they will snap instantly to the center of the grid layer and be properly oriented. Now just carefully grab the X or Y vector in the comp panel and slide the asset into place. That is the easiest way I know of to line things up in 3D space to match an image shot by a camera. The last step is to duplicate the background layer, make it 3D, Shift + parent it to the grid layer then rotate in z and scale until the 3D background layer matches the 2D background layer. You can then turn off the grid and the 2D background layer.
Now that you have that "Table" Comp set up all you have to do to drop it in a new production-sized HD comp (1920 X 1080 in this example) add a camera and collapse transformations. You are not going to be able to see all of the Table comp so it's time to position the camera. Select the camera and press AA to reveal the zoom settings then hold down shift and press p to reveal the position property. You will want to adjust the zoom value of the camera to pretty well fill the frame and then make any final tweaks to line things up by adjusting the z position of the camera. Set a keyframe and make sure that you save the position data. I usually add it to a null and lock the layer so it doesn't get lost.
Your master comp is now ready to animate. If the project is not very long just start moving the camera. As you move in so that only the table top is showing set and save another keyframe. Now you can start animating the point of interest and X, Y and Z position of the camera to line up each new part of the layout. You can jump back into the nested comp to edit transitions or other effects on the images. You can duplicate the main master comp as many times as you like in the Project Panel to create different camera moves and then trim the comp to just the time you want for the move.
This will allow you to do one master table top comp and then cut together a bunch of amazing camera moves and effects where you replace magazine pages and images with your own images and videos by basically setting a few position and point of interest keyframes.
I hope this helps. Sounds like a great project for a tutorial. Every one of them that I have seen on this type of composite has concentrated on a solution to replace a single or a couple of images and while many of them were OK and a couple were pretty darn good, none of them approached the problem of efficiently handling a large number of composites.
AdChoices