Pan like a camera

Let's say I have the picture down below, the black boxes will be videos playing and the yellow is the background.

But I wan't to show each clip indiviudally, so I zoom in on the first clip like shown below:

So I show the first clip, then I want to pan to the right, show the second and so on.

So here comes my issue, since the background and the 3 black boxes are seperate video clips, if I would like to pan it, I have to move every clip at the same time individually and at the same speed which takes a lot of time. I have tried nesting it, but since the nest only shows the 1920x1080 window, I have to zoom it out a lot which lowers the quiality of the clip. I have also tried fooling around in After Effects, but I found it quite confusing, is there any effective way to do this in Premiere?