Changing background scenes

I'm mid animating in AE from CH and PS files. I wish to drop in a new background ( replacing a boat with a map) and take out other elements and characters for this scene.Then I want to revert back to original boat scene. Currently when I do this, the new background ( the map) is there from the beginning of the animation, the same when I add other new elements; it just changes the whole instead of just the one scene.
I hope someone can point out what I'm not doing. Thanks