Speed remapping automation.

Anytime you change the playback speed of a clip you change the direction. Using time remapping will not move the markers so you'll have to just go through your shot one-speed change at a time.

The practical way to do this just to figure out when you want to speed changes to happen and set a keyframe at each speed change. You need to keep notes to tell when things are supposed to slow down and speed up. When you get to a section that you need to slow down, or speed up, you'll have to select every keyframe to the right of the keyframe that starts the speed change. Then you grab and drag those keyframes to the left to speed things up or to the right to slow things down. 

You can use the speed graph to smooth out the transitions and make the speed changes look better.

I don't know how long your clip is but the first thing I would consider would be to do some editing. A five-minute clip with 100 changes in speed, could stretch into a movie that was 20 or 30 minutes long. Frankly, that would be pretty boring to watch.

There may be other options, but no matter what method you use you will need to either mathematically calculate the distance between the slow parts and the normal speed parts and the fast parts of your shot. 

The easiest solution may be to just put a cut every time you want to speed change in Premiere Pro, then do your speed changes there and forget about AE. If you want to ramp up to speed changes just split the layer in AE every time there is a speed change. You could then set a time remapping keyframe at the end in the air point of each split. Drag the last keyframe around to change the timing and then ease then both.

This is what doing everything on one layer looks like. You start with keyframes set for every spot where you want a speed change.

The little diamond in the timeline is used to add keyframes without changing the timing of the shot.

The speed between all keyframes is 1second per second or real-time. Now you start moving groups of keyframes around to speed up sections and slow others down.

Moving a pair closer together makes the speed go up. Farther apart and the speed goes down. You can tell how fast the clip is playing by looking at the ratio in the graph editor. The top part of the second pair of keyframes is at about 1.4 seconds per second so the clip is playing back about 150% faster than normal. This can get really confusing really fast.

The other workflow is to split the layer everywhere you need a keyframe, pick the layers that you want to change, assign them a label group, add time remapping to just the clips you want to change, set a new keyframe at the in and out point of each layer, then just drag the last keyframe around for each clip to get the playback speed you want. All that is left is to adjust the out points of the layers you edited and line up the cuts. Personally, I think this is an easier workflow and you never have to worry about the real-time sections playing at a different speed. That workflow looks like this:

All you have to do is add a little easing to the time remapped layers and you have a nice transition from real-time to slow or fast-motion.