Speed graph inverted

I think you are trying to control the position with the speed graph. That Can't be done. Most people also do not understand what the Ease assistant does. You also talk about moving Ancor points in the Graph Editor, but I think you are talking about the Bezier handles.

Here's the workflow for putting slowdowns in constant movement. 

Set the starting and ending keyframes. These must be linear keyframes. Now move down the timeline and set a keyframe, but do not adjust any values where you want the first slow down to begin. Then move down the timeline to where you want the speed to pick up again, then repeat the procedure so your speed graph looks like this:

The next thing you must do is select all keyframes and open up Keyframe Interpolation (Alt/Option + Ctrl/Cmnd + K - or use the Animation menu or right-click) and set temporal and spatial interpolation to Linear.

Now use the j and k keys to jump back and forth between the keyframes. You are going to adjust the value of the third keyframe first. You need to adjust the end position of the slow down. Do that by dragging the appropriate value in the timeline. DO NOT touch the graph. You get something like this:

What we have now is a constant rate of speed around 120 pixels per second between the first and second keyframes, then a constant rate of speed between 3 and 4 of around 50 pixels per second. The next step is to move to the 5th keyframe by pressing the K key 3 times. Drag the position value for the 5th keyframe until you get about 50 pixels per second between 4 and 5 so the graph looks like this:

Now you have a constant speed between 1 and 2, and a constant speed between 3 and 4, and 4 and 5, and 5 and 6. The speed between 1 and 2 is slower than the other two because everything is linear. If you want the same speed between 1 and 2 you will have to adjust the starting position of 1. Moving the first keyframe position to match the speed of all three sections will give you something that looks like this and the slower sections will be at the right time and in the right position.

 The next step is to convert the temporal interpretation of the keyframes to Auto Bezier and that is easiest in the graph editor by clicking the Auto Bezier icon. You can use the Keyframe Interpolation panel if you want.

When that is done you can control the shape of the curves by changing the values in the Keyframe Velocity panel (Shift + Ctrl/Cmnd + k) until you get something like this:

When you get this far you can start adjusting the bezier handles on the speed graph by dragging them LEFT or Right. If you move the bezier handles up or down you will start putting glitches in the speed. You want to end up with something like this.

There is no way at all to get this to work without having the proper settings in Keyframe Interpolation and you cannot do it at all unless you edit the position values in the timeline.

I hope this helps. 

I think you are misunderstanding what the speed graph is showing you. The angle of the line shows acceleration. An angle like this / is showing a constantly increasing acceleration. If the line is angled down \ the acceleration is decreasing. A straight horizontal line means constant motion between keyframes. If the line is vertical it means an instant change in speed between keyframes.

The common conception that the curve represents the timing when it represents the acceleration rate.

Your image 2 tells me that you have a constantly increasing rate of acceleration approaching the first keyframe, then the acceleration decreases and then increases between the next two keyframes, then constantly decreases as it approaches the fourth keyframe until there is no motion between 4 and 5, then the speed gradually increases then gradually decreases as the property comes to a stop on keyframe 6. Changing the length of the handles changes the acceleration curve Moving the handles up or down changes the speed at the keyframe position.

Did you follow that? In the first screenshot, the movement comes to a stop between each keyframe. In the second screenshot, the movement never comes to a stop. The layers are actually never moving slower than 1000 pixels per second. Bring the handled down so the speed at the keyframes is lower and the shape of the curves will change. The hardest part of editing the speed graph is understanding what you are seeing and realizing that the overall average speed must remain constant if the timing of the keyframes does not change.

The total time from the first keyframe to the last never changes so the acceleration curves between all of the keyframes must change because you have changed the acceleration curves approaching the fifth keyframe so much that the layer will come to a complete stop before it starts is uniform accelerate - decelerate values to get to the last keyframe. 

Edit the value graph and move the tangent handles around.

Mylenium