Disable value editing in speed graph

Your screenshot shows an instantaneous jump in speed. If you want a smooth speed transition, slide the incoming and outgoing squares together so they are on the same line.  This will change the shape of either the left, right, or both sides of the speed curve. There is no way around changing the curve if you change the acceleration. Maybe it will help if you think of the speed graph as an acceleration graph. The steeper a rising curve is, the faster it will accelerate. If the timing remains the same when you accelerate or decelerate faster, the other end of the curve has to flatten out. The curve changes because the distance and time remain the same. You can hold down the Alt/Option key to snap the speed values together or split them. 

Maybe this tutorial I did will help:

Changing the timing of the keyframes does not change a motion path. Value graphs are almost completely useless for any property that has an array like Position. Value graphs can be helpful for properties like Rotation. The speed graph should be your most used view.