Time remap graph is a straight line

Cropped screenshots are pretty useless when it comes to diagnosing problems. If you are looking at and editing the speed graph and you have a straight line then the speed of the playback is constant. You can have a bunch of keyframes, but if you have not moved any of them left or right then the speed doesn't change and the line is straight. The higher the line the faster the clip plays back. I would bet that your graph is set to the speed graph. Value Graph playing forward at a constant frame rate.

Speed Graph with constant playback rate:

Speed Graph that shows slowing and speeding up;

Same keyframes with the Value Graph:

Value Graph showing clip playing forward then backward, then forward again:

If you look at the value graph, the line is at an angle. As long as the line is angled up the clip is playing forward. When the line moves down the clip is playing backward. If you are trying to hit a specific frame at a specific time then finding the frame, adding a keyframe, then dragging it to the proper time is the easiest way to get there, Then you use the Value graph to adjust acceleration and deceleration for the time change. 

If you want to speed up and slow down and have that change consistent then manipulating the keyframe position while adjusting the curves with the speed graph is a good workflow. Your only problem is that you are not understanding what you are looking at.