Time Remapping / slow down to speed up.

You have two timelines selected and you are using the Value Graph. There is no part of either layer that is playing at a constant speed. One layer has 3 keyframes and the video slowly decelerates until it almost freezes, then accelerates after keyframe 2 through keyframes 3 and 4, then slowly accelerates to the end of the timeline. There is no way to tell when or if the layer is ever playing in real-time. The steeper the line, the faster the playback, the flatter or more horizontal the line the slower the playback. There is no way of telling what real-time is in the Value Graph unless you memorize the angle of the line when you first enable time remapping and only have 2 keyframes. On the other hand, if the Value Graph shows 1 second/second, the video is playing in real-time. It is very easy to see what you are doing.

The second layer gradually accelerates until it freezes then plays backward for a few frames then accelerates again. This is the problem when you try and use the Value Graph with Time Remapping. 

I suggest that you start over and use the speed graph. You should also spend some time studying the Graph Editor in the User Guide. If you want to slow down a section of your video do that one layer at a time. Set keyframes where you want the slow down to start and where you want the slow down to end, then grab the last 2 keyframes and drag them to the left so the time between keyframe 2 and the rest of the keyframes becomes longer but the distance (time) between keyframe 1 and 2 and 3, and 4 stay the same. Then open the graph editor, right-click and select Edit Speed Graph and try and adjust the speed curves to get something like the curve I show in my screenshots.  Working with the Value graph is not going to get you what you have said you want.