The arrow keys work without hassle when Channel is set to RGB (default). However, if I change to a different channel and place a new point on the curve, the arrow keys switch between channels instead of moving the new point.
There is a workaround, but it is not intuitive and requires an extra click: Open curves tool. Select a color channel using either the drop down manu or pressing Alt+3, Alt+4, or Alt 5. Place a new point on the curve. Then, click your mouse in the Output field or the Input field (where the number is displayed). Then the arrow keys move the point properly. This is on Windows 10, PS version 21.2.4.