Can't make straight lines after a curved line when masking

What you are describing is just the way the bezier Pen in Adobe software works.

Click and Drag to create a bezier (smooth) anchor point (vertex), this creates two  direction handles attached to that point. The first handle controls the curvature of the line you just created, but the second handle controls the curvature of the next line.

To make a straight line after a curved line:

1. Create the curved line as normal.

2. Hold down Option (Mac) or Alt (Win) key on your keyboard and drag the direction handle of the second direction line (the one that is pointing away from the line you just made) back into the Anchor Point (vertex).

3. Click somewhere else to make your straight line.

were you able to find the solution to this? I don't think it's about the Rotobezier

Select the mask path in the timeline. Go to the Layer/Masks and Shape Paths menu and uncheck Rotobezier.

If you have the pen tool selected (g) you can hold down the Alt/Option key to change the pen to the convert vertex tool to collapse (make straight) or expand and move the Bezier handles (adjust the curves). The Ctrl/Cmnd key with change the pen tool to the open selection or solid selection tool to help you select points (vertices) and groups of points. This makes editing paths easier. If the Path is selected in the timeline the selection tool (v) also works well to edit paths, but if a path is not selected in the timeline editing paths with the selection tool is difficult if not impossible.

Switch off the "Rotobezier" option in the Toolbar before you begin drawing your mask.