In the current beta build, on an existing mask, the Add/Subtract buttons of the Object Mask Tool are greyed out if you have used the Hand Tool (instead of the mouse wheel) to move the view in the Program Monitor.
Also, the new Object Mask Tool should make use of the Ctrl/Alt/Shift keys as modifiers. The old masking used Shift/Ctrl as Add/Subtract respectivally. I believe Ctrl was for point selection.