Do you have the Layers panel open? See the toggle in the lower right toolbar. You can also use F11 as a toggle. You must also have a photo open in the Editor for the Layers menu option to become active. If this doesn't solve your problem, please provide a screenshot of what you are seeing.
@default7zgumj997di5, it is hard to see why the layer mask options are disabled in the Layer menu, without seeing the layers panel. Please provide a screenshot of the panel and tell us what you are trying to do with the layer mask. In the meantime, you may want to take a look at this HelpX Article about layers.