How can I mask the transparency area of a picture?

This tutorial may help you How to Invert Layer and Vector Masks in Photoshop - YouTube

Consider an option where a mask is not needed at all. I imagine you have a template with transparent cells such as the one shown here. If that is the case:

Use the Magic Wand set to Contiguous in the Options bar to Select a cell. Marching ants will appear.

Choose an image, such as the one on the left. Select > All followed by Edit > Copy

Return to the template. Choose Edit > Paste Special > Paste Into. The result will be Figure 2

To resize the the image to fit, Edit > Transform > Scale. Hold down the Shift key to maintain proportion.

Select the inside of the frame, click on the new layer mask button at the bottom of layers panel. If it is backwards, hit cmd or cntrl + i with the layer mask selected and it will invert your selection. You can also double click on the layer mask and a properties panel will pop up - it has an invert button.