Given your description it's difficult to know exactly what procedure you're referring to here - but my guess is that you might be talking about effect compositing options.
Normally when you apply a mask to a layer, you will mask - that is, 'cut out' - the layer. In that case, assuming there are no other visible layers below, you'll see the background composition colour: typically black, as in your case.
However, there is another option when using masks... In your screen shot, looking at the timeline, you'll see 'Compositing Options' in the Directional blur effect. Click the + button to the right and you'll get a menu from which you can select a mask. If you only have one mask on the layer it will be automatically selected. Doing this means the mask now no longer masks the layer itself, but masks the extent of the effect on the layer. Is this what you are looking for?
One other related tip: Directly to the left of the 'Active Camera' menu at the bottom of the Comp window is a checkerboard icon. Enabling this will display Comp transparency as a grey / white checkerboard, which may be more helpful for you in detremining what areas are transparent.
... View more