masking not seems to be working

That seems odd at first glance.  The only thing I can guess at would be if there are any effects on the layers, perhaps ones that are drawing the shapes?

I'm not sure without seeing the project myself, (DM me if you like), but one workaround you could try is to use a track matte, assuming you can add the mask to another layer and use that instead.