Geometry Mask is adding random faces from meshes that aren't part of the mask list in the layer.

I have a couple of layers, with Smart Materials, and on each of these layers, I use the Geometry Mask to select only certain objects in my scene. The issue that I am encountering is that some other objects, that aren't part of the mask in that layer, are also affected by it, and some of their faces have the layer data applied to them. I've been trying to understand the issue, but haven't been able to find a solution so far. Any pointer or ideas about how to solve this would be appreciated.

Thank you.