How large is the image you've added a mask to? That's quite a large feather, but only if the image is 1000 pixels wide. Is it's 10000 pixels wide, then that would explain it.
It might also be the effects you're applying. With such a cropped screenshot, it is impossible to tell what's going on. Can you share your whole screen?
Sorry abotu that, full screenshot now attached. There are two of every layer, the bottom one with the blue light wrap around the edges and one on top without it. The one on top has the mask and I'm using it to control what edges the light wrap effects and doesn't. The turtle is 4k footage which I've rotoscoped.
To control what edges have a light wrap, I think it's best to use two separate layers. I'm not sure from the screenshots why the mask isn't working - scratch that. I can see the problem. Rotobrush doesn't seem to work with masks if the mask is added AFTER the rotobrush has done it's thing. So you'll need to either:
Delete the rotobrush effect, then add the mask and then redo rotobrush.
Use a track matte with the mask on another layer
Use the Set Matte effect with the mask on another layer.