Mask from path/selection appears pixelated

I'm making complex masks, and using the path tool to create vector paths that I then use as a selection and create a mask from the selection. It's been working normally, then suddenly on one layer the mask from the selection is very pixelated. 

 this is a layer with a mask from selection, looks normal

 this is the pixelated edge with a mask from a selection

I have found a work around which is to make a vector mask, which gives a clean edge. My concern is why it is happening in the first place.

I have tested it and found that: every selection mask on this layer has a pixelated edge, and no other image in the stack has this issue, so obviously it's something to do with the layer. but I just don't know what, because I processed all the images the same. Can anyone figure out what might be different about this layer? is the file size too small? is there a setting that might have accidentally been checked? Thank you for your help

more images attached downloadable