Since there is no uservoice page for Photoshop, the topic is posted here. Not sure if this is a bug. It's rather something that simply doesn't work as expected. Steps to reproduce:
1. Create a new blank file of any size and then press Ctrl+a to select all
2. Apply "Edit mask -> Round" and set any radius
"Radius" is related to a circle. In this case the edges should be rounded and look like 1/4 of a circle. The result is different (picture below uses a stroke for illustration). If I create a circular mask with a radius of 30 pixel and place it over the rounded edge, then it looks as expected. The workaround would now be to actually merge 4 of the circular mask with the edges to get the desired result.
Note: someone might want to interject that this comes from an image not having a ratio of 1:1 like a circle has, but when creating a square in step 1, the result is the same. Strictly speaking, when creating a square of, let's say 300 pixels and rounding the edges by 150 pixel radius, the result should be a circular mask.