In the new Photoshop CC 2020 version, there's a bug with layer effects in groups. They seem to clip other layer effects within the group in a rectangular shape.
To reproduce:
- Create a new document
- Create a circular shape as a new layer
- Apply a stroke layer effect to the shape layer
- Put that shape layer in a group
- Apply a drop shadow layer effect to the group (play around with the distance value to make the effect more obvious)
Result: the layer effects are cropped in a rectangular shape
I tested it in Photoshop CC 2019, it works fine there.