First let me thank the Adobe developers for adding the capability in CS6 for Group Layer Styles. This capability is extremely helpful.
However, when I uncheck the "Transparency Shapes Layer Option" and use the Group Layer/Vector Mask to determine the Group Layer Shape (where Layer Style is applied) then virtually all Layer Style types can form incorrectly under very normal conditions.
Here is a simple example with Transparency Shapes Layer unchecked using Stroke-Outside Layer Style Option (Note: Light blue rectangle is from the View > Show > Layer Edges begin enabled)
Observed Bug Conditions to have Layer Style generated incorrectly 1) Group Layer has Transparency Shapes Layer unchecked 2) Group Layer/Vector Mask is used to define the Group Layer Shape (bug also occurs when there is no Group Layer/Vector Mask as well) 3) The Group Layer Edges (as determined by the contained Layers in the Group) fall within the Group Layer/Vector Mask area.
In the above example, in the top image, the Group Layer Edges are the same as the document edges as set by the "LL & UR Corner Dots" Layer. In the lower image the Group Layer Edges are being determined by the boundaries of the Blue and Green Circles (corner dots Layer turned off).
Here is a table indicating where I have been able to create incorrect formation of Layer Styles (marked by Red) so far.
From a more detailed examination it appears that when Transparency Shapes Layer is turned off, that the Shapes in the contained Layers in the Group are still influencing the Group Layer Shape and specifically, it is being influenced by the the rectangular Group Layer Edges.
I have a more detailed writeup, the PSD file, and example images of all the Layer Style formation failures and would be more than happy to forward that to the appropriate Adobe person.
This could always be pilot error on my part yet this looks like a serious bug under these conditions. I have been able to make 80% of the Layer Styles fail. Workarounds for this issue are quite doable yet inconvenient.
Hmmm, 9 days and not replies. Well here are some additional examples identical to the first post yet just changing the Layer Style. The one on the left is what is expected and the one on the right is what Photoshop delivers:
This one is for the Bevel and Emboss using the Outside Bevel Option:
Here is Bevel and Emboss using the Inner Bevel Option:
Here is when using the Inner Shadow Style:
Here is when using Pattern Overlay:
Here is Drop Shadow Layer Style with Layer Knocks out Shadow option:
Still hoping someone at Adobe will give this a look.
Hi Jeff
It's been a couple years yet both CS6 and CC show the same issue. It fails many different strange ways yet let me give one simple example (I will post the PSD as well).
In the image below, I created a simple stroke Layer style two different ways. Both have the Transparency Shapes Layer Turned off
1) The lower image uses a Vector Mask determines the edge where the Layer Stroke is applied.
2) The upper image creates the same stroke by creating a Group and applying the Vector Mask and layer Style at the Group Level
You would think they would come out the same yet they don't. This is just the tip of the iceberg of strange effects with this bug.