First let me thank the Adobe developers for adding the capability in CS6 for Group Layer Styles. This capability is extremely helpful.
However, when I uncheck the "Transparency Shapes Layer Option" and use the Group Layer/Vector Mask to determine the Group Layer Shape (where Layer Style is applied) then virtually all Layer Style types can form incorrectly under very normal conditions.
Here is a simple example with Transparency Shapes Layer unchecked using Stroke-Outside Layer Style Option (Note: Light blue rectangle is from the View > Show > Layer Edges begin enabled)
Observed Bug Conditions to have Layer Style generated incorrectly 1) Group Layer has Transparency Shapes Layer unchecked 2) Group Layer/Vector Mask is used to define the Group Layer Shape (bug also occurs when there is no Group Layer/Vector Mask as well) 3) The Group Layer Edges (as determined by the contained Layers in the Group) fall within the Group Layer/Vector Mask area.
In the above example, in the top image, the Group Layer Edges are the same as the document edges as set by the "LL & UR Corner Dots" Layer. In the lower image the Group Layer Edges are being determined by the boundaries of the Blue and Green Circles (corner dots Layer turned off).
Here is a table indicating where I have been able to create incorrect formation of Layer Styles (marked by Red) so far.
From a more detailed examination it appears that when Transparency Shapes Layer is turned off, that the Shapes in the contained Layers in the Group are still influencing the Group Layer Shape and specifically, it is being influenced by the the rectangular Group Layer Edges.
I have a more detailed writeup, the PSD file, and example images of all the Layer Style formation failures and would be more than happy to forward that to the appropriate Adobe person.
This could always be pilot error on my part yet this looks like a serious bug under these conditions. I have been able to make 80% of the Layer Styles fail. Workarounds for this issue are quite doable yet inconvenient.