According to some YouTube videos and Adobe's documentation, this option, which is checked by default, it is designed to limit the application of Effects to the opaque area of a layer only. See this example. All I have, here, is a black circle created on a new layer, filled with a Pattern overlay layer effect :
The effect is only found inside the black circle -- logical and as expected. But if, for some reason, I choose to uncheck the Tranparency Shape Layer option, all of a sudden, the Pattern overlay is applied to the entire layer. Which is logical and intuitive to me.
Now some other YouTube videos demonstrate that this option is found to also have an effect on some of the more recent Blend modes?
-- I probably would never use this option in my work but I confess I'm puzzled by its "function"?