Let Stereo Mixer effect work in Adjustment Layers too

I need to make a lot of animations where audio is coming from either left or right in different versions.

Was hoping to achieve this by adding a Stereo Mixer effect in an adjustment Layer on top off all, but unfortunately, the effect seems to have no effect when applied in an Adjustment Layer.