Use color correction (levels) to turn your green layer (swf) black.
Put a white solid behind/below it and then pre-compose the two layers to serve as a matte.
Put the pre-comp layer directly above video layer 2 & set VL2s track matte to "Luma Inverted".
Put video layer 1 at the bottom of the stack and you should have what you want.
Assuming i understand you correctly
Also AE can automatically create animated masks (or do it's best to , anyway) from alpha channels using the auto trace feature under Layer Menu.
But a mask is not what you want. You want a matte.
Of course you could just create your animation in black and white to begin with.
Your mistake was thinking you need "green screen". Nope, matte.