Hopefully a simple one - how does my effect ignore masks that are applied to its layer?
I thought it was something like ensuring PF_OutFlag2_REVEALS_ZERO_ALPHA was set during Global Setup, and extra->input->output_request.preserve_rgb_of_zero_alpha = true during Pre Render (my effect is SmartFX) but applying a mask still cuts it off.
I also tried hacking the output->result_rect to be the same size as the in_data->width & height values (just to see what happened). It worked, in that my effect wasn't affected by masks any more, but did funky things with the background image (scaling it depending on the bounding box of the mask). I know that's the wrong way to do it, but wanted to see if it was at least on the correct path (no pun intended!).
It must be possible - the Checkerboard effect ignores masks with the default blend mode selected, although that might not be a SmartFX plugin.
PF_CHECKOUT_PARAM() with the deafult param 0, should give you layer's buffer without any maskas or effects.
as far as i know, there's no way to get the input buffer without masks but with previous effects applied, which makes sense since maks are processed before effects, and previous effects are not guaranteed to keep the rgb data of 0 alpha pixels.