Creating an effect plugin for a new blend mode

[jamesastro]

Hey Folks. I'm developing an effect plugin that will implement a new blend mode. Here is how it will work from the user's point of view:

1) Add a background image layer to the composition.

2) Add a foreground image layer.

3) Set the "adjustment layer" flag on the foreground layer.

4) Apply my affect to the foreground layer.

Here's the problem: In order to blend the foreground layer with the pixels behind it, I need to get the pixels for the foreground layer, and the pixels behind that layer. Getting the pixels behind the layer is easy. I simply call checkout_layer_pixels() on the layer that my effect is applied to, and since it's an adjustment layer, it gives me the pixels behind it. However, I don't know how to get the original foreground pixels in my layer.

How would I get the foreground pixels?

[shachar carmi]

perhaps an layer param set to "self" as a default would fetch the layer's original pixels? if that works for you, you can make that param invisible so it would appear to happen automagically.

you'll probably also need to use the old checkout meachnism instead of the new checkout_layer_pixels(), as they differ in the returned result. the old one fetches the original layer pixels pre masks and effects, while the new one fetches post masks and effects.

[jamesastro]

You mentioned this: "You'll probably also need to use the old checkout meachnism instead of the new checkout_layer_pixels()."

I'm not familar with this old checkout system, and I can't find anything about it in the SDK documentation. Can you point me in the right direction? What function should I call?

Thanks!

[shachar carmi]

you need to checkout a layer param.

PF_ParamDef checkout;
ERR(PF_CHECKOUT_PARAM( in_data,
index_of_layer_param,

in_data->current_time,
in_data->time_step,
in_data->time_scale,
&checkout));

the fetched pixels will be at checkout.u.ld

[jamesastro]

Darn. That approach doesn't work. Adding another layer parameter to "self" doesn't return the original pixels. Since self is an adjustment layer, it returns the pixels that are behind the layer.

[shachar carmi · Accepted solution]

ok, we'll need to escalate.

you can force AE to give you any item's image buffer by asking for it directly.

1. use AEGP_GetLayerSourceItem() to get your layer's itemH.

2. use AEGP_NewFromItem() to get a AEGP_RenderOptionsH for that item.

3. use AEGP_RenderAndCheckoutFrame() to get a receipt.

4. AEGP_GetReceiptWorld() will fetch you the pixels for the receipt.

it's a bit more wasteful that regular layer checkouts, since AE can't tell in advance you need some frame and therefore keep it in cache. however, it's not THAT a big a difference...

[jamesastro]

This works great. Thanks so much!

[shachar carmi]

you're welcome. 🙂