Checking out image layer works but pre-comp layer does not

[Richard Rosenman]

Hi gang;

I am pulling my hair out trying to solve this seemingly simple issue.

My plugin checks out a layer, and transforms it elsewhere using Transform_world. If the layer I check out is an imported image (circle.png) or an After Effects shape, it works correctly as expected. However, if I make a new comp, draw a circle, and then checkout that pre-comp as my layer, it does not. It simply outputs black.

Please take a look at my code:

First I check out my layer:

	AEFX_CLR_STRUCT(checkoutLayer);
	ERR(PF_CHECKOUT_PARAM(in_data,
		SKELETON_PLAYER,
		(in_data->current_time* in_data->time_step),
		in_data->time_step,
		in_data->time_scale,
		&checkoutLayer));

	ERR(PF_ABORT(in_data));

If the checkout layer has data (if checkoutLayer.u.ld) then I create a world for it:

ERR(suites.WorldSuite1()->new_world(in_data->effect_ref,
checkoutLayer.u.ld.width,
checkoutLayer.u.ld.height,
flags,
&particle_world));

ERR(PF_ABORT(in_data));

Then I copy the checkout layer to my newly created world:

ERR(suites.WorldTransformSuite1()->copy_hq(in_data->effect_ref,
&checkoutLayer.u.ld,
&particle_world,
NULL,
NULL));

ERR(PF_ABORT(in_data));

And finally, after setting up my transform_world matrices, calculating them, and other Transform_world setups, I output:

PF_Rect dest;
dest.bottom = in_data->height;
dest.left = 0;
dest.right = in_data->width;
dest.top = 0;

err = suites.WorldTransformSuite1()->transform_world(NULL,
quality,
PF_MF_Alpha_STRAIGHT,
PF_Field_FRAME,
&particle_world,
&mode,
NULL,
&mat1,
1L,
true,
&dest,
output);

And after that destroy the world:

ERR(PF_ABORT(in_data));

ERR2(suites.WorldSuite1()->dispose_world(in_data->effect_ref, &particle_world));

So, like I said, this works perfectly well if my layer is an image layer that has been imported. But if the layer is a comp that I made with a shape, it does not.

I should also say that if I create a shape in my main comp, and use that as my checkout layer, it also works correctly. So it is specifically pre-comps that aren't working. I also tried enabling and disabling the collapse transformations to see if that would make any difference.

Any ideas what I'm doing wrong?

Thanks,

-Rich

[James Whiffin]

Hi Rich

I would try copying the entire buffer you received from CHECKOUT_PARAM to the output to ensure you're getting what you expect. If it is correct, then you know that part of the code is fine and it's going wrong afterwards. CHECKOUT_PARAM for layers does not offer much flexibility in which regions of the layer you need, it is likely giving you an entire comp sized buffer when the shape itself may be very small. It's recommended to use smartFX to gain better performance and control over exactly which pixels you request.

You can also transform the checked out layer instead of copying it to an intermediate effect world, but I suspect you did that for another reason. 

[Richard Rosenman]

Hi James!

As always, thanks for helping me with your replies.

"it is likely giving you an entire comp sized buffer when the shape itself may be very small."

Is seems you are correct. I checked the debugger and it does indeed provide the comp size, not the layer size. Sheesh! That's a royal pain in the butt to convert the plugin to a SmartFX just to have this kind of flexibility which you'd assume would be quite basic (getting a checkout layer's resolution).

Bottom line is there's no easy way to do this if not creaating a SmartFX plugin? To convert it, would you recommend I start from the SmartyPants demo and go from there? And while I have you, if I am able to convert it, where would I find more info about retrieving the resolution of a checked out layer?

Thanks James!

-Rich

P.S. I found this post by you - it looks like this describes how to retrieve a checkout layer's true resolution, given it's a SmartFX plugin? https://community.adobe.com/t5/after-effects-discussions/find-3d-collapsed-layer-s-efficient-bounds-...

[James Whiffin]

No worries!

SmartFX is a bit to wrap your head around but once it's familiar it's not much extra work for the extra performance and being able to do 32bpc. Smarty Pants is the starting point, it demonstrates a fair few things you likely won't need. Using smart checkout, vector layers will be given at their actual size (with the dimensions being the ceil of the floating point values). You can individually expand/shrink the layers you request as well as the output buffer size. 

The thread linked is me trying to get the exact efficient size if they are 3D which I still haven't been able to work out :'(

[Richard Rosenman]

Hey James;

Perhaps I have the liberty for one last followup to this thread. I've successfully converted my plugin to a SmartFX one.

Referencing your thread I mentioned above (https://community.adobe.com/t5/after-effects-discussions/find-3d-collapsed-layer-s-efficient-bounds-...) you specify how to check in the input layer in a SmartFX plugin and that the in_result.result_rect contains the true dimensions of the input layer, which is what I'm after. However, I see that this must be done in the PreRender stage.

My question is: how do you then access this data in the SmartRender stage? I have looked through the SDK examples and I have yet to find an example of this. The checkout plugin example isn't a SmartFX so it's not helpful.

If there is any example that exists and shows me how to do this, perhaps I missed it. In that case, please let me know which one it is and I'll check  it out.

Thanks,

-Rich

[James Whiffin]

Hi Rich

You can use Pre-Render data. The SDK_Invert_ProcAmp sample has sample code demonstrating this:

extraP->output->pre_render_data = infoP;

extraP->output->delete_pre_render_data_func = DisposePreRenderData;