Issues accessing layer contents with alpha in AE

[Anmol5FA8]

I have full HD 1080p comp. One of the layer is a 512x512 image. I have settings to enable alpha values.

out_data->out_flags2 = PF_OutFlag2_PARAM_GROUP_START_COLLAPSED_FLAG |
		PF_OutFlag2_FLOAT_COLOR_AWARE |
		PF_OutFlag2_SUPPORTS_SMART_RENDER |
		PF_OutFlag2_REVEALS_ZERO_ALPHA;

and in PreRender

	req.preserve_rgb_of_zero_alpha = TRUE;

I access the layer with a layer selection.

	PF_ADD_LAYER("Matte Layer", PF_LayerDefault_MYSELF, LAYER_ID);

When this layer contents are accessed, the image always comes up in the top left corner.

ERR(suites.IterateSuite1()->AEGP_IterateGeneric(layer->height, &c_data, CopyDirect));

When I view the image, it is centered, with transparency all around it. However, inside CopyDirect, the image always starts from 0,0

What am I missing?

[Anmol5FA8]

For reference, here is another discussion asking for out of comp area pixels.

is-it-possible-to-access-all-input-pixels-in-smartfx-if-the-input-is-offscreen 

@shacharsuggested to use preserve_rgb_of_zero_alpha to "true" and PF_RenderOutputFlag_RETURNS_EXTRA_PIXELS

What is this PF_RenderOutputFlag_RETURNS_EXTRA_PIXELS ? and where should I use it ?

[Anmol5FA8]

I got one possible answer from a post by @James Whiffin  - here smartfx-expand-layer-bounds-shifter-example-project 

I have the PreRender as suggested, however when I checkout the layer (which is an image smaller than comp size, I get the image shifted to 0,0 and I have no way to access the actual location of the image)

	PF_Err err = PF_Err_NONE;
	PF_RenderRequest req = extra->input->output_request;
	PF_CheckoutResult in_result;
	PF_CheckoutResult matte_result;

	AEFX_CLR_STRUCT(in_result);
	AEFX_CLR_STRUCT(matte_result);

	ERR(extra->cb->checkout_layer(in_data->effect_ref,
		DANCE_INPUT,
		DANCE_INPUT,
		&req,
		in_data->current_time,
		in_data->time_step,
		in_data->time_scale,
		&in_result));
	req.preserve_rgb_of_zero_alpha = TRUE;
	PF_LRect fullLayer = { 0, 0, in_data->width, in_data->height };
	UnionLRect(&fullLayer, &req.rect);
	ERR(extra->cb->checkout_layer(in_data->effect_ref,
		LAYER,
		NUM_PARAMS + 1,
		&req,
		in_data->current_time,
		in_data->time_step,
		in_data->time_scale,
		&matte_result));

	extra->output->flags = PF_RenderOutputFlag_RETURNS_EXTRA_PIXELS;

	UnionLRect(&in_result.result_rect, &extra->output->result_rect);
	UnionLRect(&in_result.max_result_rect, &extra->output->max_result_rect);

[James Whiffin]

Hi Anmol

I'm not sure I understand your situation exactly, but you can get the layer's position in the output buffer by the matte_result rect after calling checkout_layer. Then you can check where that buffer sits inside the output buffer, and see where the output buffer sits inside the composition by looking at the output_world's origin x&y.

[AnmolM]

Thanks @James Whiffin 

As in Transformer.cpp I got

matte_rect.left = -in_data->output_origin_x;
		matte_rect.top = -in_data->output_origin_y;
		matte_rect.bottom = matte_rect.top + matte_worldP->height;
		matte_rect.right = matte_rect.left + matte_worldP->width;

This gives me the correct dimensions of matte_world. In this test case, matte is 150x150. Position 960,540 and Anchor Point 75,75.

When the matte_world is obtained, the position and anchor is not respected. The matte_world always has position 0,0 and anchor point 0,0

You stated :-

a. you can get the layer's position in the output buffer by the matte_result rect after calling checkout_layer.

b. Then you can check where that buffer sits inside the output buffer

c. where the output buffer sits inside the composition by looking at the output_world's origin x&y.

I believe I got a.

How do you do the b. and c. part?

[AnmolM]

Its in the examples. The origin of the matte layer can be deduced

matte_origin = {(A_long)suites.ANSICallbacksSuite1()->fabs(output->extent_hint.right - matte_worldP->extent_hint.right) /2,
                                              (A_long)suites.ANSICallbacksSuite1()->fabs(output->extent_hint.bottom - matte_worldP->extent_hint.bottom) /2};