After Effects camera + OpenGL matrix

Report · Dec 13, 2013

Hello,

I have a question/problem that has been bugging me for quite some time.

I have a functional AE plugin that has its own 3D OpenGL context/world.

This 3D world is synced to the AE camera like described in this thread: http://forums.adobe.com/thread/570135

After many tries, everything is working like expected now (phew!), but unfortunately, when I add a camera to an AE comp, its position gets initialized to (comp_width/2, comp_height/2, some_negative_z_value).

My OpenGL scene is however centered around (0,0,0). So, to actually see my scene, I have to change the AE camera's position and point of interest both to (0,0,0) as well. Is there any way around it?

I can of course translate my camera (or rather the whole scene) in OpenGL so that it is centered around (comp_width/2, comp_height/2, ???), but that does not work if adaptive resolution/downsampling kicks in, as the width and height then changes, but the coordinates don't.

I don't now if I explained that clearly enough, so feel free to ask me details.

Any input appreciated!

Thanks,

Tom

Report · Dec 13, 2013

hi Tom! welcome to the forum, where everyone arrives beaten to find help

but instead finds other beaten developers.

as for your question, there's actually no right or wrong here. it's just a

matter of what's the lesser of two evils for you, and that depends largely

on the complexity of your existing code.

in AE, {0,0} is not the center of the screen but rather it's upper left

corner.

all inputs you'll receive from AE, from point params, though pixel

positions to the camera's matrix, will act according to that assumption.

so either adapt your code to the "real" location of the objects in the

scene, or normalize all inputs to fit our existing code.

it's up to you.

same is true for downsampling which, like it or not, is a headache you have

to deal with to get a release worthy plug-in in AE.

there are popular approaches to each of these issues, so... ask.

and the camera's default z position... i don't recall the formula by heart,

but it can be found in many expressions and scripts out there.

it's supposed to mimic the comp's default view when there's no camera

around, so the scene doesn't change when you add a camera.

i think the default view is a 35mm fov view... either that, or 50mm... not

sure...

Report · Dec 13, 2013

Cool, thank you very much for your first and fast feedback!

I used the code in the Resizer (getting camera matrix) as a base, which means converting the output matrix from AEGP_GetLayerToWorldXform() to column-order and serializing it in a float array.

On the OpenGL side, I use the following snippet before drawing:

glMatrixMode(GL_PROJECTION);

glLoadIdentity();

gluPerspective(45.0, (GLdouble)widthL / heightL, 0.1, 100.0);

glViewport(0, 0, widthL, heightL);

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glMatrixMode(GL_MODELVIEW);

glLoadMatrixf(glMatrix);

widthL and heightL are the current comp dimensions, glMatrix contains the converted and serialized camera matrix from above.

Now as I said, I am a bit lost concerning proper handling of the downsampling.

I can of course read out the current downsampling factors in in_data->downsample_x and _y. Every downsampled layer has apparently also already adjusted its width and height.

So I added a "downsampling-aware" translation to XY-space in my world like this:

glTranslatef(-widthL*0.5f*downsample_x, -heightL*0.5f*downsample_y, 0.0); // downsample_x and _y here already have nominator/denominator calculated

This however did not work, as the camera did not show my scene when downsampled.

By debugging I found out that in the camera matrix, it was still using the coordinates as if no downsampling took place.

Here is an example what I am seeing:

- new comp, dimensions: 1920x1080

- new solid layer, same dimensions as comp

- new camera (35mm): default position is 960,540,-800 (=center of comp)

- add my plugin to solid layer

If downsample factor = 1, everything works when I translate OpenGL world by half of the comp size.

If downsample factor < 1 (0.5, 0.333, 0.25, etc.), in the camera transform matrix the translation values are still at 960,540,-800, which means to me that when the transformation matrix is calculated, the camera position does not take the downsampling into consideration (correct me if I am wrong).

To be honest, at the moment I don't quite know what the right question to ask is 🙂

I am still trying to figure out what approach I should use when I just want to use the AE camera controls in my OpenGL plugin to move around.

Report · Dec 13, 2013

i don't know the detail of your plug-in's rendering logic, and how

customizable it is to some curve balls, but downsampling can also be

uneven. (meaning different x and y downsample values)

downsample can be viewed much like pixel aspect ratio (which you also

should support!).

a circle in your "logical world" may be elliptical in the render buffer.

you just can't for a few reasons, treat the incoming pixel buffer as your

guide to actual world dimension.

some render processes can be adapted to render directly to these

"non-real-world-dimensional-correlation" buffers, and some processes are

much better off rendering to a normal, weirdness free buffer, and then

translating it in 2D to the actual, distorted buffer.

what sort of rendering do you do?

shed some light (if you can), perhaps i could come up with more specific

suggestions.

Report · Dec 13, 2013

Yes, I know that downsampling can be uneven (so I had two variables downsample_x and downsample_y in my example).

Concerning the rendering: for simplicity's sake, let's consider old school OpenGL 1.x code (fixed pipeline, no shaders, no vertex buffers) just for getting a cube drawn on screen (the standard textbook example):

// setting some defaults

glEnable(GL_DEPTH_TEST);

glDepthMask(GL_TRUE);

glDepthFunc(GL_LEQUAL);

glDepthRange(0.0f, 1.0f);

glDisable(GL_CULL_FACE);

glClearColor(0.f, 0.f, 0.f, 1.0f);

glDisable(GL_LIGHTING);

glDisable(GL_TEXTURE_2D);

// projection matrix

glMatrixMode(GL_PROJECTION);

glLoadIdentity();

gluPerspective(60.0, (GLdouble)widthL / heightL, 0.1, 100.0);

glViewport(0, 0, widthL, heightL);

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

// modelview matrix

glMatrixMode(GL_MODELVIEW);

glLoadMatrixf(glMatrix); // converted camera matrix from After Effects

glTranslatef(0.0, 0.0, -15.0);

glBegin(GL_QUADS); // Draw A Quad

glColor3f(0.0f,1.0f,0.0f); // Set The Color To Green

glVertex3f( 1.0f, 1.0f,-1.0f); // Top Right Of The Quad (Top)

glVertex3f(-1.0f, 1.0f,-1.0f); // Top Left Of The Quad (Top)

glVertex3f(-1.0f, 1.0f, 1.0f); // Bottom Left Of The Quad (Top)

glVertex3f( 1.0f, 1.0f, 1.0f); // Bottom Right Of The Quad (Top)

glColor3f(1.0f,0.5f,0.0f); // Set The Color To Orange

glVertex3f( 1.0f,-1.0f, 1.0f); // Top Right Of The Quad (Bottom)

glVertex3f(-1.0f,-1.0f, 1.0f); // Top Left Of The Quad (Bottom)

glVertex3f(-1.0f,-1.0f,-1.0f); // Bottom Left Of The Quad (Bottom)

glVertex3f( 1.0f,-1.0f,-1.0f); // Bottom Right Of The Quad (Bottom)

glColor3f(1.0f,0.0f,0.0f); // Set The Color To Red

glVertex3f( 1.0f, 1.0f, 1.0f); // Top Right Of The Quad (Front)

glVertex3f(-1.0f, 1.0f, 1.0f); // Top Left Of The Quad (Front)

glVertex3f(-1.0f,-1.0f, 1.0f); // Bottom Left Of The Quad (Front)

glVertex3f( 1.0f,-1.0f, 1.0f); // Bottom Right Of The Quad (Front)

glColor3f(1.0f,1.0f,0.0f); // Set The Color To Yellow

glVertex3f( 1.0f,-1.0f,-1.0f); // Top Right Of The Quad (Back)

glVertex3f(-1.0f,-1.0f,-1.0f); // Top Left Of The Quad (Back)

glVertex3f(-1.0f, 1.0f,-1.0f); // Bottom Left Of The Quad (Back)

glVertex3f( 1.0f, 1.0f,-1.0f); // Bottom Right Of The Quad (Back)

glColor3f(0.0f,0.0f,1.0f); // Set The Color To Blue

glVertex3f(-1.0f, 1.0f, 1.0f); // Top Right Of The Quad (Left)

glVertex3f(-1.0f, 1.0f,-1.0f); // Top Left Of The Quad (Left)

glVertex3f(-1.0f,-1.0f,-1.0f); // Bottom Left Of The Quad (Left)

glVertex3f(-1.0f,-1.0f, 1.0f); // Bottom Right Of The Quad (Left)

glColor3f(1.0f,0.0f,1.0f); // Set The Color To Violet

glVertex3f( 1.0f, 1.0f,-1.0f); // Top Right Of The Quad (Right)

glVertex3f( 1.0f, 1.0f, 1.0f); // Top Left Of The Quad (Right)

glVertex3f( 1.0f,-1.0f, 1.0f); // Bottom Left Of The Quad (Right)

glVertex3f( 1.0f,-1.0f,-1.0f); // Bottom Right Of The Quad (Right)

glEnd();

What do I need to consider to adapt this to the After Effects environment?

I know about downsampling and pixel aspect ratio, but I am not sure where they figure into the OpenGL code. The only "interface" between AE and OpenGL so far that I found was the modelview matrix which is being calculated from the comp's camera matrix.

Any hint appreciated 🙂

Tom

Report · Dec 13, 2013

Also, I could not yet find a resource/script/post about calculating the default camera z position from x and y (or maybe other parameters). Maybe I did also use the wrong search term. Can anyone tell me where I might find this equation? Thanks!

Report · Dec 14, 2013

well... looking at the documentation i found this:

AEGP_GetDefaultCameraDistanceToImagePlane()

the manual calculation (also in the docs! i was surprised!)

Θ = 1/2 field of view

tan(Θ) = 1/2 composition height / focal length

focal length = 2 tan(Θ) / composition height

Report · Dec 14, 2013

Thanks! I tried that out (getting the comp handle and then calling that function). For my current comp (1920x1080) and a 15mm camera, it always returns 2666 as default distance.

A new camera however always seems to default its z-value to "-800" in that comp, so I am not sure if/what went wrong or if I need to consider anything else...

Report · Dec 14, 2013

when there's no camera layer in the comp, the comp is rendered with a 50mm

camera.

when you create a new camera, the most recently used camera setting (as

changed by the user) are used, and not necessarily the default settings.

Report · Dec 14, 2013

Thanks, that was a misunderstanding from my side. So, I only need the default distance if no actual camera is present in the comp (and the default 50mm is used internally). For anything else, I can just grab the focal length of the actual comp cam, as the camera's position is apparently initialized to be (comp_width/2, comp_height/2, -focal_length). At least in my tests, focal_length was returned as "800" and that corresponds to the default z position of "-800" for the camera.

Sorry for asking so many questions Shachar, but you have been extremely helpful!
I also kind of managed to get around the matrix/transformation issues I had, most of the stuff is working now, only problem is the objects are all drawn in black

Report · Dec 14, 2013

right on!

now i',m not really an openGL guy, so you might want to ping someone from

another openGL thread, or post a link to this discussion there.

that will draw some more informed people's attention to this issue.