Using After Effects to generate keyframes that measure the luminosity of video

[johnm27994604]

It's relatively easy, using Convert Audio to Keyframes, to use audio amplitude to drive changes in video: opacity, position, scale, etc.  But what if I want to use features of the video image, e.g., luminosity, to drive audio amplitude or pitch?  Is there a way to do this that's native to AE?  Or a plug-in?  Thanks! 

[Mrtn Ritter]

You can use expression and sampleImage() to get [r,g,b,a] values of a given point or area. Then you can use those values to drive, whatever you want.

https://helpx.adobe.com/after-effects/using/expression-language-reference.html

*Martin

[Rick Gerard]

Pitch can be controlled with Time Remapping. You can directly control Levels or you can use the audio effects to control the audio. You won't get the amount of control you would with an actual audio app like Audition, but you can certainly manipulate the audio.

sampleImage() can look at a single-pixel or a group of pixels. If you try and average the luminosity of an entire shot it's going to slow down to the point of being unusable. There are ways to average the luminosity of an entire frame using a combination of effects and then you would only have to sample a few pixels. 

The sampleImage method generates values between 0 and 1 in a four-digit array: [R, G, B, A] 

If you converted a layer to black and white, added a few effects to average the luminance, you could use something like this to drive audio levels:

smpl = thisComp.layer.("Luma Sample");
point = [thisComp.width, thisComp.height]/2;
t = smpl.sampleImage(point. radius = [10, 10])[0];
newVal = linear(t, 0, 1, -20, + 20);
value + newVal

There may be some typos in that expression. I am answering on my iPad, but the idea is to sample a modified footage layer (Luma Sample) that you don't see in the render, probably blurred, pick a point on the layer to sample (I picked comp center), then run an interpolation where the sample value (t) which can run from 0 to 1 for the red channel increases or decreases the current property value by as much as 20 units. If the sampled value of red is .5 there would be no change. If the sampled value is 0 the value would go down by 20 and if it was 1 it would go up by 20.

If I knew exactly what you were trying to do and had a screenshot of the video or better yet a copy of the video you were trying to sample I could be more specific.

[johnm27994604]

Rick, thanks so much for this detailed reply.  I'll give this a shot as soon as I can parse the script you've shared.  I'm new to AE so even something apparently simple like working with expressions is a challenge.  I do get where you're going, conceptually.  I'll take a peice of footage and export it as a 1x1 desaturated mosaic, which will give me a clip with basic luma info.  I'll then try to use this sampleImage expression.   Thanks again!

[johnm27994604]

Ok, yes, so here's perhaps an assignment; no urgency, no obligation.  A video and a tone are here: https://drive.google.com/drive/folders/1GnIWNSVPsrH1xAziq-33m3JnxV36AjXX?usp=sharing

Can you create an AE project where the brightness of this video changes the volume of this sound.  And perhaps where the color (maybe using warmth and coolness, so orange to blue) shifts the pitch?  That's what I want to do.  Thanks!