Expressions access to media replacement comps

[Aaron Cobb]

The potential of Media Replacement in MOGRT's for creating responsive design is huge, but it is hamstrung by the inability for expressions to access any information about the media itself. A simple, concrete example is that I am building some MOGRT's for photo treatments, with a simple white frame that needs to match the aspect ratio of the photo the editor drops into it. This would be trivial in After Effects using footage replacement, but with Media Replacement through Essential Properties in AE or in Premiere, when expressions reference the footage they only see the original footage used in the template. This means the editor has to manually adjust the dimensions of the photo frame using fiddly controls in the EG Edit panel.

[DaciaSaenz]

Thanks for this request @Aaron Cobb 

Its something we have definitely heard in the past. There is an innate limitation for your example workflow 

> A simple, concrete example is that I am building some MOGRT's for photo treatments, with a simple white frame that needs to match the aspect ratio of the photo the editor drops into it. 

The Media Replacement Control is bound to the dimensions, duration, and frame rate of the image or video that the control was created with. So there is no way for the frame to match the aspect ratio of the new media even if you had access to it with expressions if you were also trying to use Media Replacement. 

This is set in stone and was foundational to get the feature working to begin with so I don’t see it changing any time soon. But I’ve seen some interesting workarounds by creating a control that has the largest possible dimensions and using alpha / non-alpha information to drive the response of other elements in the design. Maybe that is a possible pathway?

The concept is that the shape looks for the area in where there is no alpha so the white frame could be set up to "see" the boundary where the replaced image and the alpha of the MR Control (with larger dimensions) meet and frame the new new image via expressions. sampleImage() is what you can use to sample the alpha channel. Depending on your design you may get mileage out of it. Fair warning sampleImage() is performance intensive and will slow down your mogrt a fair bit. 


But! We do still need to add more access to the replaced media for other use cases. So thank you for the Feature Request.  

Thank you,

Dacia

[JohnColombo]

Per @DaciaSaenz's response, marking this thread as Acknowledged.

- John, After Effects Engineering Team  

[Aaron Cobb]

Even if the media replacement itself can't adapt its properties to the new media, wouldn't it be possible to access the contents of the layer inside the media replacement comp, given either direct API access to the replacement comp or just a reliable method of predicting the media replacement comp's name?

In case I'm not making myself clear, my presumption is that the behavior when replacing media through the EG panel in Premiere is the same as the behavior exhibited when replacing media through the Essential Properties in After Effects. What happens in AE is a new folder is created in the project called "Media Replacement Comps", and inside that folder a comp is created as a container for the replacement footage. The renderer then, when traversing the network of comps, substitutes references to the original footage with this replacment comp.

If my supposition is correct, then all I should need to access the attributes of the new footage is the ability to construct a valid reference to this media replacement comp. My original hope had been that the source property of the layer that is being replaced would point to the new comp, much the way other EG-fed properties send updated values to expressions that query them, but this doesn't seem to be the case. So that leaves referencing the comps by name. Currently, the name of the media replacement comp isn't predictable without knowing the name of the footage in advance, because the name is generated in part based on the name of the new footage. That, as far as I can tell, is the main obstacle to what I'm trying to accomplish, not the method AE is using to do the media replacment per se.

Does this make sense?

[Aaron Cobb]

In my experiments sampleImage() was unable to access the RGBA values of the replacement footage. It still returned values from the original template footage. Is there something I am missing?

Edit: Okay, I see now that I was wrong about this. sampleImage DOES work if you structure the comp network right.

[DaciaSaenz]

Unfortunately, the way that this works in Premiere is not the same as the way as it works in After Effects. 
As you mentioned, in AE there is a comp that is created that we call the "Embedded Comp" in the Media Replacement folder. 
But in Premiere Pro, this replaced media is added to a hidden nested sequence that is not listed or named in the Project Panel.

The media is accessible to the user by double-clicking on the thumbnail for the Media Replacement control in the EGP. This nested sequence displays the same name as the Media Replacement control. There is no new name created for the replaced media like in AE.

 

There is still work to do in this area for sure. 😞 

This seems to be the best breakdown of using sampleImage() that I've found in my quick search. This user is basing the values that he wants returned on Luminance (or lack of). And then using a linear expression to determine the values of what is visible in other layers based on that.  

• motionscript.com/design-guide/sample-image.html
• https://stackoverflow.com/questions/72324845/after-effects-how-to-change-a-property-of-an-object-bas...

I'm gonna ask the expressions geniuses around here for a bit more guidance but maybe there is something in the links above to get you started. 

Cheers,

Dacia

[Mike-Berry]

Hi -

This is not exactly how media replacement works in Pr. In Pr, there is a sequence created at the fixed dimensions of the media replacement control. Any replacements are inside that sequence, and they can be simple or complex content, just like any sequence. But most importantly, all compositing for this sequence is managed on the Pr side and AE code doesn't know anything about it. It just always gets back a rasterized image at exactly the dimensions of the MR control. So this means there is no expression access possible that looks inside the replacement that happened in Pr.

Mike

[Aaron Cobb]

Okay, I've worked out a method for getting the rectangular bounds of the alpha channel of an arbitrary shape (or array of shapes) with only four sampleImage() calls per frame measured (one call for each edge, top/bottom/left/right. If you don't need the bounds to be dynamic you can get away with only sampling four times on a single frame and then you are done. I've confirmed it works in a MOGRT in Premiere, including on footage that has been transformed using the media replacement controls in the Premiere EG panel, and while I haven't benchmarked it against other methods (namely because this is the first method I've tried) I'm pretty confident this can't be done in less than four calls, so I suspect it's about as fast a method as is possible here.

If anyone is interested I can try to do a writeup and maybe post some .ffx files tomorrow.

[DaciaSaenz]

@Aaron Cobb this is awesome info!! 

Thank you for sharing. I would love an example project if you don't mind 🙏🏽. 

Cheers,

Dacia

[Aaron Cobb]

Here is a project file with examples of the static and dynamic bounding box detection.

I'm demonstrating this using comps with nested text and shape layers here rather than include footage, but the method is relying entirely on the alpha channels of the resulting comps to measure the bounds.

The very brief version is that for each of the four sides of the bounding box I'm using Minimax and Transform effects to isolate the portion of the comp that is on the empty-alpha side of that boundary. Then I'm using a unidirectional box blur to average the alpha values of the pixels across that divide. That alpha value then represents the ratio of the number of pixels on each side of the boundary. Sample that value with sampleImage, and then the location of the boundary can be calculated through very basic arithmetic. Do this for each of the four boundaries (top, bottom, left, right) and you have the bounding box.

[Aaron Cobb]

I experimented some more and came up with a preset for detecting the bounds on a single frame that can be applied to a single layer, no duplicates needed. The preset can be downloaded here.