Sequential frame requirement in Effect Plugins

[Trenton5EC4]

Was wondering if anyone has any samples or ideas regarding effect plugins that require other frames for processing?

From what I understand, it isn't really possible to force AE to render sequentially, which is unfortunate, as in my case the processing requires processing from start->end sequentially. 

Is there a way to tell AE to "try to render this later" if it doesn't have the data it needs? 

Put simply, its a heavily optical flow based effect, that has to take into account the entire sequence, not just neigboring frames. (otherwise I know that could be a simple checkout) 

[shachar carmi]

in general, the "sequential" render scheme doesn't play well into AE's rendering pipeline.

you could:

1. do a series of input checkouts for any arbtirary time regardless of the time of the currently rendered frame (as shown in the "Checkout" sample project. but i guess you already know that.

the donwside being that if the user skips to some distant frame, the checkout time for all previous frames can be insufferable.

2. you could do your processing asynchronously, and therefore sequentially. for exmpale, AE's camera tracker does it's thing on a separate thread and applies the result on the main the thread once done.

3. if during a render you report back to AE that the plugin is out of memory, AE will ignore the result in the output buffer and will not consider that frame rendered. however, that would cause ram previews to stop rendering.

[Trenton5EC4]

Asynchronous processing seems like it would probably be the way to go--
I've familiar with this in AEGP's, but not so much with Effect plugins?
Any suggestions on that?
Also, might it be possible to trigger this processing via button press?

[shachar carmi]

you'd probably want to implement the async processing on an AEGP. however, an effect plugin can also act as an AEGP! you can just call the same aegp registration callbacks on an effect and have it receive idle_hook calls. just note that these calls will not be associated to any specific instance. it will act exactly like a separate AEGP that is unaware of it hosting plugin.

as i already know that you know, interacting with AE can only be done during idle hook on the main thread. aquire frames during these time slots using the render suite, and then pass them to some other thread for processing to reduce the ui thread's blocking to a minimum. you know the drill.

yes, the overall process will be slower than if you blocked the main thread and went to town will frame checkouts, but hey, it's an acceptable compromise between performance and user experience. (IMHO)

as for launching that process with a button, it's very much possible. create a button control and make it supervised. you'll receive a call to your "user changed param" function, from which you can trigger the process.

[Trenton5EC4]

OOOOH! Sorry for coming back to this after so many days-- 
I have an idea now though!

Looking at the warp stabilizer plugin--

I assume that in its render func, it simply outputs the input pixels, while storing frames in sequence data, which are then processed in the background, and when completed, the "Analyze" button unlocks. 

I'm thinking a similar approach could be useful..

Would it be possible to, from the button press, alter the pixels in the effect? How would that be done? 

[shachar carmi]

a couple of thoughts:

1. keeping very large data (such as numerous fully rendered frames) has it's problems... AE makes A LOT of copies of the sequence data which it keeps around for it's own mysterious reasons, so storing 4 HD frames would give 100mb per instance, that can easily take up 1gb of ram for said extra copies.

very large data is better off being wirtten to some temp files. that downside of that is that these files woun't be collected along with the project to be archived or moved to another machine.

2. if you did some background processing and you'd like to apply them to frames once done, you need to invalidate AE's cached results that rely on the previous output your plugin supplied.

you can do that in a couple of ways:

a. during pre-render (which AE regularly calls) put some new value into GuidMixInPtr().

b. during a UI event call, set the FORCE_RERENDER out flag.

c. the most flexible way. have some invisible param on your effect, and change it's value to invalidate the effect's cached results.

[Trenton5EC4]

In that case might the compute cache be a happy medium for storing frames?

[shachar carmi]

good question. the compute cache's logenvity is while AE is open (at the most... i don't know if it might get dropped mid-session). so when the user re-launches AE, the data in the compute cache is gone.

so now it depends on what the user is expecting of your plugin. some users (if not most, IMHO) use the disk cache, so they expect the project to be "as they left it" in terms of responsiveness thatnks to the cache still being there. in that case, losing your compute cache will cause your plug-in to be precieved as "slow" or "ugh that @#$%&* thing..."

on the other hand, if your effect's outputs have been cached by AE's frame cache they'll still be available. but for that to happen, your effect needs to report the same value into the GuidMixInPtr() on pre-render. can it do that without the compute cache available?

on the other other hand... having to relaunch the "process" button on every session... i think that's a big no no for users. they don't know the difference from that and from having to re-analyze wrap stabilizer on each session which we can agree would have been a deal braker.

[Shoggoths_Son]

Are there any good examples of async processing footage with an AEGP? do those plugins typically create a new footage layer? if so how do they insert, or composite frames, into that layer?

[shachar carmi]

well... that very much depends on what user experience you're aiming for.

for example, AE's camera tracker does it's thing async in the background. however, the camera tracker creates tracking info, much like creating keyframes as opposed to processing the image. so user expectations differ.

warp stabilazer does the same, but IS a filter that manipulates the pixels. it stores it's distortion data and then processes each frame with that data on request. (as opposed to storing processed frames and displaying them)

storing very large data (such as multiple rendered frames) in-instance, be it in the sequence data or in an arb param, is VERY problematic in AE, is AE makes multiple copies of these data chunks and creates a huge bloat in ram usage. in such cases, it's probably best to write the frames to disk and either import them into ae, or have the plug-in read and display them directly.

[Shoggoths_Son]

Right i'm looking for two things.
1.) An example of an AEGP that does offline processing and then imports a new footage layer. say one that takes a selected layer, then runs a complex algorithm over it, maybe without even blocking the UI thread. then adds that footage to the project.
2.) An example of a plugin with the warp stabilizers UX. But which stores the frames in the os local version of /tmp/, then when the frames are ready, triggers some invisible param change and renders the frame properly as opposed to having a big *analyzing* loading bar. Ideally this one would not respond to changes in parameters until the user pressed a submit button.

[Shoggoths_Son]

Coming back to this, I made my own example, it harvest any frames and parameters needed on the main thread and passes them to a background thread, updating progress in the UI as it goes, then when it's finished it inserts a png or exr sequence. see golob_plugin/background_job.rs , idle_task.rs, and footage_utils.rs for an idea of how it works.

https://github.com/mobile-bungalow/golobulus-rs

[shachar carmi]

amazing! thanks for sharing your excellent work!