Skip to main content
imeilfx
imeilfx
Inspiring
January 6, 2019
Question

Multiprocessing/multi machine rendering pipeline

  • January 6, 2019
  • 1 reply
  • 3738 views

One quick question for you guys... or maby not so quick and not preciselly one question

So as we all know AE is not so good with using multiple CPU cores (multithreading/multiprocessing) efficiently for rendering... so we have to use some kind of workarounds to try to compense that. We know that there are some third party pugins to help us with that as f.eg. Render Garden.
But I'm wondering if thyere is different way. So what I'm thinking is:
1. use multiple instances of AErenderer to render composition to image sequence
2. take that sequence and render them to f. eg. ProResWe know that we can do that manually and it work as it should and works well in most cases.

But what about some kind of optimalization?

We can make image sequence with AE and output it to AME watch folder and AME will render for us to anything we want. But we can't set AME to start render after 100% of our sequence is rendered so it wil start at any set time intervals. We can try to predict how lon our AE render will take and set AME to that time but it is not to officient. Or if maby there is some way that I don't know - let me know.

We can also use post render actions in render queue and make AE to: after rendering to image sequence, render that sequence for us to video file ... but as we know and as i said - AE by it self does not work to well with multi cores so any of those renders will use multiprocessing.

But... what if we use multiple AErenderers in that configuration?
So first render goes perfect because if we set output setings as we should (skip existing files) each AErenderer renders separate frame and use more cpu and renders faster but second render to movie file - if we have f. eg. started 6 AErenderer instances, AE will render that movie file 6 times ovewriting (because rendering into movie file we can't check "Skip existing files". As final we will have of course 1 final f. eg. mov file but why wait and use unnecessary CPU ressources?

So, I have two scenarios in mind for that (i mean that trick with post render actions) and I need you to help me figure it out if any of those is possible and how:
a) multiple aerenderers start and renders sequence but after that only one stay for rendering that sequence to movie clip
b) multiple aerenderers start and renders sequence but we can some how tel aerenderers to stop rendering if any of them already started rendering to movie clip

I know that we can start f. eg. 5 aerenderers with only render to sequence in queue and 6 one with sequence and export to movie file but do we have certainty that that 6th renderer wil be the one that will render last frame of our sequence and start render to movie clip? If I'm overthinking it - let me know

This topic has been closed for replies.

1 reply

D
Dave Merchant
Brainiac
January 6, 2019

Your "render to images then transcode to video" idea is precisely how RG etc. work. There is no possibility, either within After Effects or using the command-line renderer, to distribute anything other than an image sequence to a network.

AME's Watch Folder engine does not support image sequences as inputs, precisely for the reasons you're talking about. 'Farming' tools like RG run the image sequence into a temporary folder then either send a conventional queue job to AME using the API controls, or transcode using a separate engine (such as FFMPEG). Since the farming tool has created the image sequence job from within a running copy of After Effects in the first place, it can tell when it finishes.

To home-brew something that can run external to AE (e.g. as an OS-level batch script), you just get it to periodically count the number of image files in the AE destination folder. You cannot predict the order that the frames will be written, but if your composition is 1234 frames long then you know it's done when that many images appear.

    imeilfx
    imeilfxAuthor
    Inspiring
    January 6, 2019

    Thank you for your advice.
    I know that AME Watch folder is not to good idea because it work as you desribed but in most discussions - AME watch folder is the thing that is being adviced - in my opinion wrongly. That why I mentioned it.
    I know how RG and such works and I know that it have to first render to series of frames to then transcode to video but I was wondering if there is some way to use standard or maby not so obvious AE/Adobe tools to (hence Post Render Actions) do that and I just did not yej figured it out or there is none.
    As for extrenal script counting frames - that is a great advice ... I will play with that and we will see.

    PS.
    I'm not new to AE and I'm using that tool for long time and I did use several tools as Render garden (who even remember thing like Nucleo Pro from GridIron) and I can't still figure it out why AE nativelly don't give that kind of capabilities.

    Dave_LaRonde
    Dave_LaRonde
    Inspiring
    January 6, 2019

    imeilfx  wrote

    ...I can't still figure it out why AE natively don't give that kind of capabilities.

    Because, despite Adobe's constant threats to improve performance, the company's more interested in adding sparkly new features to attract new renters.  THAT's the priority.   Then Adobe has them hooked!  If you stop the rental payments, you lose access to the software.  So the rental money keeps flowing.

    The improvements to performance move at glacial speeds. 

    Hey, Adobe's a business -- it only cares about users if they affect its bottom line.  So far they haven't.

      Terms & ConditionsAccessibility statement
      Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices