Multiprocessing/multi machine rendering pipeline

[imeilfx]

One quick question for you guys... or maby not so quick and not preciselly one question

So as we all know AE is not so good with using multiple CPU cores (multithreading/multiprocessing) efficiently for rendering... so we have to use some kind of workarounds to try to compense that. We know that there are some third party pugins to help us with that as f.eg. Render Garden.
But I'm wondering if thyere is different way. So what I'm thinking is:
1. use multiple instances of AErenderer to render composition to image sequence
2. take that sequence and render them to f. eg. ProResWe know that we can do that manually and it work as it should and works well in most cases.

But what about some kind of optimalization?

We can make image sequence with AE and output it to AME watch folder and AME will render for us to anything we want. But we can't set AME to start render after 100% of our sequence is rendered so it wil start at any set time intervals. We can try to predict how lon our AE render will take and set AME to that time but it is not to officient. Or if maby there is some way that I don't know - let me know.

We can also use post render actions in render queue and make AE to: after rendering to image sequence, render that sequence for us to video file ... but as we know and as i said - AE by it self does not work to well with multi cores so any of those renders will use multiprocessing.

But... what if we use multiple AErenderers in that configuration?
So first render goes perfect because if we set output setings as we should (skip existing files) each AErenderer renders separate frame and use more cpu and renders faster but second render to movie file - if we have f. eg. started 6 AErenderer instances, AE will render that movie file 6 times ovewriting (because rendering into movie file we can't check "Skip existing files". As final we will have of course 1 final f. eg. mov file but why wait and use unnecessary CPU ressources?

So, I have two scenarios in mind for that (i mean that trick with post render actions) and I need you to help me figure it out if any of those is possible and how:
a) multiple aerenderers start and renders sequence but after that only one stay for rendering that sequence to movie clip
b) multiple aerenderers start and renders sequence but we can some how tel aerenderers to stop rendering if any of them already started rendering to movie clip

I know that we can start f. eg. 5 aerenderers with only render to sequence in queue and 6 one with sequence and export to movie file but do we have certainty that that 6th renderer wil be the one that will render last frame of our sequence and start render to movie clip? If I'm overthinking it - let me know

[Dave Merchant]

Your "render to images then transcode to video" idea is precisely how RG etc. work. There is no possibility, either within After Effects or using the command-line renderer, to distribute anything other than an image sequence to a network.

AME's Watch Folder engine does not support image sequences as inputs, precisely for the reasons you're talking about. 'Farming' tools like RG run the image sequence into a temporary folder then either send a conventional queue job to AME using the API controls, or transcode using a separate engine (such as FFMPEG). Since the farming tool has created the image sequence job from within a running copy of After Effects in the first place, it can tell when it finishes.

To home-brew something that can run external to AE (e.g. as an OS-level batch script), you just get it to periodically count the number of image files in the AE destination folder. You cannot predict the order that the frames will be written, but if your composition is 1234 frames long then you know it's done when that many images appear.

[imeilfx]

Thank you for your advice.
I know that AME Watch folder is not to good idea because it work as you desribed but in most discussions - AME watch folder is the thing that is being adviced - in my opinion wrongly. That why I mentioned it.
I know how RG and such works and I know that it have to first render to series of frames to then transcode to video but I was wondering if there is some way to use standard or maby not so obvious AE/Adobe tools to (hence Post Render Actions) do that and I just did not yej figured it out or there is none.
As for extrenal script counting frames - that is a great advice ... I will play with that and we will see.

PS.
I'm not new to AE and I'm using that tool for long time and I did use several tools as Render garden (who even remember thing like Nucleo Pro from GridIron) and I can't still figure it out why AE nativelly don't give that kind of capabilities.

[Dave_LaRonde]

imeilfx  wrote

...I can't still figure it out why AE natively don't give that kind of capabilities.

Because, despite Adobe's constant threats to improve performance, the company's more interested in adding sparkly new features to attract new renters.  THAT's the priority.   Then Adobe has them hooked!  If you stop the rental payments, you lose access to the software.  So the rental money keeps flowing.

The improvements to performance move at glacial speeds. 

Hey, Adobe's a business -- it only cares about users if they affect its bottom line.  So far they haven't.

[Dave Merchant]

I was wondering if there is some way to use standard or maby not so obvious AE/Adobe tools to (hence Post Render Actions) do that

That is exactly the example given in the help file. Remember that while you cannot predict the frame order of an image sequence job distributed to multiple computers, you can predict the order that jobs in the queue will be started.

As to why AE doesn't have an obvious tool (such as a "transcode to.." option in the post-render actions menu) it's mostly because AE was never designed to be at the delivery end of a workflow. In 'classical' VFX/post you would always want to render image sequences if the job was heavy enough to warrant network submission, because chances are some frames will need re-doing. The editor, in turn, would want the image sequences because every transcode, however carefully done, messes with the picture.

[Rick Gerard]

If you are concerned with render time and you are primarily doing the final editing in an NLE like Premiere Pro then I would strongly suggest that you invest a hundred bucks in Render Garden. You can render as suitable DI while continuing to use After Effects to produce more product, experience little or no performance issues in AE if you have even a modest system, and RG will even render an H.264 file you can use for client review or posting to social media after the DI is complete. My render times on my laptop (MacBook Pro - latest and loaded) are about 60% faster than using the AME and a typical compositing project that would take over an hour to render on my iMac Pro are kicked out in about 15 to 18 minutes including the h.264 copy to send to a client. The most important improvement to my productivity is that I seldom of ever wait for renders anymore, I just keep working.

[Dave Merchant]

Rick, your entire post reads like spam soaked in snakeoil. It seems you take every opportunity to promote this company and insert links - anyone would think you're getting paid to lie for them.

After Effects may not be the most optimal system in the world, but statements like "experience little or no performance issues in AE if you have even a modest system" are just stupid. RG works by loading many parallel instances of the AERENDER core, each of which cannot possibly be any more efficient than After Effects itself - they are running exactly the same subroutines. If a frame takes 10 seconds to render, it takes 10 seconds no matter what.

The render-to-frames job might complete a bit faster if the parallel instances can utilize a multi-threaded CPU, but it has to consume vastly higher resources to achieve that - including RAM and CPU ticks, so the computer cannot ever run without a significant performance hit. Or do you believe that RenderGarden magically installs a truckload of extra RAM and a Xeon in exchange for your credit card number?

[imeilfx]

Absoutelly right. RG works exactly as you said, and the same like - i guess quite new - plugin called Render Boss or in that matter BG Renderer that if you don't know how and don't wanna to play around with commandline AErenderer can use to execute multipe instances of AERenderer and also still work on another projecy while render goes in background.

[Rick Gerard]

Dave, RG works for me on both a laptop and a desktop. Sure, the fans spin and the room heats up but there is no noticeable decrease in the functionality of AE while it's running because it is using all of the cores that sit idle. Until Adobe completely rewrites the processing routine so After Effects can look at more than one frame at a time it's the best option. I have tried all the rest and RG is a time saver that paid for its self on the first job and continues to save me many hours every week.

I'm still all over Adobe in the background pushing for more efficiency. The OP wanted help with multiprocessing and efficiency. Imeil also wanted suggestions on Multi Machine rendering. Multi-Machine rendering really only works when you render image sequences, so I didn't address the multi-machine question. RG will almost invisibly wake up unused cores and cut render time by about 50% on almost any machine. There's a free 7-day trial so what's the point of slamming a suggestion to try a product and giving my endorsement of a workaround for an Adobe shortcoming. I post workarounds using 3rd party software from everywhere else all the time. Just because I didn't join the slam Adobe for AE's underutilized processor poser chorus I'm not a bad guy.

[imeilfx]

I mentioned multimachine and multicore rendering in one sentence because technically it is the same thing/scheme/workflow. To use multi station rendering you have to use image sequence and for multithreding also - that is the only way for now with AE to render multiple frames at once.

[mardusmanicus]

Have you maybe compared Render Garden vs Render Boss? Because the latter is giving me much grief, rendering empty (black) masks on some frames and also 0 size frames (meanwhile straight AE renders being all correct)...

[imeilfx]

As for Render Garden I'm sure that when I was testing it - it could not render f.eg. Element 3D layers/models, as for other non 3D pugins that I use - it rendered it without any issues and files were ok.

[Tal_Gliks]

Hi mardusmanicus,
Did you find out how to overcome the black frames with Render Garden?
Im haveing the same problem with, driving me crazy...

[imeilfx]

Yes - I know that help document.
And as for using Post Render Actions as I mentionned earlier it works ok when you are using on AE instance. With severa AERenderers not so much (multiple renderings of the same final file).

So first render goes perfect because if we set output setings as we should (skip existing files) each AErenderer renders separate frame and use more cpu and renders faster but second render to movie file - if we have f. eg. started 6 AErenderer instances, AE will render that movie file 6 times ovewriting (because rendering into movie file we can't check "Skip existing files". As final we will have of course 1 final f. eg. mov file but why wait and use unnecessary CPU ressources?

I'm familiar with what AE and I know that AE was never "final" tool for video and I'm not asking for "transcode to.." capability but more about why AE don't have more obvious tool/technique for utilizing multiple CPU cores for rendering.