Aerender cannot find .mov files hosted on the server (windows cc2019 & 2020)

[Artists In Motion]

All our farm machines have stopped being able to render comps that contain references to .mov files if the file is on the server. Mov files that are local work without issue.

This only affects aerender. Media encoder, premiere and After effects all load and render the footage without trouble.

It does not matter what codec the file is in, as long as they are .mov extensions they seem to be unable to be found. The error is:

"error: Unable to Render: File Not Found"

Exceptions are h264 and other containers like R3D, which work fine. 

-I have tested with multiple different paths, servers, drive letters. 

-Cleaned and reinstalled creative cloud

-Tested previous versions of AE 2019 (16.0,16.1,16.2,16.3) All have the same trouble..

Anyone have any ideas or have come across this before? 

[Mylenium]

You have not provided any specific info about your systems, server configuration, file paths and so on, so nobody can tell you much. If it worked before, then something significant has chnaged, but with such generic info it's impossible to even guess. Could be anything from a broken CoDec to weird stuff liek your server's network configuration shuffling around and blocking ports.

Mylenium

[Artists In Motion]

Hi Mylenium,

Thanks for jumping right in!

If you have a reason to suspect it's a networking problem, then I can provide any specifications you would like- but we have ruled out every single network connectivity issue we could imagine. Seeing as every single other piece of software on the machines can play, process and render .mov files it is incredibly unlikely to be directly networking related. It is a singular issue affecting only the "aerender.exe" command line function for After effects 2019 and 2020.

But I'll be more thorough for your benefit.

Broken codecs were ruled out through testing about 50 seperately output files from multiple sources - spanning back around 5 years. The codecs in question are 98% prores, with some legacy mixed Photo-jpeg and Animation mixed in. The legacy codecs were tested for completeness but rarely used. No codec, from animation through to Prores will work if it has a .mov extension. 

The network configurations can't "shuffle", but we did check that because hey, crazy stuff happens. The machines all have static IP's. They are connected through standard 1Gbe Nics to a Software defined storage Server (ZFS based). The networking has not had a hiccup or interuption, but we restarted and reconnected them all to be sure. Like I said. Every other piece of software has no problem reaching the files.

The machines all run the same version of Windows 10 - 1903. the same as the workstations which do not currently suffer from this ailment.

The filepaths are network mounts and start with letters of the alpabet. We swopped the network mounts and started with D and worked our way up to R before ruling that out as a cause. If it's a network mount it fails. If it's local, it does not. That's pretty well laid out in the description. I also tried linking to UNC paths e.g. \\server\mount but that had no impact.

Blocking ports.... fair guess though so unlikely it just sounds like a catch all scape goat. But again, checked and dismissed. The farm is isolated and have the firewalls are disabled. And unless aerender is using some special never before known port for accessing the same file as every other Adobe product - it seems incredibly unlikely as a cause. Aerender will have no problem reaching jpgs, exrs, pngs, psds, tgas, mp4s, aep(x) and any other supported file inside the adobe supported file list. Only ".mov files" give a "file not found" error.

I was hoping to get some more useful information in the forums as Adobe official enterprise support's suggestion was to "rename the folder". Which though unorthodox was sadly ineffectual, but strangely still more helpful than your sass. This problem has been very deeply investigated but I have hit a wall where every single diagnostic points to aerender as doing something weird. 

[Martin_Ritter]

You seemed to investigate the problem very well, but still not everything is clear.

My first idea was access control and limits of non elevated cmd, but in this case non of your files would be reachable. If your projects with R3D and H264 render just fine and those files are FOR SURE on the network drive, we can exclude this.

You get the message: "error: Unable to Render: File Not Found" - but where? Is it written in console when running aerender.exe? What is the full output? 

I guess "file not found" does not refer to your *.mov files - since this doesn't make any sense - but your project file or some *.dll needed to encode the *.mov.

You write: "It does not matter what codec the file is in, as long as they are .mov extensions ..." and "The codecs in question are 98% prores..."

Can you please confirm that a working codecs wrapped in a MOV container will lead to this issue? And that a non working file repacked into another container will work.

When was a last time your renderpipeline did work with those files/codecs/containers in question? What happend in between?

How are you executing aerender.exe. Do you launch it from a script (e.g. RenderGarden) or software (e.g. Deadline) or how?

*Martin

[Artists In Motion]

Hi Martin, 

Thanks for jumping in! As to your thoughts.

I've put the full output of the test below. Sorry for the links, I despise walls of logs in forums.

Log from a machine with a GPU
https://pastebin.com/r0TJgPfZ

Log from a CPU only node
https://pastebin.com/rGn8YeHz
Note that the extra warnings in this log has to do with GPU acceleration (which is disabled in the comp) but AE likes to complain about it anyway. (Loadlibrary "n"...) They do not impact the rendering of the comp.

The pertinent line is:
WARNING:After Effects warning: logged one error : "error: Unable to Render: File Not Found P:\2D_Resources\Night_Sky_Blue.mov"

Permissions / ElevatedCMD
I thought it was permissions level issue as well, though it does not work with an elevated command prompt either. The R3D files are next to the prores files on the server.

As an aside, it does not work either with the "ae_render_only_node.txt" file in documents to enable royalty free rendering, or without it (using a full license). Just in case it was a codec licensing issue.

I tried rewrapping an h264 into a .mov container using media encoder but it would not accept the codec as "Transcode to a non-legacy format to continue using the media in Premiere Pro after legacy support has ended." So the only testing I have been able to do in that regard is to use containers that match the codec. h264 (.mp4) works. I tested a slew of quicktime containers this morning. One of each of the supported quicktime codecs in Media encoder.
Here is the result:
(comps tested with only the footage as input, and a jpg sequence as output)
PalDV - Not working
Cineform - Not working
ProRes422 - Not working
Uncompressed_8bit_RGB - Not working

The comp set to render is next to the footage being used.
P:\2D_Resources\ae_render_test_comp.aep
which uses
P:\2D_Resources\Night_Sky_Blue.mov (doesn't work)
P:\2D_Resources\Night_Sky_Blue.mp4 (does work)

This was also tested with the path pointing to the UNC path (\\aim1\production\2D_Resources\Night_Sky_Blue.mov)

as well as the two other servers, Q and R.

R is a synology so shares none of the sharing permissions or hardware of P and Q. Thereby ruling out anything specific to the production storage.

As for timeline.
Our projects tend to swing between comping 3D frames (not using many quicktimes) and finishing shots and grading (using mostly quicktimes). The started appearing at earliest last week Wednesday (4 March 2020), but could have been a problem earlier, our projects just weren't at that stage of production. We can't find any changes to infrastructure that happened to cause it. No systems were updated before the problem, nor was the network not functioning. All other comps have continued to render fine, which meant most compositors did not even notice there was a problem. And because the files will open and work in After effects GUI, there was no early warning sign of a problem.

Since then we have updated all the software, reset the network, restarted the whole storage infrastructure but to no avail. For completeness I set up a testing machine and did a clean install of the creative suite, working upwards from the first version of After effects 2019 (16.0) to see if it was something that broke in an update - they all have the same problem. The versions were tested in isolation, without for example having 2019 and 2020 installed simultaneously just to rule out any cross version scripts or plugins. The comp tested has nothing except a Prores quicktime file in it.

We have worked with these containers for the last 2 years and a lot more since Adobe added native support for outputting Prores on Windows.

The problem first appeared in Deadline (Version: 10.1.4.1 Release), but Deadline as the root cause was dismissed as the problem exists if run in the command line. We wasted about a day rolling back deadline versions before moving on from it. They have had a lot of trouble getting 2020 to work, so it was our first thought.

The command used to test is :
"C:\Program Files\Adobe\Adobe After Effects CC 2019\Support Files\aerender.exe" -project "P:\2D_Resources\aerender_test.aep" -comp "ae_render_test_comp" -s 0 -e 0 -v ERRORS_AND_PROGRESS -close DO_NOT_SAVE_CHANGES -sound OFF

-Notes on the command:
If closely matches the auto-generated deadline command. Tweaked for testing in isolation.
Setting the frame range has no impact on success.. It just speeds up testing running it for a single frame.
-ERRORS_AND_PROGRESS to show as much info as possible.
-DO_NOT_SAVE_CHANGES so the testing doesn't change the source file.
-sound OFF - force of habit. We don't export audio as a rule.

We don't currenlty use rendergarden in production.

If you would like any more information or have ideas for other tests let me know!

Thanks,

Neil

[Artists In Motion · Accepted solution]

Ok. I've found the culprit, and what a doozy of detective work.

This morning one of our motion designers showed up, I warned him about the farm and he informed me he had a comp that was rendering just fine using .mov files.

This was the holy grail. I opened the comp, and versioned up, then kicked off a command line render. Everything good so far.

I replaced the quicktime file with a different one and the comp refused to render...

The real clue came when I then changed it back to the original file (that worked a second ago) and the comp refused to render.

I exported both the working and the non working comps out as XML to see if there is any differences. They are using all the same settings, only diffference is that the quicktime file was reimported from the server in the non working file.

There were a few cosmetic differences in the files but the offending line is this:
Under the item tag for the quicktime file before the path is a <sppc bdata> tag
In the working file it appears as:
<sspc bdata="000000000000000000000000000000000000000000004d4f6f56000000000000114f0000114f000000a800000019000000000000001900000000001900000030010100000000000000034649454c00010000000000000000000000000000000000000000000000000000000000000000000000000000da8e58860000000c0000000100000000000000000001000000010000000000000000001900000000000800000000000000000000000000000000000000000000000000000000ffffffffffffffff00000001000200000000000078727da400000000000000000000"/>

In the re-imported (non-working comp) it appears as:
<sspc bdata="0000000000000000000000000000000000000000000044585633000000000000114f0000114f000000a800000019000000000000001900000000001900000030010100000000000000034649454c00010000000000000000000000000000000000000000000000000000000000000000000000000000da8e58860000000c0000000100000000000000000001000000010000000000000000001900000000000800000000000000000000000000000000000000000000000000000000ffffffffffffffff00000001000200000000000078727da400000000000000000019"/>

If I swop the tags from the working file to the non working file- it works. Hell I don't event have to swop the whole tag the characters between 44 and 48. Changing "44585633" to "4d4f6f56" let's aerender open and render the file.

So, I created a new project, imported a quicktime file and saved it out as xml. Did a search for the <sppc bdata> tag and changed "44585633" to "4d4f6f56". Hit save. Launched it on the farm via the command line and lo and behold. Rendering.

Now here's the really annoying thing.
The workstation I was building the comp on was AE 16.1.1 - leading me to believe this is an AE version thing... spoilers it's not.
I tried the same trick on a machine running AE 16.1.2 - but it didn't work. Simply replacing part of the <sppc bdata> tag broke the file enough that aerender wouldn't open it. The file still loaded in After effects though.

Here is a breakdown of 3 versions of ae with their codes for the same file:

16.1.1
<sspc bdata="00000000000000000000000000000000000000000000445856330000000000000fa00000094c000002ee00000019000000000000001900000000001900000030010100000000000000034649454c00010000000000000000000000000000000000000000000000000000000000000000000008000000d7db66a40000000c0000000100000000000000000001000000010000000000000000001900000000000800000000000000000000000000000000000000000000000000000000ffffffffffffffff0000000100020000000000008028fc0400000000000000000000"/>

16.1.2
<sspc bdata="00000000000000000000000000000000000000000000536f6c690000000000000fa000000a780000000000000001000000000000025800000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000000001000000010000000000000000da87384f0000000c0000000100000000010000000001000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ffffffffffffffff0000000000000000000000000000000001000000000000000000"/>

16.1.3x5
<sspc bdata="000000000000000000000000000000000000000000004d4f6f560000000000000fa00000094c000002ee00000019000000000000001900000000001900000030010100000000000000034649454c00010000000000000000000000000000000000000000000000000000000000000000000008000000d7db66a40000000c0000000100000000000000000001000000010000000000000000001900000000000800000000000000000000000000000000000000000000000000000000ffffffffffffffff0000000100020000000000008028fc0400000000000000000000"/>

Here's the catch. ALL the farm machines write the SAME sspc string - those important middle digits : 4d4f6f56.

But even after updating the workstations to the same version, even with a clean and reinstall of only after effects. They write the digits as: 44585633 - which the farm doesn't accept. Again, I can manually change those digits around and then the farm will be happy but that doesn't get me closer to automated rendering.

I went out on a limb and hoped that the hexadecimal would mean something in normal ascii...(please please please)

44585633 = DXV3
4d4f6f56 = MOoV

...Well that's weird I hear you all say. Yes, that is weird. You might recognize those shortcodes from somewhere. MOov is like OLD SCHOOL quicktime digits. People who use nuke or FFMPEG might have seen them pop up.

But DXV3? I only learned that one two weeks ago. It's the RESOLUME codec, that they use to playback files in real time, think HAP but with a better marketting. The free converter from Resolume , caled alley, is standalone but includes a dvx3 import/export addition that adds that functionality to AE / Premiere. Seems like someone there goofed and it does it a little too aggressively.

That little import/export plugin has been quietly replacing all the metadata of Quicktime files, replacing the friendly "this is a mov" metadata with "this is a dxv3 file". Of resolume, what have you done?

The plugin/codec is installed in Common, which explains why it was persistent through reinstalls of CC (CC Cleaner does not remove third party installs from common)

So in the end, delete the Resolume DxV3 folder from the common\7.0 plugin folder (or alternatively install it across the entire farm) By this point I needed to delete something so au revoir resolume.

What a struggle, and what a lesson in debugging - I'm sure this will be a movie in a few years.

Thanks to Martin for your suggestions, it kept me trying stuff and thanks to Mylennium for showing up. And above all, thank you to the ONE comp that worked despite all the evidence to the contrary. Turns out that was the one motion machine that didn't have the plugin installed. And thanks to HEX to ASCII converters on the internet.

My suggestion to the admins or Adobe staff would be this:
Make the error message describe the problem better.
"File not found" is dead wrong as an error. "File codec "DXV3" not recognized" would have been ideal.

This thread is now closed! Happy friday everyone!

[Martin_Ritter]

Wow... I mean, WOW!

That was great investigation work here - it was joy to read and I'm deeply impressed.

Thank you very much for spending the time in finding out and shareing your discoverys.

Great work!

*Martin

[Flecoq]

Almost a year later, still WOW!

Again, amazing discovery as I was having the same issue seting up a render farm.

On my side, DXV wasn't the problem, but I had HAP and NotchLC codec plugins installed in the common plugin folder. HAP would change the file description to "SDK_" ; notchLC would change it to "NLCP" on PC but it would change it to "PCLN" on MacOS.

This would happen in each version of AE21 (18.0 to 18.4.2).

The fact that this is not just related to the DXV might just mean this is a "normal behaviour" for codecs, and not just some agressive brute force developpers!

So again, moving these codecs from the common folder to AME plugins worked.

[IloveDRMthxdaddy]

this is an amazing guide. thanks so much for circling back to post your findings.

we're going through the same thing, but the bad hex in our case is 

48444558

 which by your guide = HDEX.

any idea what HDEX is?  the file that's fighting us is an .mp4 containing 
AVC Coding, 3840 × 2160, Millions
AAC, Stereo (L R), 48.000 kHz