Save & AutoSave Lag over NAS

New Here ,
Apr 18, 2018

Copy link to clipboard

Copied

I wonder if someone can help - I'm seeing some pretty long lag times when working in After Effects from my NAS.  The NAS has never really caused me any issues but it's a few years old so I'm considering its replacement, and I don't want to replace it with something that has the same problem, so I'm trying to work out where the bottleneck is.

My current NAS is the Qnap TS-419p ii.  I have been doing some testing today, watching the Qnap Resource Monitor charts while doing a few different things.  The Memory Usage hardly changes at all, no matter what I tried.  CPU usage fluctuated but never seemed to be very high for a prolonged period of time.  Bandwidth Usage was the one that was affected the most by different actions.

The area where I’m seeing a big lag working from the NAS rather than a local drive is when working in After Effects, using a plugin that loads assets from a specified folder, and saves variations of the master AE file out using each set of new assets (the files are then rendered by a few different PCs using AME watch folders).  It takes an age to update the file with the new assets and then save each file, and we’re only talking about a few MB each file.  Up to a minute or so per file, which adds up when you're doing hundreds of variations!  It’s considerably faster if loading and saving the files from / to a local drive.

My initial feeling was it might be network speed, so I did a simple transfer of a single 4GB file, the ‘Packets sent’ went up to about 100MB/s, which was about right and the Ethernet graph in my Windows Task Manager agreed with it.  I’m happy enough with this transfer speed; I think this suggests the network isn’t the bottleneck.

When AE is loading / saving, nothing really changes on the Bandwidth Usage chart, but neither does it seem to have an effect on the NAS's CPU or Memory.  This leads me to think that the hardware of my current NAS may not be the bottleneck, but I don’t yet know what is!  Is anyone aware of any issues with After Effects talking to the file system on a Qnap, or any NAS for that matter?

Views

973

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Save & AutoSave Lag over NAS

New Here ,
Apr 18, 2018

Copy link to clipboard

Copied

I wonder if someone can help - I'm seeing some pretty long lag times when working in After Effects from my NAS.  The NAS has never really caused me any issues but it's a few years old so I'm considering its replacement, and I don't want to replace it with something that has the same problem, so I'm trying to work out where the bottleneck is.

My current NAS is the Qnap TS-419p ii.  I have been doing some testing today, watching the Qnap Resource Monitor charts while doing a few different things.  The Memory Usage hardly changes at all, no matter what I tried.  CPU usage fluctuated but never seemed to be very high for a prolonged period of time.  Bandwidth Usage was the one that was affected the most by different actions.

The area where I’m seeing a big lag working from the NAS rather than a local drive is when working in After Effects, using a plugin that loads assets from a specified folder, and saves variations of the master AE file out using each set of new assets (the files are then rendered by a few different PCs using AME watch folders).  It takes an age to update the file with the new assets and then save each file, and we’re only talking about a few MB each file.  Up to a minute or so per file, which adds up when you're doing hundreds of variations!  It’s considerably faster if loading and saving the files from / to a local drive.

My initial feeling was it might be network speed, so I did a simple transfer of a single 4GB file, the ‘Packets sent’ went up to about 100MB/s, which was about right and the Ethernet graph in my Windows Task Manager agreed with it.  I’m happy enough with this transfer speed; I think this suggests the network isn’t the bottleneck.

When AE is loading / saving, nothing really changes on the Bandwidth Usage chart, but neither does it seem to have an effect on the NAS's CPU or Memory.  This leads me to think that the hardware of my current NAS may not be the bottleneck, but I don’t yet know what is!  Is anyone aware of any issues with After Effects talking to the file system on a Qnap, or any NAS for that matter?

Views

974

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Apr 18, 2018 0
Most Valuable Participant ,
Apr 19, 2018

Copy link to clipboard

Copied

Presumably it's once again the footage integrity/ interpretation check. This stuff seems botched in recent versions of AE due to too many new features having been thrown on top such as automatic sequence gap handling. Using scripts/ plug-ins that manipulate project data could easily explian performance going down the drain because the issue would re-surface with every access to footage. Your only option would be to check al lrelevant preferences and settings and see if it makes a difference.

Mylenium

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
New Here ,
Apr 19, 2018

Copy link to clipboard

Copied

Thank you for your reply.  I do have a feeling it's something within AE, but I'm afraid I don't know what settings I should tweak to see if it has an effect.  These are my current General, Import and Auto-Save settings, along with the settings for the plugin:

Capture 2018-04-19 01.JPG

Capture 2018-04-19 02.JPG

Capture 2018-04-19 03.JPG

Capture 2018-04-19 04.JPG

Can anyone spot anything I've got wrong, or something I could try changing to see what effect it has?  I'm going to try changing the Auto-Save to per minute to see if that has a bearing, but I'm not sure what if anything I can change to affect a footage integrity / interpolation check.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
New Here ,
Apr 19, 2018

Copy link to clipboard

Copied

OK, changing the Auto-Save so it doesn't save when starting the render queue sadly didn't have any effect (not entirely surprising as the render queue isn't actually started; the plugin just saves a new AE file into a specified folder).

I've timed an export of 10 files, firstly with AE reading everything from the NAS, and secondly from a local HDD.  The difference is astonishing - 10 minutes to save the files to the NAS vs 1 minute to the HDD!!

I've recorded a small portion of each in case it helps identify where the lag happens.  (Please excuse the recording from my phone; I didn't want any screen recording software to have a bearing on the result).

NAS:

AE lag 01 - YouTube

HDD:

AE lag 02 - YouTube

As far as I can tell, the actual updating of the assets only takes a second or two, it's clearly the saving of each file that takes ages.  Does anyone have any bright ideas?  Each AE file is only 7.7MB!!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
Most Valuable Participant ,
Apr 19, 2018

Copy link to clipboard

Copied

What are the project settings? Cache settings? Also try and turn off auto-reload of footage. Could well be that it's simply locking up files to keep track of them. Other than that clearing the caches or some project-related issues (coilor space?) could be at play here.

Mylenium

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
New Here ,
Apr 19, 2018

Copy link to clipboard

Copied

Thanks for your further reply.  I've tried clearing the cache (settings below) and turning off auto-reload of footage to no avail.  I'm not sure what you mean about coilor space I'm afraid - could you possibly enlighten me?  There's very little in my project settings that looks like it could have any bearing.

Cache settings (I don't have enough space on my system SSD):

Capture 2018-04-19 05.JPG

Strangely if I do a File>Save or >Save As of the master file which has all footage and references linked to the NAS, it takes a while whether I'm saving the master to the NAS or locally.

Conversely if I do a File>Save or >Save As of the master file which has all footage and references linked locally, it's really quick, whether saving to the NAS or locally.

So it's not specifically saving the file to the NAS that's the issue, it seems to be related to where all the footage is linked from.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
New Here ,
Apr 19, 2018

Copy link to clipboard

Copied

Spent some time this afternoon in an online chat with an Adobe support agent.  He was very helpful, but only in confirming there was an issue and it's not something I'm doing wrong!

The first thing he said was "Premier Pro and AE does not have a full support over NAS."  Say what?!?  So, bearing in mind this software is not just used by individuals, do they expect collaborative users all to work from local drives on each individual machine?!  Surely not!

I let him view my screen and he tried tweaking a couple settings within my AE, but it didn't change anything.  After I showed him the difference in save times between the 2 files, his comments were, "Seems like NAS access, and that is strange, really...  It somehow limits the speed or something...  Save Buffer."  He went on to say, "It seems like it takes a source files when saving, data from NAS, sends to your PC, and then saves it again on NAS."

He recommended I wait a couple weeks as there is a patch coming soon, a new version of AE.  If that doesn't help he suggested contacting the developer team by posting something on their Feature Request / Bug Report Form.

So I'll try again in a couple weeks.  Unless anyone else can shed any light before then?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
LEGEND ,
Apr 19, 2018

Copy link to clipboard

Copied

You know, you can get all Adobe software since CS6 if you're a Creative Cloud renter.  If you don't need fancy new features, why not try using a previous version that actually works? 

I think the Gold Standard in that regard as CC 2014.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2018 0
New Here ,
Apr 20, 2018

Copy link to clipboard

Copied

That's a really good call - I didn't think of that, nor did I realise I could.  I might just go back to a previous version if all else fails, and see if it's any better.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 20, 2018 0
New Here ,
Apr 20, 2018

Copy link to clipboard

Copied

I found the cause!!  Still not sure exactly why it's happening, but I've found a fairly straightforward solution at least.

The project included an image sequence of 5000 or so frames.  AE clearly does some kind of check of every image / video / asset in the project as it saves the file, so it must have been doing this more than 5000 times per save.  Clearly that check must be quicker on a local drive than on a NAS, hence the resulting difference in time.

I'm not sure if it's fair to expect this to happen, or if it should be flagged as a bug, but either way it's a fairly easy fix: pre-render the sequence then the check AE is doing is only one one file rather than thousands.

One thing I tried prior to replacing the image sequence was to place all the assets on the local drive of another PC on the network.  I put them in a shared folder and accessed them via my workstation.  This was a half-way house - just over 3 minutes to save 10 files so considerably longer than from the local drive, but also significantly quicker than from the NAS.  Clearly whatever check AE does will be slowed a bit by working over a network, and possibly slowed further by reading from a NAS.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 20, 2018 0
Contributor ,
Nov 07, 2018

Copy link to clipboard

Copied

I reckon there are save issues across networks in the AE 2019. We work on a 10Gb network and our QNAP has an m.2 SSD installed for cache so our whole setup is rapid. All assets are saved on the network. Saving a 1MB project locally happens in an instant yet on the NAS AE hangs for around 5 seconds before saving.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 07, 2018 1
sabifure LATEST
New Here ,
Jul 08, 2020

Copy link to clipboard

Copied

And could you solve this in any way?
We are working on Synology NAS with m.2 cache on a 10Gb network and saving even small 3MB files takes seconds....
ANy tips are highly appreciated.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 08, 2020 0