Creating small proxy jobs becomes unusable the larger the source folder on disk becomes

[Humanclock]

Summary:

a) Creating proxies for the 10 specific files that reside in a disk folder that contains 100 movie files (90 of which are not imported to Premiere) is relatively quick.  

b) Creating proxies for the same 10 specific files that reside in a disk folder that contains 2000 files (1990 of which are not imported to Premiere) will cause the proxy creation process to completely hang and fail.

It should take the same amount of time to create a proxy job for 10 files, no matter how many other files might reside in the source folder on disk.

I'm having this bug on Windows 10, Adobe Premiere 24 and 25.  All my disks have plenty of free space.

Setup to ensure expected operation:

1. Create a file on disk call "mytest" with 10 movies files in it

2. Import these 10 files into Premiere

3. Create proxies for these 10 files, Premeire will quickly Export a Proxy job to AME and AME will start running on those 10 files..

4. Stop AME, clear out any jobs and clear out any proxies it generated.

Now, the bug:

5. Place 500 more movie files into the "mytest" folder on disk, DO NOT import these into Premiere.

6. Do step 3 again with the same 10 files, you'll notice that the Creation of the Proxy job to AME will hang, but after a couple minutes it will slowly start working again.

7. Stop/clear out any jobs in AME again, delete any proxy files it might have created.

8. Place 1500 more movie files into the "mytest" folder on disk, again, DO NOT import these new files into Premeire.

9. Do step 3 again with the same 10 files, Premiere/AME will basically hang for hours, the proxy creation job will need to be cancelled and AME will need a force quit.

WORKAROUND:

1. Split all your source folders on disk into folders of around 150 files. 

2. Import the folder, create proxies from it

3. Keep doing it for each of the folders you created in step 1

Even if it should only care about 10 files in a given folder, Premiere/AME seems to be wanting to do some sort of operation on all files in the disk folder they reside in which causes it to fall over it there is a large quantiy of them.  

This is a problem for me because all my files are broken down by year/month, so some folders can have 2000 files in them.  No other software or Adobe products have problems with these larger folders (eg Lightroom).

[jamieclarke]

Hi @Humanclock - Thanks for submitting your bug report. We need a few more details to try to help with the issue.  Please see, How do I write a bug report?

Can you provide your system specs?

Can you explain step 3 how are you creating proxies?

Can you explain why you would stop the proxy job on step 4, are you setting up watch folders?

[Humanclock]

1. Windows 10, AMD Ryzen 5950x, 128gb RAM. NVIDIA 3080RTX 12gb card. PP v 25.0

For proxy creation, it's just the standard:

a) Highlight the 10 files in the project panel
b) Right click and select "Proxy -> Create"
c) Select half-size, Prores Quicktime

Step 4 isn't 100% necessary, it's just to avoid waiting for the proxy creation to finish to proceed to step 5, I should have specifed this.

Steps 1-4 are just to demonstrate that proxy generation of 10 files (with only 10 files in the source folder on disk) will in fact, work.

However, if the source folder on disk contains hundreds of movie files (that aren't even imported into PP), the bug/issue shows up before any proxy files are actually created by AME.  When PP goes to send the proxy job of 10 files to AME, the progress bar basically never shows up and it sits on the looping progress bar on the "Creating proxy job" (not on my PC to get the exact message).

Another way of testing is just to time how long it takes to send the proxy job of 10 files from PP to AME, when there are only 10 files in the source folder on disk (eg: D:\myTestFiles).  Start adding 100 movie files at a time to D:\myTestFiles (again, do not import these new files to PP).  Now time it again with the same 10 files.  Keep doing this and eventually everything will just hang at the handover from PP to AME on D:\myTestFiles has more than 1000 files in it.

[jamieclarke]

Hi @Humanclock - I would expect that when you send a 1000 files to create proxies in Media Encoder that it would take a very long time.  If you are saying that media encoder has stopped generating proxies then that is a different issue.  

When Premiere Pro is in this state of looping progress bar can you open task manager under processes select Premiere Pro, then right click and select "Create Memory dump". If you can upload it and send me a link then I can have a dev look at it to find out what is going on.

Thanks for your help.

[Humanclock]

No, please read my steps carefully. I am only ever trying to create 10 proxies in any scenario. I should have used "disk directories" so it can't be confused with "folders" that Premiere has.

"b. Creating proxies for the same 10 specific files that reside in a directory ON DISK that contains 2000 files (1990 of which ARE NOT IMPORTED into Premiere) will cause the proxy creation process to completely hang and fail."

Here is a visual description of the problem that might help things:

If my hard drive directory layout is this, Premiere will work great for making these 5 proxies:

d:\myFiles (contains 10 files total)

movie0001.mp4 (Imported to Premiere and making a proxy of)
movie0002.mp4 (Imported to Premiere and making a proxy of)
movie0003.mp4 (Imported to Premiere and making a proxy of)

movie0004.mp4 (Imported to Premiere and making a proxy of)

movie0005.mp4 (Imported to Premiere and making a proxy of)

movie0006.mp4 (not imported into Premiere)

movie0007.mp4 (not imported into Premiere)

movie0008.mp4 (not imported into Premiere)

movie0009.mp4 (not imported into Premiere)
movie0010.mp4 (not imported into Premiere)

Now if my hard drive layout is this, Premiere will fall over and freeze trying to make the same 5 proxies:
d:\myFiles (contains 2000 files total)

movie0001.mp4 (Imported to Premiere and making a proxy of)
movie0002.mp4 (Imported to Premiere and making a proxy of)
movie0003.mp4 (Imported to Premiere and making a proxy of)

movie0004.mp4 (Imported to Premiere and making a proxy of)

movie0005.mp4 (Imported to Premiere and making a proxy of)

movie0006.mp4 (not imported into Premiere)

movie0007.mp4 (not imported into Premiere)

movie0008.mp4 (not imported into Premiere)

movie0009.mp4 (not imported into Premiere)
movie0010.mp4 (not imported into Premiere)
.....(not shown: hundreds of files movie0011.mp4 through movie1999.mp4 not imported into Premiere)

movie2000.mp4 (not imported into Premiere)

[IanB_360]

Hi @Humanclock 

When you choose to create proxies are you actively choosing the Proxies location? In location option are you selecting Next to Original, In Proxy folder? Or are you choosing a custom location for the newly created proxies?

Thank you

Ian

[IanB_360]

@Humanclock Also what is the exact amount of files that are in the folder that you start to notice the proxy media to no longer run as quick as you would expect it to? Are you on HDD,SSD, or NAS? Premiere it self should always function the same and does not change performance on making the proxies. However the OS may have certain limitations depending on how many files are within the specific folder you are looking at. So if your example of 2000 files is the exact number that is causing the process to slow down or hang then it could be that the filesystem/OS is slowing down at managing the files there, and that could slow down Premiere. But it's not that Premiere cares.

Thank you

Ian

[cora08]

I'm using the same Proxy target location each time in both tests (which is a directory on a completely different drive).

It seems like...something...in the Premiere->AME handoff is scanning/analayzing/processing the entire source disk directory of files, even if those other files have never been imported into Premiere and should be ignored entirely.

The only real world analogy I can make is someone tells you to go to a party and pick up Bob and give him a ride to his hotel.

You show up at the party, immediately find Bob and tell him it's time to leave.

However! Instead of leaving, you decide to talk at length to all the other people at the party whom you've never met nor should be talking to, you are talking to them simply just because they are there. The more crowded the party is, the longer you are going to be there before leaving.

[cora08]

It's over a NAS currently, but I'll try it on a local disk also (one directory with 10 files, another directory with 2000 files) to take that out of the equation.

I don't have this problem with any other Adobe software (eg Lightroom or Bridge) on the NAS with directories much, much larger.

[MyerPj]

Ha, ha, ha... excellent analogy.. love it. 🙂

I don't think it's right regards PP, but the analogy stands!

[Humanclock]

(sorry, my wife and I share a computer and was logged into her account)

Also, it starts slowing down when there are >500 files in the directory.  It just takes noticeably longer the more files you add.

[Humanclock]

Yeah, I just don't understand it. The extra files in the source directory have never been imported to Premiere, but for some reason Premiere seems to be "distracted" by them.  I'll try experimenting with smaller vs larger movie vs non-movie files.  I feel like it's scanning them. (and my source directory isn't a watch folder or anything, or does AME "watch" a directory a source file comes out of even if it's not on the list of "watch folders"?) If this were the case I could see it causing things to fall over.