Long Form Project Optimisation

Hey Theo, I stand by my own frustrations with this kind of thing broadly-speaking but I also have to say, if you're only starting to run into any slow downs at the 1 hour mark, that doesn't seem all that bad to me.  On complex projects, I have in the past gone old-school and worked in "reels" (ie: break the whole film into 20 or 30 min chunks, in separate timelines/projects), and then just put them all together to export, where a few seconds of lag to copy-paste big chunks is a small price. .  

I was working in an 85 min timeline on the doc I cut last summer, and as I said, it was generally pretty much sluggishness-free, but I do think that part of that was becuase the film had shot relatively little footage (as feature docs go), and certainly torwards the end of the process, I did have some beachballing and slowdowns here and there, which didn't surprise me too much.  What was aggravting was how the slowdowns seemed kind of random, in that I couldn't easily pinpoint what circumstance had changed when things suddenly got sluggish.  Sometimes just restarting Premiere would suddenly breathe new life into the performance, which was weird, and other times I had no issue scrubbing around the timeline.

Hey guys, 

@Ryan Fritzsche to answer your question, I have not yet tested working the production exclusively out of an SSD, I could try it to see if the Editshare is causing the issue, but that would be quite shocking to me, considering this is what the product is built for (and was also a solid 15k euro investment). 

@Bruce Bullis here is our production panel is it readable? 

I have organized all the source media from each shooting day into seperate projects for each shooting day, in addition, the source clips are seperate from the Daymaster stringouts of the footage that the editors have been pulling clips from to create the master sequence of the film. With this workflow, everything was running quite smoothly until we hit the 1 hour, 1.5 hour mark (the final film will be around 1.5 hours) in building the master sequence. Then the lead editor started running into problems when cutting the master sequence. 

Also to note, the proxy clips themselves all have a Lumetri Color effect on the source clips for the LUT we are using. 

In general I try to keep almost every video clip used in the edit as a Pro Res Quicktime, as h264 is not ideal. 

As it stands currently the editors are able to work, but we are losing significant amounts of time due to small slowdowns (1-5 second delays for small actions, etc.), and its frustrating to work with overall. 

Regarding sending materials, Im not sure how I would be able to upload everything, it is quite a large project (almost 1 TB of proxies alone). 

I have ran into issues before with Premieres caption workflow when it comes to slowing down a sequence, as we have quite a large amount of subtitles inside the project (half of the spoken dialogue is in Spanish). Could this be a cause for the slowdown? 

Thanks for the input!

Theo

Soudns good; thanks for the additional details!

Hey Bruce, my wife is/was using an OWC 14-port Thunderbol 3 Dock.  I think part of the issue there may be that she had a monitor running off of it too.  The mnonitor is working fine and she has some other drives on it currently, but we weren't doing anything that's unsupported, as far as I know.  

TBH, this sort of thing with big projects getting crazy sluggish goes back years for me in terms of versions.  I just kind of accept it, and like I said, love using Productions as a way to streamline just how much media I have open at any given time, which seems to have greatly reduced how often I encounter it.  

For example, I had a large project about 6 months ago, where I was using a production, but I had one partuclarly large bin of broll footage.  I'm going to guess it was around 500 clips.  And whenever I had that particular project open, everything in my editing sequences would slow down, even if I wasn't accessing any of the footage in that particular project.  If I closed that project, then my edit sequences (which were in a separate project in the production, and mostly under 10 mins in length), would behave mostly fine again.  Is that expected behavior?  

The panel discussion project I was mentioning earlier: I will test re-linking it to the media on my RAID rather than my internal SSD and see if that sluggish behavior resurfaces.  I don't recall what version of Premiere I was on when I started working on it (I know I'm on 24.3 now).  I will circle back if I can reproduce.

Thx Bruce; I hear you, and I like your metaphor.  I guess my only point was that it's easy for us as editors to say "oh my storage is fast enough to play back 6 streams of this or that hi-res footage in RT" but that's not the whole story; those litlte start-n-stops as you describe are obviously vital.  And maybe what I'm wondering is whether a storage system that's fine at playing back X # of streams of high resolution may or may not be good as servicing the all the start-n-stop traffic.  It's out of my depth, but thinking about stuff like the size or number of packets that are being passed thru the 10Gbe?

So to apply those ruminations practically here... @Theo30896725bss4, have you been able to test any particularly offending projects or sequences in a situation where the EditShare is out of the picture?  Do you experience sluggishness at the same point if you test working with the footage off of an internal or local SSD as opposed to the EditShare?  I wouldn't think that SHOULD be a problem, but maybe in fact, it is?

>It just shouldn't be so hard for Premiere to work with large projects that have lots of footage, especially when using productions and proxies.

For most users, it isn't; happy to help further, provided some actionable specifics. 

>There's just something weird about why a multicam sequence that plays like butter before I start editing, suddenly gets sluggish and draggy and beach-ball-spinning once I've made a few dozen edits in a timeline with it

We fixed a problem between 24.1 and 24.2, which could have been the problem you've described. Also, multicam sequences are a great way to make very demanding sequences. 

Also...which "TB dock" did you use (and encounter trouble with), on your wife's production? My CalDigit T4 seems issue-free...

While the Fincher team does rely on custom panels to enforce workflows (and keep metadata up-to-date), I can confirm that they're using the same PPro as everyone else; no custom versions.

Custom panels have zero impact on timeline performance, or data throughput. 

Yeah, I don't know what to say other than to kind of join the complaint.  It just shouldn't be so hard for Premiere to work with large projects that have lots of footage, especially when using productions and proxies.

I tend to favor using 960x540 for proxies, which is a little pixelated if I start doing punch-ins on 4k footage (which I do in a lot of 1080 deliverables for corporate clients).  

I did cut a feature doc last summer on my Mac Studio M1 Ultra, 128 GB of RAM.  I used a Production, worked in UHD the whole way thru, with 540p proxies attached to all the original footage, in multicams, and I was pretty pleased overall with the performance (compared to my 2013 Mac Pro which I only got rid of last year).  Footage was on a 4-bay Pegasus RAID, with proxies on another 8-bay RAID.  For the most part, I could scrub the timeline like it was SD.  But then I'll work on smaller projecst from time to time, like a 3 camera line-cut of a 1 hour panel discussion I was doing recently, and things started getting stupid sluggish about 10 or 15 mins into the sequence.  I don't understand what's going on.  There's just something weird about why a multicam sequence that plays like butter before I start editing, suddenly gets sluggish and draggy and beach-ball-spinning once I've made a few dozen edits in a timeline with it.  But I've observed it happen many times on many different sorts of projects with differnet sorts of media.  For this particular panel project, I found everything improved a lot when I moved my media and proxies off my 8-bay RAID onto the internal SSD, which is weird b/c my RAID is plenty fast enough to support the 3 streams.  I just don't understand why premiere sometimes seems to want to do a ton of extra work to play things that don't seem complicated.  My solution is usually to just segment projects a ton in productions.

One pattern I've noticed is that the amount of footage in a project (or in a Production, the total amount of clips of footage that are open across multiple projects) seems to have an impact, even if my sequence(s) aren't complicated.  Like I've had projects where if I have a project in the production with, hundreds of clips in it, whenever it's open, everything else runs slower.  But as soon as I close it, everything runs better.  So practially-speaking, it seems to help to keep those individual projects within the production as small as possible, and open and close them as I need them.  

One other recent experience I've had: my wife is cutting a feature doc right now on a MacBook Pro M2 Max, in 1080p, with 540p proxies off of 1080p source footage, and was having massively sluggish behavior with some long select sequences.  We shortned sequences and set up a production, etc, and it helped but was still slow.  But once we moved her media drive (a single USB HDD, but still perfectly capable of the throughput she needed) off of an extneral OWC Thunderbolt 3 dock and directly onto one of the TB ports on the Mac, everything transformed, and Premiere runs great now (still using a Production, of course).  Obviously there's something funky about the TB dock there, but that's why I think about the requests to the SAN story that I mentioned earlier; her drive had just fine throughput on the dock when we tested, but perfomance was totally different once the dock was out of the picture, and I think about the # of requests or something.  Maybe I'm combining 2 things that are unrelated, but it feels potentially similar.  Regardless, I think we still had her cache on the internal SSD.

As far as Fincher, I don't know much but I know his team does a lot of customization.  I went to an LAFCPUG event with his post team back when he did the Zodiac film (still somewhat early days of all-digital workflows in motion pictures - maybe in 2007 or 2008?), and I think they might have been using FCP 7 at that time, but point is, they had a dedicated software engineer on the team (or at least, he seemed like one to me) who spent all day writing code or scripting to generate proxies for editing and move around media.  So like.... they were operating with more robust technical resources than the average production, and following some very customized workflows. 

Hey Ryan,

Thanks for the quick reply. We are editing in a HD sequence with HD Pro Res 422 Proxies, that I transcode myself in Davinci Resolve from the original camera material. I just noticed our preview files for the master sequence are set to Pro Res LT instead of Pro Res Proxy, would that make any difference? I doubt it but just curious. The original source camera files are in UHD, but we edit in HD as we will be delivering in HD. With our workflow the original camera clips do not come back into play until turnover to Color anyway. 

We have little to zero nested multicam, However many clips are stacked on top of one another currently as we are still deep in the edit and a few weeks away from picture lock. 

I ran a disk speed test on our Editshare server, it hits around 520 MB/s write and 1020 MB/s read. That should be sufficient. 

Everything I have read from Adobe states a SSD for the media cache (seperate from boot drive/premiere itself) is the most efficient setup. 

It is frustrating to say the least, as I believe I have followed Adobes rules to the letter with this project, and we are still encountering similar performance issues. I am getting close to abandoning Premiere as an option for long form projects all together and just sticking with Avid, but I know David Finchers team has been editing their features exclusively with Premiere since Gone Girl, so there must be something I am missing, its just impossible for me to really say what it is. 

What resolution are your sequences and original source media?  Are your editing sequences, using proxies, working at the same resolution as the original? 

Your description gets at a general frustration I experience often, whenever sequences start to get long, especially if multicam clips are involved.  It seems that whether my media is on SSD vs. HDD RAIDs makes some difference, not b/c the RAIDs are struggling in the slightest to support the data rate, though (everything is plenty fast).  I know this sounds absurd, but it almost feels like Premiere wants to access everything in a given timeline at once, rather than just the upcoming frames that it needs for playback, and the longer the sequence, the more it chokes on doing anything, even if everything is proxy'd or rendered, or whatever.  

I worked at a facility some years back, in my earlier days working with premiere (2015, maybe?) with Mac Pros on a Fiber SAN and remember being told to make sure my media cache was on my internal drive (which I know is good common sense but that's not my point) and I asked why, and what i was told was that the SAN was plenty fast enough to play back our footage (it was XDCam 720, I think) but that what the SAN struggled to support was the number of requests that Premiere makes at any given moment for media cache files.  The cache files are obviously tiny, but still required requests to be accessed, and so the SAN would choke not because of insufficient bandwidth to move large files, but b/c the server couldn't handle the thousands of requests for tiny files.  Now I may be misunderstanding exactly what I'm talking about or mis-extrapolating, but I think about that a lot: that is, the difference between the bandwidth needed to playback a large video file, and the traffic jam of requests to just check on files that may actually require virtually no bandwidth but do require traffic.  It seems to me that there might be some deeply embedded inefficiency in how premiere handles sequences or projects with a lot of footage.  Like, it's checking in on all the files in a project or sequence more often than it seems like it really should need to?  I don't quite know what I'm talking about but I do know that it makes no sense when I'm on a fast RAID, on a fast Mac with tons of RAM, line-cutting a basica mutli-cam sequence, and things just start to get to stupid slow once the sequence starts getting much past 10 or 15 mins in length.