Project File Size Grows Consuming RAM and Locking Machine

[knbstewart]

Across several productions I've worked on I've come across this issue (working primarily on narrative fiction feature films):

My personal workflow involves me 'versioning up' sequences as I work so that I have past edits readily available if I need to reference something I've done before, or if I get a note to revert back to something I had previously, etc. Typically I'll break the film down into 10-15 'beats' comprised of several scenes each, with each 'beat' having it's own sequence that I am versioning up as I get feedback and make changes. So, as I'm increasing the number of sequences in my project, the proj file size increases proportiontely (always by several MB more than I'd expect as I'm not duplicating any media, just sequences, which seems odd to me, but anyway it makes sense that the project file increases as I'm adding more data to the project).

My issue is: as my project file grows in size it will get to a certain point where 2 operations - opening the project and saving the project - suddenly begin to consume enormous amount of RAM, sometimes asking for upwards of 80GB. This locks up my machine and then the project will become unworkably sluggish until I reboot my machine and go through the process again. My current workaround is just to eventually delete older sequences from my current project until the file size is manageable again. If I need to refer back to older work I then have to dig into older project files. So all this said, here are my questions:

TECHNICAL:

-is this a common or known issue that having a large project file size will destroy your RAM usage (specifically on opening / saving operations) to the extent I'm describing? Or does this sound more like a bug or hardware issue on my end?

WORKFLOW

-This is subjective, but does my workflow of versioning up and keeping multiple sequences in one project sound strange? Do people have best practices for managing large productions while keeping project file sizes low? I personally really prefer to keep as much as I can in one proj file, as jumping around to older files seems counter-intuitive and less organized to me, but is that just what most people do?

Specs:

-I'm operating on a 2021 M1 Max Macbook Pro | 32GB RAM | 10 Core CPU | 24 Core GPU > operating on the latest version of Premiere 23.6.0 (Build 65)

-For what it's worth I have zero tech issues using large quantities of media, exporting renders, handling multiple layers of high res graphics, etc. Only hurdle I've run into with this machine is this RAM usage on large project files. 

[R Neil Haugen]

Are you using the Production mode of Premiere? With the overall Production folder, with subfolders holding separate project file parts?

So the top level Production folder has your folder structure, and the actual projects are in subfolders?

Like Media in one folder sub tree, sequences in a different project file in a different subfolders tree?

[knbstewart]

No, I'm working all in one Proj file. I'll be honest and say I wasn't even aware of 'Productions' until your post and I've just looked it up now. Seems like a nice way to keep file sizes manageable (just by splitting everything up into seperate proj files). 

My bin structure looks more like this (within one proj file) - see screengrab - where my 04 SEQUENCES > SCENES subfolder has most of my versioned sequences. 

If anyone else comes across this and also isn't familiar with "Production mode" I found this video useful:
https://www.youtube.com/watch?v=gV-IEVsMz58

[R Neil Haugen · Accepted solution]

The Productions mode was designed for the larger projects with tons of assests, large organizational structures, and tons of metadata. Like ... for long-form (either docs or fics) or episodic TV/streaming.

The entire project works because of metadata, right? The project file is nothing but metadata ... the sequences are only metadata ... the information about each clip is only metadata ... and on.

In the 'normal' mode, every bit of metadata in the project is loaded into RAM/cache off the bat. Which, with a project with tons of assets & meta to track, is a HUGE amount of data clogging RAM and the on-disc cache system.

In the Production mode, in practical terms, the only meta loaded is what is in the active sequence at this moment. Which means that there is a TON less meta being handled at any one moment. So ... you don't get the lagginess, the other things that slow down 'stand-alone' projects when they get too big.

I'll pass along the documentation on this.

Premiere Pro Productions Introduction

Using Productions in Premiere Pro

Adobe Long-form and Episodic Best Practices Guide 

Jarle’s blog expansion of the pdf Multicam section: Premiere Pro Multicam

[knbstewart]

Neil, you're a legend. Thank you!

[R Neil Haugen]

You're mwelcome, of course!

Productions is a bit of a different beast. In one of their documents, they mentioned conceptualized thinking something like think of your subfolders as top level organization, and the project files inside them as bins.

So you have a Media folder say, with projects for Day 1, Day 2, or maybe Red, Arri, Drone, Stock.

And a separate folder tree for sequences with folders for either Ep 1, Ep 2 ... or 1st act, 2nd act.

That sort of thinking and organization.