I work with very large files what can I do to improve responsiveness

[__Joel__]

Hey everyone I edit multiple 30-70GB files at once and waiting for my computer to repond can be rather painful at times. specs are ryzen5800x, rtx 3080, 32gb ram and my files and scratch disk as a 1tb 4th gen NVME. I was thinking of getting more ram but unsure of how much this would help and was wonding if that is actually the problem or would getting a dedicated scratch disk be more hellpful? Thanks!

[Peru Bob]

Try proxies:

https://helpx.adobe.com/premiere-pro/how-to/proxy-media.html

[__Joel__]

i'll look into that

thanks!

[R Neil Haugen]

Like Peru Bob said, you may been proxies.

What codec are those files? If long-GOP H.264/5 media, you may need to transcode or proxy. You might be able to work with an Nvidia quicksync CPU, 64GB of RAM or better.

[__Joel__]

They are layered mp4 (two audio tracks), would the extra ram help? I just dont wannt buy more and it be relativly similar lol. and by transcode you mean to resize them or something different?

[R Neil Haugen]

Download & install MediaInfo, then drag/drop a file onto its icon on the desktop. It will say exactly what is in that file. Going into MediaInfo's "Tree view" gets a nice listing of the file metadata. Then you know exactly what you're dealing with.

MediaInfo download page

I strongly suspect those are long-GOP files, which are not just "compressed", it's the way they're compressed. It isn't stored as individual frames, but as an actual compressed video frame every 9-60 or so 'frames', and in between are data sets of 1) pixels that have changed since the last real frame (iframe); 2) pixels that will change before the next iframe; 3) or both.

So the computer has to uncompress some frames, and then compute all the frames between that and the next iframe before it can show anything. That's ... a load on CPU, RAM, and everything else.

Some Intel CPUs have hardware included for decompressing long-GOP stuff. I don't think it's typically found on most AMD CPUs, but @RjL190365  is the user here that knows that stuff cold. And maybe, the right CPU/RAM/GPU setup could make that workable even with long-GOP.

Transcoding is converting the files to another codec, like say a ProRes or DNx variant. Depending on the color level ... 4-2-0 or 4-2-2, and data rate of the file, you would choose either say ProRes 422 or 420 probably, or DNx maybe LB or whatever.

Then use those files instead of the original for editing. There is a HUGE problem here, though ... the resultant interframe ProRes or DNx files will be 4-5 times the size of your originals. They'll play back better, but ... wow.

So making proxies ... ProRes quarter-resolution proxies would be my suggestion ... would be probably better. But could still make files as big or bigger than the originals.

You would import the originals, select in the bin, "Make proxies" ... then use the Program monitor controls to switch to proxies for playback/editing. Exports will be from the original media.

Neil

[__Joel__]

Is there a better format to save these files to in the future to make this easier to deal with? As in when I and recording is there a better format out of the gate that still allows multiple audio tracks? Thanks!!!

[R Neil Haugen]

That's always a deep discussion.

What are your capabilities and actual needs for color depth and framesize/rate?

H.264/5 stores media in a very small (comparatively) space. It's incredibly handy for writing to from camera, for deliverables of finished projects, and for storage.

But lousy to edit, do grading or fx with.

But ... better working codecs make much bigger files. Always a debate ... how many huge drives versus what format to store/transport/work in.

Neil

[__Joel__]

I only use these for youtube videos that I like to do on the side so I don't really need a super niche recording type. I just need to be able to edit 150GB+ of files without wanting to bash my head agaist a wall lol (it feels like im spending more time waitng for the computer to load than actually editing which is sad becuase this is a high end rig)

[R Neil Haugen]

I'm still blown away by the massive sizes of those files, even though they're in long-GOP compression. Just ... wow.

@Warren Heaton might have some useful ideas ...

Neil

[__Joel__]

lol some are around 7 hours long recordings at 18kbps and I work with about 24 hours at a time per vid so its a little chonky thats for sure

[Warren Heaton]

@R Neil Haugen @__Joel__ 

The first official hardware recommendation for long-form content is adding more RAM.  With 7-hour source clips, this certainly qualifies as long-form.  I'd try to get from 23GB to 64GB or 128GB of RAM right away.  I'd also upgrade the CPU (higher end with a faster clock speed).

It sounds like the source footage might be a variable frame.  If so, I'd look into if recording a constant frame rate is an option.

If this was something I was working on, I would get a high-capacity standard hard drive (probably 16TB to 20B) and transcode to ProRes422 Proxy at full frame rate and full frame size.  It's likely to make a significant difference in responsiveness by switching to a mezzanine CODEC - even an entry-level one.  It'll also allow for Smart Rendering.

150BG isn't all that large.  For the settings I tend to edit with that's about 2 hours of 1080p ProRes 422 HQ or 45 minutes of 2160p ProRes 422 HQ, but I guess it's all relative.

[__Joel__]

The Cpu for this board doesn't really get much faster it already is 4.7 boost but I did order more ram and we will go from there I suppose

[Warren Heaton]

Being that you're working with MP4 H264/H265, a 13th generation i9 Intel processor should offer a good performance boost.