CPU and RAM maxing out during rendering, rendering before export causes an error or crashes

[arcbound34519447d6ho]

I'm doing some mild edits to Twitch VODs before putting them up on YouTube. It's always been slow because of how long the videos are, typically right around 4 hours, but it wasn't until the last few that it's been a problem. Up until the current one, I was able to solve the problem by clearing the media cache, but now that's not helping. If I try to do a direct export, it runs fine up until about 25%, then it steadily slows down until the estimated time remaining starts increasing, all while using anywhere from 90% to 100% of my CPU and RAM. Sometimes it uses upwards of about 45% of my GPU, sometimes it doesn't, I can never predict when it will do so, and using it doesn't seem to help. If I try to render before exporting, it throws a frame creation error at me. If I just export, it keeps going until it crashes, and the crash is so bad that it very nearly takes the entire system down with it. I usually have to go into task manager and restart Windows Explorer. I've tried doing this in an entirely new project, I've tried cleaning the media cache, I've reinstalled Premiere Pro, and I don't know what to do at this point. I've been scrounging around looking at videos and forum threads for new things to try, and so far nothing has helped. Thoughts?

[arcbound34519447d6ho]

I don't know if it's important, but I should add that when a render fails without crashing the software, it won't release the RAM it's used. In addition, if I render before exporting, the render files are beyond ridiculous in size. The final files for these VODs have ranged from 20 to 45 GB, and this last rendered file hit 88.7 GB when the program crashed, and it wasn't even halfway done yet. The program also apparently isn't cleaning up after itself. The drive it's using to store these started out today at 1.1 - ish TB free, now it's down to about 850 GB, so it's just EATING hard drive space with all of these failed attempts.

[R Neil Haugen]

Premiere needs 4-5 times the expected file size available while doing render or export work. Do you have adequate free space?

[arcbound34519447d6ho]

I've got more than 50 times the expected final file size as free space, and that's highballing the final file size. 

[arcbound34519447d6ho]

I don't know why I put 50 there, should be 20. I have more than 20 times the expected final file size as free space. That's still more than ample free space.

[R Neil Haugen]

What format/codec are those VOB files?

[arcbound34519447d6ho]

They're all .mp4 files.

[R Neil Haugen]

Long-GOP?

[arcbound34519447d6ho]

I have no idea what that means.

[R Neil Haugen]

Long-GOP is a process that is used in many cameras, especially phones & drones, most prosumer cams, and most screen recording apps.

It's great for quickly mashing the data down to a much lower level of stuff to write to card or disk. Because it's based (for all camera work) on having specialized chips in the device to do the encoding process.

It has complete actual video frames called iframes only every so often. Between 9 and 30 was the old standard, but now some drones push this up above 100 frames apart. In between are datasets called p and b frames. And they are datasets, not images, of 1) the pixels that have changed since the last iframe or 2) the pixels that will change before the next iframe or 3) both.

And some drones and other devices use "partial iframes" to push the numbers way past 100 frames between complete actual video images.

So the computer needs to create the entire stretch of frames between up to 200 frames, do all the decoding/decompressing for all those frames, and store to RAM/cache ... in order to display the next frame sequentially.

It's not too hard on simple players, because they don't put any load on the system. But an NLE, like Premiere, just in running, puts a load on the system. Now ... to grab a few seconds of long-GOP clip here and there, it's doing a TON of decoding/decompressing/writing to RAM/cache before it can display a sequence.

Or, in your case, do an export.

So the next question is, what is your CPU? Some Intel and Arc CPUs come with hardware to help with long-GOP work. Some don't. AMD CPUs ... fuhgedaboudit. (I run a 24 core Ryzen 3960x, but avoid long-GOP like the plague it is ...)

[arcbound34519447d6ho]

These VODs are downloaded from Twitch. How Twitch gets them into .mp4 format, I have no idea. All I know is that it's in .mp4. As for CPU, I have an AMD Ryzen 7 3700X. 

[arcbound34519447d6ho]

I'm fully confident the CPU isn't the issue though, as I've rendered quite a few videos at this point with all the same hardware. All of the VODs I've uploaded to YouTube have gone through this process with Premiere Pro and the same setup. 

[MyerPj]

You can use the free MediaInfo program to get info on your clips. Get the program from the link below and using the 'Tree' view post a screenshot of your clip info: https://mediaarea.net/en/MediaInfo/Download

[arcbound34519447d6ho]

 

[MyerPj]

Yes, everything is against you in that clip. 3+ hours long, very long GOP, the number of frames between and actual frame. Otherwise, an actual frame comes up, then for 120 frames only the differences are stored and the computer has to figure out the math and what the changed pixels will look like. It's UHD footage, that's harder on the system also. And as Richard mentioned, the dreaded Variable Frame Rate (VFR)  which typically causes problems. Try Shutter Encoder to convert the files (before editing) to constant frame rate. Maybe try ProRes LT or even ProRes PROXY format within Shutter Encoder for easy to edit formats with 'better' files size.

https://www.shutterencoder.com/en/

[Richard M Knight]

Your footage also seems to be variable framerate, this adds a whole new level issues. The usual advice is to trancode it to constant framerate using a third party program like Handbrake.

[arcbound34519447d6ho]

I'm running it through HandBrake now. Estimated time remaining is 3 hours. It's going to be awhile before I can update as to whether this works or not.

[MyerPj]

Try Shutter Encoder, it's a much nicer, more intuitive UI to the same end.

In Shutter Encoder right at the top of the list of options is to 'Cut without ReEncoding', Grab 10, 20, 60 seconds of that clip and cut it without reencoding. Then you can experiment on that clip, don't have to wait 3hours for the test.

[arcbound34519447d6ho]

It's already 10% in, I may as well let it finish. Unless Shutter Encoder is less of a resource hog. HandBrake has my CPU at 100% use, and it CAN'T be good to run it that hard for hours.

[arcbound34519447d6ho]

Ok, the file that I put through HandBrake DID work, but time investment wise, this is way too much. HandBrake had my CPU at 100% for just over 3 hours, and the project in Premiere Pro still took another solid hour to render. Tying up my system for 4+ hours at a minimum of 90% load is just too much. Way too much. I'll burn my system out doing this.

[R Neil Haugen]

Just ... wow.

First, that is not just VFR, variable frame rate, that's the widest framerate variance I've ever seen! "Nominal" fps is 60, but the variance goes between the slowest at 20fps to the fastest of 62.5fps. I'm stunned Premiere can work with that media at all.

I'm used to seeing a range of maybe 51 to 62 for "60fps" VFR media, but ... 20 to 62? I'm ... boggled here. Never seen anything remotely that  varied before!

I would say to convert that entire VOD to CFR ... constant framerate ... in either Handbrake or ShutterEncoder prior to importing into Premiere.  It would work vastly better!

Next, the long-GOP thing? That's got the relevant numbers in the 'format' line. M=3, N=120. That second, N figure, says there's up to 120 frames between complete iframes. Holy Terwilligers Batman! That's ... as nasty as any drone media I've ever seen. And again, if you get normal playback of that within Premiere, just ... wow. Your system ain't doing to bad.

But combining the massive load of that wide framerate constantly shifting, and the load of computing probably over 150 frames just to be able to compute and render any specific single frame ... yea, that's putting a MASSIVE load on your system.

[arcbound34519447d6ho]

And yet that setup has been trucking along just fine until recently, and I can't figure out why. I don't know what's changed between when it worked and now other than Windows and Premiere Pro updates.

[R Neil Haugen]

Shear dumb luck? You have some of the nastiest media that I've ever seen. Way nastier than most. That you've been able to use that At. All. ... is what amazes me. Not that it's now an issue.

That said, Pr24.x does naturally put more load on a system than 23.x, that's what happens.

[arcbound34519447d6ho]

Finished an export of the same file with DaVinci Resolve a bit ago. Even without the GPU (since it's the free version), it handled the whole thing without so much as a hiccup. If I can't get Premiere Pro to handle these files again, I think that's going to be my solution moving forward.

[Media Editor]

When I have very long videos to encode, I break them into smaller videos.  When they are encoded, I use Shutter Encoder to merge the videos.  It merges the videos quickly and does not reincode the them.