Transcode or Proxies?

[CC268]

Hey guys,

I am a hobbyist who enjoys shooting some drone footage and family video (vacations, etc) - so please keep in mind I don't do this for a living.

I've been trying to determine the best workflow for me and whether or not I should transcode my raw footage to Cineform/DNxHD or just use a Proxy workflow. The problem with transcoding all my raw footage is that the file sizes are massive. I transcoded 63GB of 2K/4K footage to Cineform from my Hawaii family vacation and it was 690GB! That is massive.

It seems to me that for a hobbyist, the proxy workflow might be much more reasonable and certainly more archivable as I am paying for my own external storage and I don't have a customer to simply charge for an external drive for all that footage. My thoughts are that archiving the final render and the raw compressed video (h.264/h.265) is so much more practical than storing 690GB of transcoded footage.

What are your guys' thoughts?

Thanks!

[excited_Genie16B8]

I find the Proxy workflow to be ideal for tough to edit footage.  You get easier to edit footage at more manageable sizes.

Just be sure to use Cineform proxies, not H.264 proxies.

[CC268]

Thanks for the feedback. Yes I have been using Cineform proxies! One reason I ask this question is I am doing a big storage upgrade to optimize my system and I am deciding between 500GB or 1TB SSD/NVMe SSD. Going with 500GB drives will save me several hundred dollars. If I use the proxy workflow I can go with the smaller drives no problem.

[excited_Genie16B8]

500 GB is a pretty small Media drive.  You might consider looking at a multi-Terabyte spinner.  The speeds are plenty fast enough for media, and you'll get a LOT more storage for the same money.

[CC268]

Oh no that is ONLY for current (keyword current) Project and Media Files. I will have a separate SSD for Media Cache & Scratch as well. Archived projects and footage will be placed on WD Reds. 500GB should be more than enough for current projects unless I’m shooting a Hollywood movie lol. Keep in mind I am a hobbyist I don’t do this for a living. That said I may go with 1TB just to be overkill. This is all through research I have done over the past month and is the recommendation of Puget Systems, who in my opinion is one of the best in regards to optimization. They are one of the few who have done real world testing on storage optimization for Premiere Pro.

Optimizing Storage for Premiere Pro - YouTube

All this said, I don’t want to get off topic as I have spent months researching all this and I’d like to focus on the question I originally posted (with all due respect).

[ardengrier]

Can I ask why cineform instead of h.264 proxies?

[Kevin-Monahan]

Can I ask why cineform instead of h.264 proxies?

The rule of thumb is that H.264 is terrible to edit with. Period. So when you're creating proxies for being easier on your machine, why throw one of the hardest codecs to decode as your proxy format? It doesn't make sense.

The only case where it sort of makes sense is if you have a large amount of footage and a small amount of storage and you need a very "portable" version of your project. If you're in this position, make sure your storage is very, very fast portable storage or you're going to be back where you started from with a badly performing timeline.

Cineform, DNxHD/HR, and ProRes are all cross platform, visually lossless, high quality, professional "mezzanine" (some editors refer to them as "intermediate") codecs. These formats are intraframe formats, not interframe formats. Explanation: Intra-frame vs Inter-frame Compression These formats are equally great, and work the same, though some have their preferences for formats, quality choices, and in the case of proxies; frame sizes.

Interframe "Long GOP" codecs (.mov or .mp4 wrapped H.264 or H.265, typically) are increasingly common (mobile phones, DSLR, drone, VR camera, screen captures, webcam). However, these formats are exceedingly difficult for your computer to "decode" for performing playback and other editing tasks.

NLE software from every manufacturer (Premiere Pro included) will have much better performance during playback as proxies with these intermediate formats. That's why they're great choices as proxy formats.

From my perspective, having transcodes to Cineform, etc. is an even better scenario because you have the benefit of great playback performance, you can use smart rendering to make your Cineform exports lightning fast. Fixing mistakes on watch down can be easily repaired and a re-export takes a fraction of the time a H.264 export would take. If any of the transcoded files play back with difficulty, you can always create individual proxies for those files.

The best scenario, I think, is to have both proxies and transcodes—but that's just me.

The only cost is:

• Time: To create the proxies and/or transcodes.
• Temporarily filling up a great deal  of drive space.
  • Drive space for your transcodes can be deleted once your master is created.

Please let me know if you have any questions about these kinds of workflows.

Thanks,
Kevin

[ardengrier]

This is so helpful, thank you!

You're saying you ideally have proxies and transcodes meaning proxies of your transcodes? Or Transcodes in cineform in addition to proxies of your raw files?

[rodneyb56060189]

Kevin is really good and will get back to you. Be careful with terminology cause it can be confusing. The original poster mentioned the 'original raw files of H264.." or something like that.. and was never corrected. The word RAW should not be used. Instead, the word " ORIGINAL or NATIVE .." or something like that should be used. RAW is a very special different kind of media that has nothing to do with this particular thread at this point.

So, you want to know if the proxies are from your native files and transcodes are from OTHER native files... In general I think what Kevin means is that he likes to ( if he can and it makes sense with his storage space, etc. ) transcode first .., so now you have totally new wrapper and codec to work with.

Let's say you had mov H264 4K native footage. That is really hard to edit.  So a transcode might be mov DNxHR HQ 4k …  OR ELSE if you are going to deliver full HD, you can transcode to a new codec that's nice to work with and make it 1920x1080p... and THEN you've essentially halfed the amount of disk space you need compared to the 4k transcode.

Once you get out of the H264 thing you will have much larger files ( transcode or proxy )… which is why storage comes into play.

At any rate, let's say you made the DNxHR HQ 4k as your transcode and your editing is struggling with THAT IN SOME PLACES WHERE YOU ARE ADDING EFFECTS ETC.... Now you can "Proxy" just THAT part to 1080p if that is the timeline you are working in.

hehe... it gets complicated but once you know it , then it is like everything else... it's easy to understand after that … and you can tailor what you do to fit each project you work on,

good luck !

[rodneyb56060189]

=============

Interframe "Long GOP" codecs (.mov or .mp4 wrapped H.264 or H.265, typically) are increasingly common (mobile phones, DSLR, drone, VR camera, screen captures, webcam). However, these formats are exceedingly difficult for your computer to "decode" for performing playback and other editing tasks.

============

GOP means 'Group Of Pictures'.  essentially ( I'm an idiot so take this with grain of salt ) what happens is ( like with H264) the internal (in the camera ) process of creating images ( say 24 fps ) is using only one complete full frame at the beginning and end of the 'group'. The rest of the time it is guessing ( yes, even before it sees the next frame it is guessing what it is likely to be, based on the stuff before it...usually having to do with the motion of stuff and color fields ( like a big blue sky etc. ))… and throwing away the pixels it doesn't need to define the image and INSTEAD gives the substitution of mathematical definitions of those areas etc.  So that is the Compression part of the process.  CODEC means COmpression DECompression. When you play it the same program that made the compression now does the opposite, and Decompresses the info from math stuff to pixel info … and THAT IS REALLY making your CPU run around like the fastest hamster you ever saw running in a wheel !  It is really hard on the computer resources (mostly cpu ).

That's what makes things start to stutter and stall out and get not smooth when editing with H264.  It is the need to decompress on the fly at the same time that you are changing things via the editor ( cuts, effects, etc. ).

If you are just 'watching' a H264 video it's fine, and looks nice and plays smooth, because it was originally meant just for viewing, not editing.

[rodneyb56060189]

In defense of the camera manufacturers... the use of H264 allows them to put a LOT of time (footage) onto an SD card. Otherwise nobody would buy the camera. If you filled up a 62 gig SD card in one minute it would not be fun. So they use H264.

Then, most would like you to think that you can edit this stuff and so on, but it really comes down to people learning that they have to either transcode, proxy, ORRRR struggle with the material ( stutter, not smooth playback, etc. )… So it kinda makes sense that so many people ( Nikon, canon, Sony, etc. ) have the option to record H264 (and there are other formats that are also very compressed ).

Luckily there are secret scientists at work doing secret stuff ( nobody is allowed to leave work for years at a time so it stays a secret ) to make new and improved CODECS.

[Kevin-Monahan]

GOP means 'Group Of Pictures'.

That link from the Wolfcrow site has a pretty good explanation. Check it out.

KM

[Kevin-Monahan]

• RAW is a very special different kind of media that has nothing to do with this particular thread at this point.
• Kevin likes to ( if he can and it makes sense with his storage space, etc. ) transcode first .., so now you have totally new wrapper and codec to work with.
• a transcode might be mov DNxHR HQ 4k
• You can transcode to a new codec that's nice to work with and make it 1920x1080p... and THEN you've essentially halfed the amount of disk space you need compared to the 4k transcode.
• Once you get out of the H264 thing you will have much larger files ( transcode or proxy )… which is why storage comes into play.
• DNxHR HQ 4k as your transcode and your editing is struggling with THAT IN SOME PLACES WHERE YOU ARE ADDING EFFECTS ETC.... Now you can "Proxy" just THAT part to 1080p if that is the timeline you are working in.

Thanks Rodney. These are good key points.

I especially agree with the issue of storage space. Ol' skool dudes like Rodney and I are used to massive file sizes for these intermediate codecs as BITD you rarely worked with such things as H.264, but now it's the NORM.

However, what Rodney and I are not used to are the HUGE sizes of the 4K versions of these files. I can't imagine how shocking this might be to a newer editor.

Now, to play the NLE game with full transcodes and proxies—you have to have a zillion terrabytes of high speed storage hangin' around. But to me, that's the cost of a stable workflow, and In the end, you can delete a lot of those files. No big whoop.

Kevin

[Kevin-Monahan]

You're saying you ideally have proxies and transcodes meaning proxies of your transcodes? Or Transcodes in cineform in addition to proxies of your raw files?

Hey, great question.

I would probably strike the proxies from the transcodes as they would be conformed from the non-standard original files. This is especially the case with frame rates, which can often be variable in Long GOP originals. Color sampling differences would likely take longer for encoding the proxies than if you used the transcodes, as well.

You might note that you cannot create both transcodes and proxies as a single process in Premiere Pro. It will take 2 passes through Media Encoder. Although, that would be a sweet feature request.

Thanks,
Kevin

[Kevin-Monahan]

Just be sure to use Cineform proxies, not H.264 proxies.

I believe I'm going to have this tattooed on my chest. In bold!

KM

[Kevin-Monahan]

HI,

I've been trying to determine the best workflow for me and whether or not I should transcode my raw footage to Cineform/DNxHD or just use a Proxy workflow.

SInce you are keeping the cost down for the drives you're planning on installing, the Proxy workflow will work just fine.

That said, you can instead use a hybrid "smart rendering" workflow by changing Sequence Settings' Preview File format for playing back clips with effects better and getting faster exports. In this workflow, you don't need proxies, you just are rendering more files to create files intermediate codecs. Of course, the preview files take time to render and take up a lot of room too!

Here's a video that explains that hybrid smart rendering workflow I was talking about:

Let us know if you have any questions.

Thanks,
Kevin