Project Manager - Consolidate and Transcode problems

[Jeff Bugbee]

A while back I had several issues with clips not trimming properly during the Consolidate and Transcode process. I reached out to Adobe support and was given this fairly comprehensive list of when and why clips will fail. For some reason, this list is not posted publicly anywhere. I've copied it in it's entirely below. I've bolded several line items I feel are fairly important. There are now essential features in Premiere that I skip using (such as the Master Clip Effect layer) since it breaks the archive process.

These issues affect all versions of Premiere, including the current CC2017 version - 11.1.2 (22)

Hopefully this will help some people with project management.

• If any one of the 4 major video parameters (frame size, frame rate, fielding, PAR) do not match or are not supported by Preset/codec, then entire clip will copy.
• If the clip has multiple stereo channels of audio. These can only transcode to QuickTime, all other formats(e.g.: DNxHD) will fail and copy, since the other exporters do not support multi-stereo (QuickTime is the only exporter that currently supports that and only with uncompressed audio)
• Time remapping is not supported. Will copy entire clip without warning.
• There area number of formats/presets that are not supported for transcode (e.g. mostly still formats)
• Merged Clips are not supported.
• Copy of folder structures (e.g.: P2, AVCHD, etc.) is not supported. Depending on format, some files may be left out or other issues.
• Master Clip Effects are not supported, and will be copied without no warning. This includes all the formats that now have source settings as Master Clip Effects (e.g.: R3D, ARRI, etc.)
• Audio only files
• If used in a merged clip. Unused merged clips will also copy.
• Render and Replace files, even if unused
• Clips with dual mono (2 mono) will copy, although they may also transcode (this is a known bug), but the project will always link to the copied clip and not the transcoded clip (using up even more space)
• Clips with single mono audio will also fail to transcode.
• Modify > Audio Channel　changes may also result in failure to transcode.

[PGDVP]

A.   Just to focus on one of the points... Audio Clips only. So if it's a music track...??!!

B   "Master Clip Effects are not supported, ... source settings ...(e.g.: R3D, ARRI, etc.)". So all raw media.??!!!

Basically it would have to be a pretty basic Project / TL to enable a good consolidation. And we're paying subscription for this.!!

[Jeff Bugbee]

PHVP  wrote

A.   Just to focus on one of the points... Audio Clips only. So if it's a music track...??!!

Correct, music tracks (wavs, MP3s) will not be consolidated. They will only be copied over (and no message or indication that a copy took place, even if consolidate is selected)

PHVP  wrote

B   "Master Clip Effects are not supported, ... source settings ...(e.g.: R3D, ARRI, etc.)". So all raw media.??!!!

Not just raw media, ALL media. If you have any sort of video clip and you apply an effect on the Master Clip level and attempt to consolidate, that file will be copied regardless of your consolidate settings and it will NOT give you any sort of error or message that the file was not consolidated.

[excited_Genie16B8]

it breaks the archive process.

Well, it breaks your archive process.

To my thinking, the best archive process is to copy the full card over to the archive location and leave it there in perpetuity.  Do this with a proper file manager like Windows Explorer or Finder as a first step, before you even copy the media to the working drive.

[Jeff Bugbee]

Jim Simon​

I should have been more specific regarding the word "archive." It breaks the archive functionality of the Consolidate and Transcode option of the Premiere Project Manager.

This thread is not about file organization, it's about a specific function in Premiere. I'm currently sitting on about 100TB of raw footage that we have to pare down for archive. "Copying the full card to the archive" is not a valid archive solution, hence this thread pertaining to the non-working Consolidate and Transcode functionality.

[excited_Genie16B8]

about 100TB of raw footage

Damn, dude!  Even with the largest file sizes possible (4K Uncomprssed RAW), that's well over 500 hours of footage.  What kind of project are you working on?

[Jeff Bugbee]

This is 6 months worth of projects. We're a 24/7 network and shoot daily. Suffice it to say, a working consolidation workflow is absolutely essential for us.

[PGDVP]

If you have DaVinci Resolve, you could use that to consolidate - Kinda!. Not perfect but a possible workaround.

[excited_Genie16B8]

This is 6 months worth of projects.

That makes more sense.  But then I'm kind of back to what I feel is the better archive method.

Keep it all.  A 4TB external can be had for about $100 these days.

[Jeff Bugbee]

We have neither the time nor the space to keep everything, nor do we NEED to keep everything. Once a project is done and edited, we no longer need the raw footage. We only need the media used in the project (plus 10 sec handles). With a proper consolidate and transcode, we can take a project from 1TB down to 50GB using ProRes422HQ as an intermediate codec. We're not going to take a 95% loss in storage efficiency because of a crippled functionality of Premiere.

[Editror]

Basically you're saying the project manager is pretty useless... by that list, in 99% of the cases you simply can't archive your projects.

The guys at Adobe should really get their heads together and address this problem,  it is a big issue that been around for too long (we already at PPro cc 2019).

most of the projects today have mixed materials (different formats, fps, resolotion, etc.) - Project manager most be able to deal with it successfully.

MAKE IT WORK guys!

[Philliams]

I agree with Dror82 & Jeff Bugbee,

unfortunately this appears to still be the case with cc2019 -

"Project Manager" calculates an estimated output project size of 56Gb for a 207Gb project,

I hit OK, wait for the "transcode" and still get a 189Gb project - and an error message that some files couldn't be transcoded

Our client wants a trimmed copy for their archive, and I do too.....

it looks like best option is still:

1. make a new project
2. import the final sequence and all content used with it
3. deal with the file size

which is exactly what I did to get this project file size down to 200Gb anyway

that isn't what I expected from "Project Manager"

Please Adobe, you can do better surely?

[Jeff Bugbee]

I'm sure you know this but what's happening is that 56GB estimate is for if everything can be trimmed down. Since it's not, use that checklist and see if any of your clips fall within those parameters. This post is almost 3 years old now so perhaps some of it is fixed but you should be able to narrow down what is preventing you from properly trimming.

[Editror]

You absolutely right and that's the sad thing... Over 3 years and nothing

was changed/fixed.

I'll take my chance of sounding a little bit angry and add to the

matter... (It's

not anger it's disappointment)

Adobe do absolutely nothing with it

If Adobe wants video editors to totally imbrace Ppro It must bring it up a

notch.

I really enjoying editing on ppro and if I could I would work only on it,

but poor post-project media management is not an option for me... Its a

deal braker for 80% of my projects

that's the reason most of my projects still being edit on Avid

[rubenc62783441]

Hi,

After a lot of trial and error we found a workaround to make trimmed down consolidates of our projects.

- Make a duplicate project from the last project, name it something like "ProgramX_Archive".

- Render and replace all the media of the final sequence, and select a specific folder, don't select "next to original media file". We also include 50 frames handles.

- Temporarily rename the folder with the original footage, so Premiere thinks the original media went offline.
Doesn't matter, since the render and replaced media is still online.
- This is now your trimmed down project.

Best way to check if verything worked out is to render and replace to another hard drive, so you can check if everything works after you totaly disconnect the drive/sever with the original media.
Another great way of archiving the audio is to make an AAF from it, and then import it back in the sequence.

Ruben

[stevewhite999]

Ruben - thank you for your post with a solution and to the OP for outlining the problems (which I've just run in to). Information on this seems very thin. Very pleased and grateful to have come across something so helpful as your workaround.

An additional problem I have is Premiere copying all my proxies too. It would be useful to be able to choose whether they are included or not (I am having to maually hide them and force them offline).

[George Langley]

Thanks - that probably explains what I'm seeing too. (Premiere 2018 as of Octobe 2020.)

[SampsaS]

This actually used to work for me with many different profects. I got nice small trimmed H265 clips of the parts I used on the timeline in my final sequence when I used the consolidate and archive tool and got my original 200GB raw footage down to few gigs. But since maybe one update back before the latest updates during Adobe MAX i've had the same problem. I just get the general error that "some clips could be transcoded and were copied". What is the intended function of TRANSCODE and consolidate if it accepts only 1% of formats/use cases? Any ETA when this will be fixed?

[Trevor_Asq]

This all seems to continue to be a problem. The note about not consolidating clips modified to stereo explains why I'm getting copies.

The render + replace workaround has potential, going to test that out. AAF export/reimport could work for audio.

But really we need a *consolidate* option that works (ie transcode should be an option in addition - consolidate should be to native format).

[George Langley]

SO, another year, and still no workable Media Manager... Am tired of copying hours of raw footage for a short edited video.
Has anyone tried PlumePack?
https://www.autokroma.com/PlumePack

It looks like my ideal solution - keep all files in their original format, but trim to just what is used. AND, it says that it can handle nested sequences, which is huge for me.
Just concerned about the number of known issues:

https://www.autokroma.com/blog/PlumePack-Limitations-Known-Issues

[Daniel22379067xmlj]

Hello friends,

We are using Project Manager to send some files to a remote colorist. We want to take our 30-second spot and only send them the clips used in the sequence with 24-frame handles. We have "Consolidate and Transcode" selected  with Apple Proes 4444, and "Include Handles" selected.

My understanding is that Premiere is supposed to TRIM each clip included in that sequence so the resultant project would be MUCH lighter weight. 

The problem we're experiencing is that no matter which preset/format/option we select, Premiere always is collecting and copying the full size/length of the included clips to new location - it is not trimming and transcoding the clips to be much smaller file size (only what's needed + 24 frame handles). This is resulting in 100GB project size instead of the maybe 10GB we expect to send to our colorist.

Is this a bug? Are we expecting premiere to do something it is not supposed to do?

Would appreciate any input from the community. Thank you.

[Kevin-Monahan]

DRTrade,

Sorry. What is the source footage you are trimming? Is it also ProRes or is it in a Long GOP format like H.264/HEVC? If the latter, I believe it is expected behavior that the resulting media will not be consolidated. If the source footage is intraframe based, then the resulting media will be consolidated.

Let me know what's going on.

Thanks,
Kevin

[Daniel22379067xmlj]

Hi Kevin - thank you for the reply.

The raw footage is MXF shot on an Alexa Mini.  I think that camera shoots in a few flavors so I'll need my editor to give us more detail if you think that's the issue. I also found some sample footage here but not sure that might help elucidate https://www.arri.com/en/learn-help/learn-help-camera-system/camera-sample-footage 

Is this behavior noted in the documentation for project manager? If you're re-encoding/trimming why would this be an issue anyway? In these cases, the precision of the handles is less important than being able to consolidate for color in to a smaller package that is portable, so If I could put in a feature request it would be to make an option to make Long-GOP footage work this way. 

[daapap]

It's not only long gop:

Once or twice a year i give the project manager a try to see whether it has improved. It has not. Forget about it, if your footage and  target format is not all of a kind. And if it works it may take ages and even 128gb-ram systems may stall because the have not enough memory. For color correction we just shipped an 18tb-raid across the country for a 75 min program because project management is not an option.

[Nicolas from Autokroma]

Hello @Daniel22379067xmlj ,

We experiended the same problem with MXF ProRes files. Maybe it comes from the extension (MXF) and it could work better with a MOV extension ?  

Otherwise, you could consider the 3rd party plugin "PlumePack". It's a Project Manager plugin, free for Collect & Copy and with a Premium License you can Trim your files without re-encoding ! So it's even better than the Project Manager transcoding feature in your case, because your ProRes files will keep their timecode, metadata, exact quality, ... Same than the Project Manager you can set handles, and the final files will be organized in the same way you organized your media in the Project Panel. 

Best,

Nicolas from Autokroma

[Nicolas from Autokroma]

@DR Sorry I just realized that the MXF you produced could be ARRI RAW files actually ? The bug you had could come from this ! PlumePack doesn't support those footages for now, but they are under investigation. 

However PlumePack supports ProRes, ProRes RAW, R3D, BRAW, DNxHD, Cineform, and even Long GOP such as H264/H265 but in a special Beta mode for now. 

Best,

Nicolas from Autokroma