Project Manager - Consolidate and Transcode problems

[Jeff Bugbee]

A while back I had several issues with clips not trimming properly during the Consolidate and Transcode process. I reached out to Adobe support and was given this fairly comprehensive list of when and why clips will fail. For some reason, this list is not posted publicly anywhere. I've copied it in it's entirely below. I've bolded several line items I feel are fairly important. There are now essential features in Premiere that I skip using (such as the Master Clip Effect layer) since it breaks the archive process.

These issues affect all versions of Premiere, including the current CC2017 version - 11.1.2 (22)

Hopefully this will help some people with project management.

• If any one of the 4 major video parameters (frame size, frame rate, fielding, PAR) do not match or are not supported by Preset/codec, then entire clip will copy.
• If the clip has multiple stereo channels of audio. These can only transcode to QuickTime, all other formats(e.g.: DNxHD) will fail and copy, since the other exporters do not support multi-stereo (QuickTime is the only exporter that currently supports that and only with uncompressed audio)
• Time remapping is not supported. Will copy entire clip without warning.
• There area number of formats/presets that are not supported for transcode (e.g. mostly still formats)
• Merged Clips are not supported.
• Copy of folder structures (e.g.: P2, AVCHD, etc.) is not supported. Depending on format, some files may be left out or other issues.
• Master Clip Effects are not supported, and will be copied without no warning. This includes all the formats that now have source settings as Master Clip Effects (e.g.: R3D, ARRI, etc.)
• Audio only files
• If used in a merged clip. Unused merged clips will also copy.
• Render and Replace files, even if unused
• Clips with dual mono (2 mono) will copy, although they may also transcode (this is a known bug), but the project will always link to the copied clip and not the transcoded clip (using up even more space)
• Clips with single mono audio will also fail to transcode.
• Modify > Audio Channel　changes may also result in failure to transcode.

[Nicolas from Autokroma]

> Just concerned about the number of known issues:

Thanks for mentioning PlumePack. Which "known issue" or limitations is a problem for you ?  

[Baffy19]

I'm wondering if these gaffes have changed before the current version 23.1 Premiere? Or is there still the same mediocre work at the moment? In general, the Consolidation function is very useful when it is necessary to archive a project, reducing unnecessary frames and freeing up a lot of disk space. When will they finally take it up and make it fully functional and hassle-free? To take archive the project and nothing will be broken. I wrote a little earlier on the forum that the cropping function, which is very much in demand, does not work correctly. Then the point of all this is that it is being implemented into the Premiere product, which cannot be used to the full.

Can I ask an Adobe employee who is responsible for this?

[Warren Heaton]

Transcode all camera originals to ProRes (or better yet, shoot ProRes to begin with) and drop them into Sequences with Video Previews set to matching ProRes and then most issues go away.

It does not have to be ProRes.  DNxHD also works.

[cmush012]

I've had a lot of success with PlumePack.  I would recommend at least trying it.  I've never gotten the built in project manager to consolidate things correctly.  It is the one thing I miss from Avid.  Consolidating was never an issue with managed media.

[Baffy19]

From the experience of using it, Plume does not work correctly. When collecting files for consolidation and moving to a new space, Plume does not take into account some types of AE files and compositions. I personally encountered this that year. At the exit, I received a message about files that were not copied correctly. And this was confirmed by checking the project. Not all files were transferred. Imagine you have a big project and how much time you will spend collecting missing files placed throughout your system (I mean various effects, overlays, pictures, music, sound effects, etc.). It's not a sweet treat. The standard Premiere tool copies all the materials that you have in the project. But, there is one BUT. The file cropping function does not work properly. Thus, you do not save the workspace of your disks, but simply duplicate the project by collecting all the files in one place. These are all consolidation and pruning possibilities. And I've been writing about this feature for several years to the Adobe team. It's sad that no one sees this function as useful. I am sure that most of those who work with numerous projects are obliged to use this opportunity to save space on the hard disk. And this is not a small space.

[Nicolas from Autokroma]

Hello @Baffy19 ,
PlumePack (the third party plugin for Premiere Pro you mentioned that we are developing at Autokroma) should take into account all kind of files for copy. It is even better for this than the native Project Manager ! You mentioned AE files / compositions : The native project manager only copies the AE project file, PlumePack does more than that : it also collects the AE compositions dependencies (image/video used in the composition). 
It is also better if you want to select only a part of your project, and/or to detect "unused" stuff. 
If you want to see the full comparison, we wrote an article about this. 
Sorry that you encountered such problem with PlumePack. Maybe it was a bug in older version. In any case if you try PlumePack a new time, you can send us the logs and we should fix quickly the issues you have (whatever if you are using the free or premium version). 
Only "Beta Codec" bugs (So Trimming with H264, H265 mainly) are harder to track and could not be fixed soon, but we generally fix quickly all other bugs quickly.
Also PlumePack has the advantage to show you exactly what it did with the analysis and report ! Which files it copied (successfully or not), which sequences it takes into account, etc.. 
Best,
Nicolas from Autokroma

[Baffy19]

Cropping files is needed in full working order in Plume. This is a huge saving of disk space. This is the essence of this function. As for consolidating dependencies, you may have fixed it. It pleases. But, we need a clearly working cropping of unused frames from the source. Let's say my source is recorded in good quality, weighs a lot of disk space and is quite long in time. I need to take a couple of seconds out of it. Imagine how much space I can save on my hard drive when using the consolidation, cropping and copying to a new space.

[Nicolas from Autokroma]

>  Cropping files is needed in full working order in Plume. This is a huge saving of disk space. This is the essence of this function. As for consolidating dependencies, you may have fixed it.

It is more complex than it looks like. For the same codec (let's say H264), each camera can write a video media in different ways. So even if it's the same codec, trimming a video taken in a Camera Brand A would not be the same than trimming a video file coming from a camera Brand B. That's why for H264 it works well on some files and other not at all. 

We need to make choices and set priorities. Saving space with Trimming on a compressed H264/H265 file is not the better case I think. We tried to implement it, but when we discovered how hard it would be to get a really "workable" solution for it, we decided to let it as a "Beta Codec" with a dedicated option (by default, those files will not be trimmed).  

The first top priority is to get a stable and trustable tool which is going to collect all needed material in the project, and which can remove useless media not used without considering Trimming. So we pay attention to support all edge cases to be sure the final project has everything needed and more important : if something is missing, tell it to the user inside report/analysis so it knowns something is missing and there is no hidden missing stuff. PlumePack is now quite stable for it thanks to users which reported some special cases ! 

Then, priority comes to the codec we are supporting for trimming. For us first it is RAW kind of footages (R3DRAW, BRAW, Prores RAW, ARRI RAW, ...), because those files are generally huge and saving space on them can make a huge difference, and also because users generally want to keep the "RAW" parameters of those files (+ quality) as much as possible (and so transcoding is not an option). PlumePack works really good with those kind of footages (the RAW codecs which are supported).

The second priority on supported codec for trimmign is the "huge and high quality footages", so the files without GOP compression : ProRes, DNxHD, Cinerform for example. Those files are easier to trim and are well supported by PlumePack.

But I agree with you though, PlumePack would be better if all codecs were supported, and we will try to improve this little by little, by also paying attention to keep the product as stable as it can be. Thanks for your feedback on this. 

EDIT: We fixed a lot of issues for H264 and H265 Codecs from PlumePack 2.2.0, so they are now almost always trimmed successfully (And no worry, there should be no hidden issue !). They are not in the "Beta Codecs" anymore and we had a lot of positive feedback from users which were waiting for improvements on those codecs ! 

Best,
Nicolas from Autokroma

[Baffy19]

Thank you for your understanding and clarification on supported formats. We will be waiting for good news from you in the development of the product.

[J28689319pi1e]

Unbelievable. Another core feature in PP that just doesn't work - since at least 2017, as I read here. It's just depressing.

I tested* the free version (just copy & collect) of PlumePack with two simple projects and it worked perfectly. Thank you Autokroma for your great work!
......
* macOS Ventura 13.2.1,  PPro v23.2, PlumePack v2.1.1

[J28689319pi1e]

The longer I work with Premiere, the more curious I get as to how the hell larger post offices are supposed to be able to work with this software.
Just the fact that projects can't be consolidated properly should actually be a no-go. In addition, the problems with multi cam ... Imagine this in an AVID world.

I have given up posting the problems here in the forum. When I come across a problem, I regularly find threads here where the first post describing the problem is, for example, from 2017 and the latest from this year: The problem in question has not been solved to this day.

Just like in this thread. I'm becoming more and more convinced that all these functions, which are really crucial for post production, will simply NEVER be fixed.

[J28689319pi1e]

most issues

By @Warren Heaton

Unfortunately, that is not enough.

[Warren Heaton]

@J28689319pi1e 

In addition to using CODECs that are good for editing, I also find it helpful to hold back a version.  For example, currently I have Premiere Pro 23, After Effects 23, and Media Encoder 23 installed concurrently with Premere Pro 22, After Effects 22, and Media Encoder 22 and it's the latter that I use to deliver marketing content for broadcast television and social media on an ongoing basis.

[J28689319pi1e]

Warren, thank you for your reply.
I always edit with ProRes proxies.
And I also try not to update immediately. BUT I always hope that the latest version will fix some bugs that are important to me 🙂

[Baffy19]

Adobe engineers, aww-aww-aww. I, like many others, am tired of indifference to the most powerful and important in the direct sense of the ability to correctly assemble and send the project to the client. I have two questions for you: 1) When will the project consolidation and copying function work correctly? (since 2017, no damn changes); 2) Does anyone, in general, except for the timeline, look anywhere further? This is disrespectful not only to the users of your Premiere product, not only to the customers, but also to the entire film industry. We need this native ability to do the simplest things. Make the product professional and painless.

[Kevin-Monahan]

Hello @Baffy19, @J28689319pi1e, & @Nicolas from Autokroma,

I read this thread and can agree with a lot of these issues. Can you file individual feature requests here? Then, we can all upvote the requests to have the Project Manager improved.

The reason that Plume Pack can work for some of these requests is that it uses open source-based code and software for encoding and decoding certain codecs. While that's a cool idea, Premiere Pro doesn't use open source software under the hood. Adobe needs to develop its own tools for that, and as you can guess, that takes added effort and energy to get that done. More than what it seems from a surface-level view, anyway. That said, let's light those feature requests up and we can upvote them as a community effort.

BTW, feature requests should be made individually and not as a "list" of requests. Those "bundled" requests are not preferred by the product team. "Just fix the whole dang thing!" will probably not work out well. Issues around the project manager are best made as I suggested.

We might start with support for the ability to consolidate Long GOP formats. That would go a long way to assisting a lot of users like ourselves.

Thanks for adding your comments.

Cheers,
Kevin

[Odobe]

Hi Daniel and Kevin, 

I have the same thing - trying to project manage this sequence - manually i can get it to 3gb but its 300gb with Project Manager

Files are MXF and VFX are requesting 4444 QT trimmed.

[scrozier]

Project manager still woefully broken after many years. Needs to be robust and flexible, instead has a labyrinth of dos and don'ts to navigate to just get it to function at all. Not sure why they can't get it to make folders or at least approach the reliablity of the AE Project Collect. Not trimming a file becasue a master clip effect is applied is insane. It should just disable all master clip effects and render. The code should not be that complicated.

[Kevin-Monahan]

Thanks. I'll move this to the Bugs forum.

Kevin

[Odobe]

Solution for this in the end was

1. Copy edit into a new sequence
2. Reconnect transcodes to source media - remove any nests / unwanted effects / lumetri etc. and take note of any retimings using time remapping- these are going to be the things that could go wrong.
3. Check the edit appears as you want to handover - grade removed if needed  - also can check the frames are all there using an offline and difference opacity.
4. Render and replace with 4444 QT highest quality  - match source and Individual clips- here you can add handles if you want - and choose a new folder labelled clearly for VFX 

This worked really well for us in this instance.

[Andy Adkins]

Wow. 6 years on and this still doesn't work. I learned to not trust project manager for anything big, but thought I'd try it today for something very simple.

I just wanted to get some partners a bin of music files that are spread across my local storage. I create a tmp sequence, dropped in all the music then thought I'd project manage the sequence (copying, not transcoding), and send the files. Should be easy. But it takes forever and starts copying what seems like everything in the project. This is with the exclude unused box checked, of course. We're talking like 11 small wav files here. Should take like 10 seconds.

It's just insanse to me that such basic stuff like this doesn't work. There are zero things about what I'm trying to do that is advanced in terms of programming/logic. These kinds of you're-kidding-me fails are why I've been firing up Premiere less and less (as I migrate to Resolve and steer people I work with in that direction as well).

[J28689319pi1e]

Yes, I had this experience too.  I would be very happy if Adobe employees could explain to me why such a basic program function cannot be impemented/fixed/provided.

I would also be interested to know how other companies or freelancers deal with this problem (apart from using PlumePack, which also can't handle PP Productions).

In our case, the 18 TB hard drives with the BUs are stacked to the ceiling.

[J28689319pi1e]

BUT nice to know that we live in the Bugs forum now 🙂

[Nicolas from Autokroma]

Hello @J28689319pi1e ,

What would you want PlumePack to do exactly to your PP Production ? 

With PlumePack you can consolidate a project inside a PP Production. The new PlumePacked project created will be independent from the Production, which means that any media in any timeline of this project will have its Project Item inside this same project (and not in another one like we can see generally in Productions). 

Please contact us with an explanation of your workflow, and how you would want to work with PP Productions, and we will see how we could improve PlumePack for it => https://www.autokroma.com/contact
Best,
Nicolas from Autokroma

[Baffy19]

Hey, @Andy Adkins .

I have been writing about this to the developers for a long time and was sad myself until I abandoned this idea and just changed the NLE. What is happening now with the development of Premiere is utter chaos and engineers' misunderstanding of what users want. This is extremely sad. And the prerequisites were to develop this idea. I just spat on it.