Any way to prevent dupes in Premiere Project file?

[RyanPatch]

Due to working with both PluralEyes as well as multiple editors on the same project, I deal with importing a lot of XML and PRPROJ files into existing project files that I've worked with.

This, on a large scale, becomes untenable because every time I import another editor's work or a synced sequence, I get duplicates of all my media in the project. I'm trying to figure out if there any way to have Premiere check to see if the media is already imported, and reference that media in the imported sequence? If not, I will just keep getting larger and larger project files with multiple references to the same media, because both me and my other editors are working with the same media, but for different sequences.

As far as I can tell, when you import an XML, or even a sequence from another Premiere project, it creates new master clips for every clip used in the imported sequence. It does not recognize that you may already have a master clip that you're using in another sequence that is referring to the same media.

So if you want to keep both sequences in your project, you need to keep both master clips. If you delete a master clip from your bin, it will not stay in the sequence(s) that refer to it, it will get cut out, leaving a big steaming crater where it once was. Even if the sequence is not currently open. This is one of the things I loved about FCP7. You could delete every single one of your master clips and your sequence would be totally unaffected. You could even recreate the master clips by dragging them from your sequence to the bin.

I think if there really is no way to manage these duplicates, this is a HUGE problem for professionals who are working in environments with multiple editors. This isn't just a "well, learn how to deal with a new editing system" - this is actually a deal-breaker; and actually the only one that I see REALLY preventing Premiere from being the go-to choice for larger post houses. This problem becomes so big so fast that it makes true collaborative editing downright impossible. In my office we might have three people on a project, all editing and revising segments and passing them back and forth. On FCP7 this was easy as pie - we'd just cut and paste between project files and use basic versioning best practices. In Premiere, our project files quickly become nightmares and work is often inadvertently deleted or lost.

I would like to see:

- Smart media handling when importing sequences and projects. Premiere should look at the filenames and file location and attempt to relink any duplicate media. If it stumbles, it should ask for help like FCP.

- A media consolidation inspection feature. I'd love to see an option for inspecting your project for duplicate media references. When found, Premiere should automatically consolidate.

- Streamlined sequence exporting. You should be able to export a single sequence. I know there's some version of this in Project Manager, but we all know it should be easier than this!

Anyone have any ideas on how to fix this?

[1690x90x]

@Transcoding: Thank you for the tip. Unfortunately, following those steps (making all sequence clips independent, before exporting XML) did not work for me. Trying to finish up an FCP7 project using Premiere (on a deadline!!!), our project is an unsightly mess of needless clips. Nothing new with this, though. Very frustrating.

Adding to the frustration, we're trying to file feature requests, and the Captcha nonsense isn't working. This is a gripe for another time and place, though.

@Kevin-Monahan: We appreciate your suggestion, to create feature requests, to get this closer to the top of the queue. While I disagree with former assertions of "asinine" behavior, and while we greatly appreciate your position as the liaison between Adobe CC clients and engineers, there is something off-putting about being asked to hit a hidden target. In other words, give us a clear goal-- How many feature requests do we need to submit, to move this forward? Ten? One Hundred? One Thousand? And will the $#^&% Captcha nonsense work when we organize to do this?

It is doubtful, that the feature requests created by the handful of people reading this particular forum, will be enough to move this forward. And yet, we try to submit our feature requests, while also editing / screening / responding to notes, etc. In my opinion, your request (that we submit an undisclosed number of feature requests, while also working, and, often, paying ourselves for CC licenses) is not appropriate in this client-vendor relationship (with Adobe being the vendor, and us editors as the clients).

As an example-- When clients text me (to my personal cellphone) notes on an edit, while I might find that annoying, and borderline unprofessional, I put those notes in email, and send to my client (for a searchable, more readily accessible record of the request). I don't ask my client to write an email instead. I don't demand a reply with confirmation. I don't demand anything of them (except for respect, a paycheck that doesn't bounce, etc). They are my client! My job is to make my client's job easier. My job is NOT to impose my arbitrary sense of business communication decorum on them.

In short, I politely request that you fill out a feature request, every time you see a feature requested, instead of asking us (your clients) to do the work for you. I say this respectfully, with nothing but efficiency and common sense in mind.

[R Neil Haugen]

My understanding of the Way Things Work in the Adobe back-end is that what gets "counted" are the feature/bug reports filed by users. Kevin and other staffers that work the forums DO discuss with the program teams what they've worked with here on the forums, however ... neither Kevin nor other Adobe staff are allowed to fill out the ... things. So he can't, much as many of us might prefer.

And yeah, that is the WORST captcha I've ever used.

Neil

[1690x90x]

Thank you Neil, for your insight into the politics.

Following up on Transcoding's workaround for migrating FCP7 projects into Premiere Pro, at the bottom of p1 of this thread. While I didn't have any luck with his process, I arrived at the following, resulting in an uncluttered Premiere project:

- Export FCP7 XML; import into Premiere.

- Premiere project will have a mess of undeletable reference clips. Save this project, naming as first import.

- Then Save As, naming as Media Only.

- Delete all sequences and ref clips from the Media Only project version, so it is only media.

- Import your sequences from the First Project import.

- You will then have all the media and sequences, without the unnecessary reference clips.

- Save As.

[lehestro]

So after going through this thread, and others, and trying out various solutions on my own, it still seems like Premiere lacks this ability, which is immensely frustrating. I have filed feature requests about this back in November, but clearly people have been doing so for a couple of years now. There are some workarounds for specific situations, but no clear elegant solution that I can see – when you delete master clips it deletes the clip refs in every sequence containing those master clips, and there is still no easy, silver bullet process for removing dupe media. This wouldn't be a problem, except Premiere's dev has still not tackled the project bloat issue, making duplicated media a real problem for large projects.

I am currently working on a feature documentary with 100+ hours of footage, plus graphics, etc. It's me and another editor in another country working on different sections, and it's impossible, STILL, to manage duplicates in a way that doesn't require me cherry picking through 1000+ video files. XML is a solution sometimes, but XML can't be relied upon in my experience for audio sync, titles, subtitles, and even I/O points sometimes.

Kevin-Monahan, you are incredible for the time and patience and grace you show on these forums, for which I thank you, but as much as I love the Adobe ecosystem, I just downloaded FCPX, cuz I'm fed up with this. FCPX!! An NLE I never thought I'd have to place my hope in. Can you speak at all to anything that the dev team is doing to finally solve this issue? I don't know code, but it's something the other NLE's have managed to handle smoothly for years now. I don't want to leave Premiere, but this + the absurd Project bloat that happens (and the accompanying crawl at which Premiere will operate) is a deal breaker in the long run.

Snazzy marketing things like Morph Cut make for a great press release, but I believe I am not alone in wanting the dev team to give us an update that eschews flashy features and focuses on making Premiere a kickass, smooth running beast, because I think everyone here likes enough about it that we want to stick with it.

All this being said, thank you again, Kevin-Monahan‌, for everything you do for us here.

[ebenabbaan]

[Kevin-Monahan]

Hi Ebenabaan,

I understand your frustration, but the product team does not engage customers in this forum, as it is largely user to user based. However, the product team does read these requests. The product team is also a finite number of individuals, so it's impossible to get to absolutely everything deemed "important." Lots of things are important and prioritized accordingly. Sorry.

Kevin

[evcarpenter]

I know I never used to have this problem and I'm not sure why. But what worked for me today was to offline everything in the sequence before I imported it. Adobe imported the sequence and reconnected automatically, no dupes.

[Guest]

Experiencing extreme frustration with this even in CC 2015!!! Have filled a bug report, I really hope this can be addressed soon!!!

Hard for me to convince large productions to make the switch to Premiere when little silly things like this still exist...

[R Neil Haugen]

Hello,

One idea you may not have tried (not a 'fix' but easier!) ... have you tried opening the XML in a new project and then importing the Premiere sequence into the larger project? Immediately 'dump' the new project. It isn't the solution you might want, but might save a ton of work right now ...

Neil

[Guest]

Hey Neil, I have indeed tried this and unfortunately yields the same results

Thankyou for the suggestion though!

[Kevin-Monahan · Accepted solution]

Experiencing extreme frustration with this even in CC 2015!!! Have filled a bug report, I really hope this can be addressed soon!!!

Hi Darius,

Please file your specific frustrations here in a bug report.

Hard for me to convince large productions to make the switch to Premiere when little silly things like this still exist...

You can avoid duplicates in many cases, but you have to start each project with a few things in mind, especially if working in a collaborative environment.

• There will be one "master" project for each production that you are working on. This project is read/write.
• You import other projects from other collaborators via inspecting the project file itself via the master project's Media Browser (you can open multiple Media Browser and save multiple favorite Media Browsers). These projects will be read only.
  • These items can be sequences, graphics, music, etc.,
• Prior to beginning work on the master project, make sure that you enable 'Write XMP ID to files on import' in Preferences > Media.
  • This will tag each piece of media with a unique ID, which Premiere Pro can track.
  • Then the application will recognize duplicate clips and avoid displaying them.
  • If a clip, graphic, or other media reference does not yet exist in the master project, a clip item is imported to establish a master clip.

You can tell the "large productions" that this is the method used by both the Fincher and Coen Brothers editing teams. If you have any questions, please let me know.

See this tutorial for the workflow:

Thanks,
Kevin

[lehestro]

Kevin you beautiful man! THAT is what I've been looking for this whole time! The Write XMP ID preference. At last, a solution! It might have been there the whole time, but there are a lot of people across many forums, including myself, that never knew or could have thought to have used this option.

[jerusalemu11246123]

This tutorial is nice and all, but it does not solve the problem of importing a sequence via Media Browser while NOT duplicating the media that already exists in the target project.

And that's the main issue that many of us working against.

It balloons your project file size and becomes a massive nuisance.

The Write XMP is also a possible solution, but can that be implemented in the middle of a project? It seems that option needed to be checked upon creation of the project.

[Kevin-Monahan]

Hi Jerusalem U,

This tutorial is nice and all, but it does not solve the problem of importing a sequence via Media Browser while NOT duplicating the media that already exists in the target project.

Please describe the exact scenario in which you still receive duplicate media.

Thanks,
Kevin

[Kevin-Monahan]

The Write XMP is also a possible solution, but can that be implemented in the middle of a project? It seems that option needed to be checked upon creation of the project.

Hi Jerusalem,

No, you can't. I'll see what I can do to provide a step by step process for project setup. In the mean time, please file a bug on the fact that you can't implement this preference after the fact of starting a project.

Thanks!
Kevin

[Guest]

KEVIN THIS SHOULD BE DOCUMENTED SOMEWHERE!!

Capslock aside, thank you very much!! As lehestrosaid, this tiny piece of information is what myself and many other editors I know have been looking for. Please for pete's sake, put this information into a KB article somewhere so that others can know!

[rachell63192363]

I am super-confused.

I am currently moving a huge group of FCP7 project files from a longterm documentary project into one Premiere Pro Project. The goal for me is to be able to see all my timelines within the same workspace. This is actually the reason, the only real practical reason I finally made the move.

Am I now, after all this, only to find out that this project (which has about 100 versions from 2008 to 2015 in FCP) will pull clips into the bins cannot be deleted because they reference the clips in the timeline?

THIS will make me insane.

Please, someone tell me there is an easy way out of this.

[R Neil Haugen]

Kevin-Monahan‌ is both an expert on FCP and of course good with PrPro ... let's see if he can be called back in to assist you ...

Neil

[Kevin-Monahan]

Hi Rachel,

this project will pull clips into the bins cannot be deleted because they reference the clips in the timeline? THIS will make me insane.

Have you checked out the tutorial? Did you enable 'Write XMP ID to files on import' in Preferences > Media before beginning the project? 
Make sure you follow that tutorial to the letter and report back with any duplicate clips you have in your project.

Thank you!
Kevin

[Crisis892000]

This is still a major source of frustration for me.

My situation is that I work for a trailer house. We have many editors working on multiple projects with their own ways of organizing their projects. There is no time to make a "master project" in most circumstances because everyone is working at lightening speed. AE's are doing stringouts while editors are working on their own cuts and then sharing them.

I for one am in the minority in that I love that Premiere makes you have master clips in the browser, but as stated above, duplicate clips forced on users via the media browser is one of the most annoying things ever to me, both for project file size and organization. This is a MAJOR issue, and one that is keeping my company from wanting to fully adopt Premiere.

It makes zero sense to me how - with a program that has a media cache database - duplicate clips can happen. Even with the XMP option (which Kevin's posted video tutorial didn't really explain...), I am still having issues with duplicate clips when importing via the media browser.

Despite all of your help Kevin (seriously, you've been rocking with your replies), I don't really feel you fully understand what the problem is here, so here is exactly what happens:

1) Though I am in trailers, we often work with dailies, so thousands and thousands of clips. As an editor, I want watch and make selects ASAP as soon as footage arrives, so I import all of the footage to begin breaking it down. I organize it and load it into my own sequences. But having an edited version of the show is great. So AEs work on what we call Assemblies (which are basically a rough, edited version of the show).

2) When their assemblies are done, I need to bring them into my Premiere project. So I go to their projects via the Media Browser, go to the sequence with the assembly, and import it into my project. And this is where the issue happens...

3) When I import their assembly sequence, Premiere fails to recognize that I already have those Master Clip dailies imported. So it reimports ALL of the ones they were using in their sequence. For a feature film, this can be thousands and thousands of duplicate Master Clips.

4) If I now want to copy and paste shots from their imported assembly sequence, since the new, imported sequence connects to the now duplicated Master Clip, I will need to keep both the original and duplicate Master Clips in my project.

5) Imagine now having to import multiple assemblies over the course of a project as they get updated, or cuts from other editors, or cuts from the production house via XML. Each time you import a sequence, Premiere makes a new master clip whether you have already made one or not.

6) Then the situation becomes COMPOUNDED. If I am using a sequence that has the same clip twice but somehow references two master clips, and then Joe to the bay next to me needs to import my sequence, when he imports my sequence referencing two master clips, he now has two Master Clips import IN ADDITION to what he himself had loaded into his project. Then Joe works on it and needs to send it BACK to me. Now everything gets duped again! By the time a project is done, we are talking about some clips having as many as 20, 30, even 50 duplicates... If you are working with 3000 daily shots (not including duplicate GFX Master Clips, SFX Master Clips, or music Master Clips), preventing duplicates is ESSENTIAL.

This is still a major problem. I too have written to Adobe about it.

I love Premiere, but if I had to fix ONE issue out of about 20 user suggestions, this is the TOP of my list.

[Crisis892000]

To add to my post from last night, sometimes I can indeed import sequences without it bringing all of the files and duping them, but often time which files it chooses to dupe is quite random. And still quite annoying.

One thing I have learned is that (naturally) if an editor has retitled a clip anything (or even added an addendum like "Best Take" at the end), the clip will dupe. I am usually steadfast to keeping filenames what they are on import, but some editors like to change them, and I can't fault them if that works for them. One thing that would help this is if Premiere implemented a "Change Clip to Match Filename" feature like Final Cut had. Then I could bring a sequence into a third project, rename all of the master clips to match the filename, THEN bring the sequence into my project and avoid many dupes.

One example of this happened last night. An AE did an assembly in Final Cut, and though he didn't change any names, in FCP 7, it does not include the file extension as part of the clip name. So when I imported the XML, all of the files were duplicated. I even tried to bring the XML to a third project, then import that sequence into my project, but the files still duped.

So to amend this, I had to go into the third project, import the XML, and manually add ".mov" to over 300 clips...

But even then, I still had about 30 clips duplicate for apparently no reason. Can't tell you why. Filename, everything matched.

One workaround to this would be to say something like "Relink to other master clip." You know you can tell Premiere to relink a clip media on an HDD? Well why not be able to say to Premiere, "Any clips that are reading this master clip, have it read THIS master clip instead." Kind of like relinking INSIDE of the project file.

[Kevin-Monahan]

Hi,

I'm sorry that I don't have much time to test all your scenarios this moment, but I will once I return from NAB.

One thing an engineer suggested you to check is the "Allow Duplicate Media During Project Import" in the panel menu of the Media Browser. Experiment with disabling that setting to see if it stops or reduces the duplicates that come into your project via other projects and sequences.

Of course, this assumes that you import footage via the Media Browser because you are adhering to the "master project" workflow. All child project items must enter the project via the Media Browser in order to respect the preference.

My hunch is that this preference gets by a lion's share of editors because they avoid the Media Browser like the plague (I have found). This is really a shame because it's the key to better media handling and management.

You said, "We have many editors working on multiple projects with their own ways of organizing their projects. There is no time to make a "master project"

My feeling is that this is the main reason behind your struggles with Premiere Pro. I hope to help you figure out a more reliable pipeline to avoid this issue. Once editors adhere to those rules, things should go much smoother. Often, this takes a case study as not everyone has the same needs. The remote workflow pipeline might be different than the trailer house workflow and would require two separate designs.

"Any clips that are reading this master clip, have it read THIS master clip instead." Kind of like relinking INSIDE of the project file.

One comment is that you should look into our new proxy workflow as it may hold keys to a newer way of working between you and your team. You as the editor control the project, you ingest masters and create proxies. Have editors go hog wild on editing proxies on cheap laptops. Gather child cuts into the master project. Relink to high res masters. Conform. Output. Maybe that would work?

Thanks,
Kevin

[ManniquenMan]

years later and this is still an issue... Thanks Adobe, for considering all of our needs and then just ignoring them.

[Kevin-Monahan]

Hello Crisis,

I don't really feel you fully understand what the problem is here

Doing my best for you. I will try and get you some assistance but most of the team is in transit to the NAB conference. It may take some time to get back to you regarding solutions, but just know that this issue is a top priority for me personally.

If you will be at NAB, feel free to seek me out personally and I will put you in touch with an appropriate staff member.

Thank you,

Kevin

[RyanPatch]

Hey Kevin -

Crisis is obviously struggling with this issue right now, but this is something that everyone has been seeing and continues to see.  It's improved in last couple years, but as Crisis says, many times some clips just dupe when they come in for no reason.  It's frustrating.  I hope that the solution is not a work-around, but a robust update to the way that files are handled upon importing a sequence / project.

R