P: Include additional metadata in XMP (flags, collections, VC's, etc.)

Report · May 22, 2011

Hi,

when letting Lightroom write all the picture settings to a xmp file, both the stacking and the collection settings are missing.

Basically, I'd expect to find just every work done about the picture in the xmp file (e.g. for use with other tools. If I put several pictures in collection, I'd like to use that information and the order of the pictures from other programs.)

Even worse, when making a virtual copy of a picture, it's settings do not appear anywhere in an xmp file.

regards

Report · May 22, 2011

Is this a problem or are you requesting additional functionality? It sounds like you'd like metadata about collections/stacks stored with the file.

Report · May 22, 2011

Collections, stacks, VCs, flags and Develop history do not appear in the XMP data. There are application interoperability reasons for this, but I think they should work through as much of this as possible.

Report · May 22, 2011

Yes, I'd like to see metadata about collections and stacks (and their order) stored in the xmp files.

Whether this is a bug report or a wishlist depends on the point of view, i.e. the formal specs of the xmp file (if there is a precise specification).

If the xmp file is intended to contain _all_ information about an picture file, including all work, then this is a bug.

If the xmp file is supposed to contain only some parts of it (why?), then this is a wish.

The reason I am asking: I am currently (still) preferring Bibble5 over Lightroom, because Bibble5 runs under Linux as well and has several functions missing in Lightroom. On the other hand, Lightroom seems to produce more reliable results and to have a more complete workflow. I do consider changing to Lightroom (have a license).

A major problem of lightroom seems to be that it is based on a central database that cannot be stored on a network share. This makes it impossible to concurrently work on a large collection of pictures due to the synchronisation problem. Bibble5 solves this problem by having a so called „File System Mode”, where just everything is stored in xmp files, so you just need to synchronize file systems with tools like rsync or unison, which perfectly allows to work simultaneosly on the same collection of pictures as long as not on the same pictures.

I tested whether it is possible to do the same in Lightroom by writing out the xmp files after work and synchronizing before. Unfortunately Lightroom's xmp files are somewhat incomplete.

Another point is that complete xmp files allow to read (and change) all settings e.g. from scripts or other programs from any copy of the file system tree.

It's a feature of Bibble5 that I miss on Lightroom and which currently keeps me from changing.

Report · May 22, 2011

Writing more metadata back to the XMP is desirable, and I'd also add custom fields' metadata. However, I wouldn't justify the request on the basis of concurrent working over a network. Rather than a file system kludge, better to have multi-user network access to the database.

Report · May 22, 2011

First of all, having all relevant data in the XMP file is what gives the XMP file it's sense.

In contrast, having multi-user network access to the database does not always make sense. You're assuming that there is a high speed and low cost local network, which is not always the case.

It must be possible e.g. to take a notebook computer with a copy of the file system tree with pictures (or part of it) onto a trip, maybe in foreign countries without mobile coverage, and to work on pictures.

Then, it should be enough to synchronize just the xmp files (oder send them by email) over expensive or low bandwidth lines.

It should also be possible to send some pictures together with all settings to other people (e.g. by email) without having to completely export and import a Lightroom catalog. Or letting other people do the work of image processing and adding keywords and other stuff, and just send/sync back the xmp files.

This boils down to the central problem, that Lightroom is based on the assumption of one central database, that's always online and available. This hold's true for people sitting in agencies, but not for people travelling around (or preferring other operating systems) or participating in distributed working structures (e.g. freelancer).

Report · May 22, 2011

The database is Lightroom's great strength. When a network isn't available, offline workflows are addressed by File > Export as Catalog and Import from Catalog which does things like replicate file structures and has intelligent ways of roundtripping work.

Your mentioning of sidecars for emailing isn't right on three counts. You could send a catalogue just as easily, fortunately not all file types have sidecars - eg DNG, TIF, JPEG, - and thirdly you'd have to transmit the much larger originals in the first place.

So while I agree with your basic request, concurrent working isn't a justification for it.

Report · May 22, 2011

The way you describe it, it does not sound as if the database was Lightroom's great strength, but the dependency of a central database it's major weakness.

Exporting a Catalog does not work that way because it implies that the one working on the database knows exactly which pictures are to be exported.

E.g. I can easily synchronize a part of the tree from my server onto my notebook when I am at home, can then be on the road and do some changes, and then do a fast sync of the changed XMP files over UMTS. e.g. use the time for working when sitting in the train or at the airport.

Argueing that my mentioning would not be right does not make sense, because that's how I am working right now with Bibble5. Perfectly fulfills my needs.

Therefore, it would not be much of a problem to transmit the larger originals in the first place. It's about not having to transmit them again once done with the work.

It is also not about concurrent working (which can basically be done with a central database), it is about distributed working (which requires better synchronization methods than a central database or import/export of whole databases) and data exchange.

Just have an example:

Let's have a database with pictures A,B,C,D. And two workers with notebooks on the road. (Or myself with two distinct notebooks.) One changes A and B, the other works on C and D.

With import/export you get an extreme mess of files and versions. With those sidecar files and a regular file synchroniser you solve the problem fast and easy. And you solve it automatically, without having to manually (!) import and export files.

Report · May 22, 2011

In case this point hasn't sufficiently landed:

Many of us would *really* like an xmp sidecar option for all file-types, as well as virtual copies.

...the completer the better...

Report · May 22, 2011

Hadmut - I know most everybody who cares about the issues you have raised (not everybody does) wants Adobe to solve this problem in a native fashion, including me. But, this could also be solved by plugin - mostly solving the problem of missing data in xmp, and providing xmp sidecars for all file types & virtual copies, and also providing a simple synchronization option to avoid the issues with using catalogs for this purpose, which you have raised. Obviously the plugin solution would be Lightroom-only.

I won't go into details, but suffices to say the plugin-based solution would address most of the issues you've raised, including filling in most of the missing data (some but not all stuff that is not in xmp *is* accessible via plugin), and catalog synchronization - for those willing to use the plugin, that is.

My hopes in order of preference:
-------------------------------------------
1. Adobe addresses this natively.
2. Some other plugin author addresses this.
3. I find time to address this by writing a plugin.

Final thoughts:
===========
For those beginning to worry about a plugin adding data to Adobe-written xmp files, I imagine a separate file would be written to keep it clean, with a different extension.

-R

Report · May 23, 2011

You really are deploying some straw man arguments! And if you want to change the goalposts from concurrent working (the phrase you originally used) to distributed working, Export/Import are equally suitable and provide the level of controllability that's needed. Your Heath Robinson / Rube Goldberg workflow processes may work for geeks and can be made to work by normal users, but they're still ugly and still involve manual processes and knowing which files you're working on. Lightroom should be about doing the right thing.

Report · May 23, 2011

Hadmut, FWIW, I don't see you making any straw man arguments at all. I think your proposal makes perfect sense. I suggested something similar a while ago: I believe that XMP files or metadata in DNG files should retain edit histories. Dan Tull was supportive of the general idea and regards complete XMP files as a distributed catalogue backup.

Report · May 23, 2011

@john beardsworth: That's pure nonsense, and it does not get any better by using technical terms not understood by everyone in this forum.

Export/Import do provably not maintain a proper and synchronized data structure. A first problem is, that they generate additional data sets, i.e. the exported files. So besides the heap of pictures I have, I would additionally have to deal with all those exported files and to keep track with them. This will definitely fail (and waste additional disk space I might not have when on the trip).

A second problem is that Lightroom works under Windows and MacOS only. It requires someone to sit in front of the computer and move a mouse, and to have that particular computer with Lightroom installed turned on and someone logged in. This is a nightmare. It does not work with a central repository. E.g. there are lots of free/cheap disk space/backup providers out there, but there is no one providing a free online version of Lightroom. The requirement to have a central database entity operated manually by a human through a graphical interface is a severe failure in design.

A third problem is that this method does not deal with change sets. Imagine that I am at home at my personal network, and do synchronize my heap of 100,000 travel pictures onto my notebook computer. Then I am on a 3 weeks trip with lots of time to spend in trains and airplains, and I do some corrections, b&w conversions, add keywords and so on. How would I export from the database all settings that I have changed and not just all the 100,000 ? And how do I import them into my central database when I am away from home and thus from my central database?

Lightrooms structure of dancing around a central database on a Windows machine that forces users to operate it centrally with a human graphical interface and mouse is based on assumptions that do not hold true.

I've changed the term from concurrent to the more precise special case of distributed, since someone in the comments above reduced "concurrent" to the special case, where all concurrent entities have high-speed low-cost access to a central database. Photography is not centralized work. Not anymore. It used to be in the old days, when we had chemical labs. But today we have notebooks and mobile phones, and we leave our office for taking pictures and communicate with others.

Geeks: Using one of the common tools to keep a directory tree of regular files in sync between computers (or computers and online storage providers) is definitely more simple, comprehensible, fool proof and robust than dealing with an unstructured collection of exports, that require manual interaction and most probably lots of manul corrections and collision solving.

That discussion sounds, btw., somewhat silly, because Lightroom basically already has the required functions, i.e. writing xmp files and reading them in through synchronizing directories with the database. So what's the point of this discussion? Denying that Lightroom already has it? It's just that it is incomplete, inconsistent, not complete, and poorly implemented.

But I think we can stop discussion right here. I think I'd rather stay with Bibble5 instead of turning to Lightroom and felting my work through inconsistent databases. The guys from Bibble Labs have realized, addressed, and solved the problem. The guys from Lightroom don't. Bibble5 solves the problem, Lightroom doesn't. Sorry to say that, but it's really that simple.

Report · May 23, 2011

Hadmut - You may not be able to convince everyone, but you've convinced me.

Networked Lr with true-sync... - may be a while off, but syncing via xmp would be a short reach.

I'm hoping Adobe will remedy come Lr4, or at least augment the SDK so a plugin author could implement the solution (its "almost" doable now as plugin but would be severely limited due to omissions...)

Is this really a deal breaker for you?

Rule 5.2 - Enjoy other software if Lightroom not quite cutting it...

Rob

Report · May 23, 2011

@Hadmut - I'm not using technical terms or jargon, and your reply is really too long to go through countering or agreeing with every single point.

Note, I am not talking about simple Export/Import but specifically used the terms "Export as Catalog" and "Import from Catalog". These don't need to generate additional originals - they can, if wanted, but don't need to.

"I would additionally have to deal with all those exported files and to keep track with them." No, Export as Catalog/Import from Catalog can work without passing extra copies of the file back and forth. The process just needs a slightly more user-friendly appearance / metaphor / name to help people appreciate its elegance.

"Second problem...". Is Win + Mac only really a problem? Not sure what you're on about here, but a database is a huge advantage, not a "severe failure".

"Third problem..." Export/Import is actually a lot smarter than you seem to think! In your example I just do a smart collection based on the edit date, export those items as a catalogue (without the originals), and import that to the master. But there are a few easy ways to do it without playing with sidecars.

I just don't think your concurrent/distributed workflow is a reason for the FR. LR offers better ways to work.

John

Report · May 23, 2011

In relation to distributed or concurrent working, they sure are!

The same arguments that we covered in that thread apply here - these are areas where if the FR is ever implemented, users need control over each chunk of data that's being saved out. For example, if I sent pictures to another Lightroom user would I always want them to see my collections structure, revealing how I organise my work or other confidential information? And would they want to have my collections appear in their well-organised catalogue? The same would apply to stacking or what virtual copies I've made. So you need write and read options - or be happy with the unintended consequences.

Report · May 23, 2011

Hadmut - it does sound like you may not fully appreciate how to use Catalog Export / Import. Its just one file, not a bunch of files and could essentially contain all the changes you'd have made on the road, as example, not a whole catalog, and no photo files... I'm a big proponent of complete xmp for all+virtual, for a variety of reasons, including the ones you've brought up. But it may still be possible for you to accomplish your objectives without too much trouble using catalog export / import. I recommend making sure you do understand how that works before making your final judgement... I mean, I'm not vested either way, and I'm not an Adobe/Lightroom defender, and I don't always see eye-to-eye with John Beardsworth, to make an understatement, but I do think in this case he's brought up some points worthy of a bit more consideration.

Report · May 23, 2011

I don't see much difference in sending someone "complete" xmp files to read into a foreign catalog versus sending them a catalog fragment to import into a foreign catalog - theoretically it would be the same info in a different wrapper. But some options for what goes out and what comes back in doesn't seem like such a bad idea, in either case...

Report · May 23, 2011

rob@edidev: This is exactly the problem. Or more, it is a bunch of problems.

First: it is _one_ file. So you always run into the collission problem for the whole export and not just for single pictures.

Second: it is an _additional_ file. I do not just have my file system tree of pictures, I also get a large heap of these export files, which I have to keep in order and organized.

Third: It requires manual interaction and thus causes lots of overheads. File tree syncing is cheap and automated, I'll to it anyways, and most business machines do it automatically without the need of user interaction. In contrast, a Lightroom export does not do anything at all automatically. Someone must sit in front of a Windows machine, start it, use the mouse, and must do the import manually.

Fourth: It's foolproof. If you have two machines with the same files, you know they are synchronized (or you can see, where they are not). If you have to machines with Lightroom databases, you never know whether they are in the same state or where they differ. Even if you have exactly the same contents, database files are never similar.

Sixth: It is not about accomplishing my objectives "without too much trouble using catalog export/import" I need to accomplish my objectives without the need to start Windows, run Lightroom manually, check all that things, and never be sure as if everyhting happend as expected, just for my daily notebook sync/update.

Don't you understand that the idea of having Windows as an operating System and Lightroom as an interactive program be started and operated with the mouse for every single synchronisation itself is the misconcept, no matter how smart it might be?

You need a way to synchronize
a) automatically without user interaction
b) reliably and in a way that can be verified
c) fast and over cheap/low traffic lines
d) without mid-air collissions and conflicts if several people (or just me working at home and on my notebook) change different files
e) work with dumb storage like hard disks, USB sticks, online storage without the need to have running instances of Windows and Lightroom

That whole concept of needing a proprietary, user interactive, license bound program depending on a graphical interface and human interaction to just synchronize machines, even worsened by the fact that Lightroom does not allow catalogs on network shares, is broken by design.

Furthermore, it is based on the obsolete idea of fast PCs or Workstations, but today we have a client/server/central storage/distributed work model.

And the more comments I read in this thread, the more I suspect that this is made intentional to increase the number of Lightroom licences customers need.

And for those who want to tell me that this is wrong or would not work: Why does it work so well with Bibble5 then?

Report · May 23, 2011

Fair enough Hadmut.

I just wanted to make sure you understand how the catalog export/import works so at least you are making an informed decision.

Your points are noted...

Report · May 23, 2011

Hadmut, FWIW, I see your points and fully agree. Manual fiddling with exporting and importing (sub-)catalogues is just too cumbersome.

You could have added that

a) LR catalogues are huge. They often compress to 10% of their size using WinZip. If you compress them, further steps are introduced. If you don't, they take up a lot of space and require a lot of bandwidth to transfer.

b) LR catalogues are (not yet?) a paragon of stability. Now and then they become corrupt. Losing edits this way is not necessary if edits are saved to XMP files. Lee Jay once wrote: I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup).

Report · May 23, 2011

Hadmut, FWIW, I see your points and fully agree. Manual fiddling with exporting and importing (sub-)catalogues is just too cumbersome.

You could have added that

a) LR catalogues are huge. They often compress to 10% of their size using WinZip. If you compress them, further steps are introduced. If you don't, they take up a lot of space and require a lot of bandwidth to transfer.

b) LR catalogues are not (not yet?) a paragon of stability. Now and then they become corrupt. Losing edits this way is not necessary if edits are saved to XMP files. Lee Jay once wrote: I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup).

Report · May 23, 2011

..and if a catalog is broken, all is lost. Where in contrast xmp files are much more robust since they are not updated with a complicated file structure, indexes and that, they are just plain files, and if they are corrupt, it's just the editings of a single picture, not the whole database.

Report · May 23, 2011

How long/much have you been using Lightroom? You really do need to look at lot closer at how File > Export as Catalog works, and at its Import from Catalog counterpart. They're the key to slick and controllable multi-machine workflows.

And for an example of a straw man argument... "If you have two machines with Lightroom databases, you never know whether they are in the same state or where they differ." Of course you do - you just have to use your brain! By which, for instance, I mean naming one master.lrcat and the other laptop.lrcat. So not a lot of mental effort.

John

Report · May 23, 2011

Well, that is Lee Jay's way of working where he doesn't use a range of LR features because they don't get written to xmp! Database corruption is very rare indeed and people are best advised to backup up their catalogues in the confidence that they are reliable!

Report · May 23, 2011

"and if a catalog is broken, all is lost. "
Another straw man? Proper catalogue backups mean all your work is backed up and easy to restore. And only raw files have sidecars - other file formats contain embedded XMP and are not "just plain files".

Adobe Community

P: Include additional metadata in XMP (flags, collections, VC's, etc.)