P: Add a Camera Raw Option to Prevent Writing Back into Input Files

[Noel Carboni]

Given:

Under some conditions Camera Raw writes data back into (overwrites) its input files. Example: A JPEG file.

Under other conditions Camera Raw maintains this same kind of information in a separate place (e.g., Sidecar XMP or central database). Example: A CR2 or NEF file.

Assuming one uses Camera Raw to open out-of-camera original files as many photographers do, depending on what mode one has captured the images in, Adobe is inconsistent about whether to keep its hands off them or overwrite them... This seems to be because some formats are proprietary and some are well documented. From a programmer's perspective, this makes perfect sense.

Trouble is, from a user's perspective, this behavior cannot be described as anything but inconsistent.

Personally, I do not want my original out-of-camera JPEGs updated/rewritten under any circumstances.

Camera Raw will not touch a proprietary raw file, such as a Canon .CR2 or Nikon .NEF. There's a whole process for remembering settings in a separate database or sidecar XMP files. So far so good.

However, if you open a JPEG, TIFF, or DNG through Camera Raw, data WILL automatically be written back into it to tell another run of Camera Raw in the future what settings you used - without the software ever having warned you it will do so.

It is true that some functions EXPLICITLY rewrite input files. You can ask the software to write new thumbnails back into DNG files, for example. This seems fine - the user has instructed the software to overwrite the file, and the user is in charge, after all.

Overwriting/rewriting an input file without being instructed to do so is NON-INTUITIVE BEHAVIOR for any application. Simply put, I would not expect an input file to be overwritten by Camera Raw.

And we do see that it causes people confusion and surprise from time to time. You may right now be reading this in disbelief. I recommend you go test it for yourself (on a copy of one of your original JPEG files).

The original file being overwritten is a chief reason why I don't configure Photoshop to open my JPEGs through Camera Raw.

Adobe:

Please give those of us who don't want our input files overwritten an option for using the database/XMP sidecar instead in EVERY case.

Thank you.

-Noel

[TheDigitalDog]

I believe that because Adobe feels proprietary raws are read only, the alternative is sidecar files. All the other files are read/write, non proprietary files. In that context, the behavior makes some sense. Is there a reason you would not want the metadata data written into the file format itself?

As for saving data without asking, isn’t LR doing this all the time? Well to the database (or the data itself if so set).

[Noel Carboni]

Hi Andrew,

Thanks for commenting. Yes, that's the reasoning I derived as well.

My point is: Yes, there is a reason I would not want the metadata written back into the file itself: I simply don't want changes to my original out-of-camera files, however trustworthy the application. Yes, I have them backed up several ways.

As it is, I can accomplish this by just not using Camera Raw on any formats that it feels comfortable doing such write-backs to. This makes sense in another way in that Camera Raw was designed for, well, raw files, not JPEG or TIFF or PSD or whatever. The line grays a good bit when we talk about DNGs.

I'd simply like to be able to configure Camera Raw to use the metadata database or XMP file in ALL cases via a configuration option. Then it would be consistent in its operation, and I would not need to worry about whether I'm opening an original JPEG with Camera Raw.

Someone else will have to speak for Lightroom. I don't use it.

-Noel

[areohbee]

Speaking for Lightroom: two thumbs up, way up. And in the case of Lightroom, support for virtual copies would also make sense, big sense...

Thanks for posing this Idea Noel.

-Rob

[areohbee]

To Forum Administrator: please add 'Lightroom" designation to title, so Lightroom users know that this applies to them too - thanks.

To be clear: this idea is about not writing back to original camera files, and handling all photos consistently, by using an XMP sidecar for all file types (and virtual copies), or possibly in a common database if not Lightroom environment.

[Noel Carboni]

Thank you, Rob.

Yes, I'm envisioning the use of XMP sidecar -or- central database, EXACTLY as is done now for raw files that cannot be written back into.

-Noel

[areohbee]

You bet, Noel.

The common objection to this 'Idea' seems to be the argument that sidecars are a workaround for proprietary raws, but some of us would prefer sidecars regardless, or in your case the central database.

I think the main barrier to adoption is that Adobe is trying to make in-file xmp a selling point for DNG.

Technically, this would be nearly trivial to implement.

I'm hoping but not expecting...,
Rob

[RobertJG60]

Lightroom needs to be patch so that xmp files can be generated for use with dng files.

dng is not a valid raw choice unless I can treat it like any other raw file. I can't.

Our raw files are archived on dvd (read only). the edit data is archived as xmp in another location. I can use acr to force the xmp files after the edit but it would be better if lightroom didn't arbitrarily refuse to do it. (It did it in beta).

I'm not interested in hearing from anyone who wishes to argue that eliminated xmp files is a good thing. It's isn't. They're tiny and they can be used to recreate an edit long after the dngs have been removed from a catalog.

Did I mention that our raw files are archived in read only format before they're edited? They're received from photographers on dvds and those discs become our archive.

[TheDigitalDog]

>dng is not a valid raw choice unless I can treat it like any other raw file.

Why not?

[Noel Carboni]

Andrew, I suspect the difference Robert is sensing is the core concept of this thread - that the Adobe software feels comfortable in writing back into the DNG file, while proprietary camera raw files are "hands off", but that's just a guess.

Some people really want their raw files to be untouched. This is not a difficult concept.

-Noel

[TheDigitalDog]

Adobe feels comfortable writing back to lots of non proprietary file formats and uncomfortable writing back to proprietary formats they don’t control. That’s not a difficult concept for both legal and other reasons. Since there are oh so many proprietary camera formats that continue to come onto the scene every time a new camera is produced, the idea makes a bit of sense. You’re ‘stuck’ with sidecar files. But none of the above explains why DNG isn’t a valid raw choice. Its a different chose but not valid?

[Lee Jay]

He did explain why not. He has read-only DNG files and would like to archive just the metadata in the XMP files, which are a few orders of magnitude smaller than the image files. That would represent a substantial savings in both space and time especially since he has the image data archived already.

[Noel Carboni]

I guess the question I didn't have answered before was "what does the software do when you have a read-only DNG file?"

I just did some experimentation on some proprietary raw files, along with JPG and DNG files, making some of the DNG and JPG files read-only (but the folder still read/write), using Camera Raw 6.4.1, and here's what I found...

When configured to use the central database, Camera Raw did NOT attempt to write back into the DNG files, and instead went to the central database to store/retrieve conversion parameters. Consistently, it used sidecar XMP files in the same folder as the DNG files when configured to use sidecar files instead of the central database.

It emitted an error when Camera Raw tried to write back into the JPG files unconditionally.

Clearly the DNG files are being treated differently than JPEGs. This may answer Robert's point and it does make Andrew's question valid...

Robert, have you actually TRIED using DNG files for the task, given your processes?

-Noel

[TheDigitalDog]

http://forums.adobe.com/message/3245182:

>It would be great to have an option whether to imbed XMP metadata into DNG file (as it is write now), or create XMP sidecar files as Lightroom does for proprietary RAWs.

Answer why its not useful:

Because it's half-baked in backup terms - you can't simply dismiss a swathe of LR work as "image relationships properties" - and it forgets that one valuable aspect of DNG is precisely that it is designed so you can safely write metadata to the format (though you'd be on your own if you do so to your only copy of the image....). Development time spent on sidecars for DNGs (JPEGs too, and why not TIFs?) would be unavailable for more important enhancements.

John Beardy

Uodate: Since you changed your post while I was writing.....

'And by the way, ACR can be forced to act like this. If you mark your DNGs as "read-only" ACR will not touch them, but will create sidecar XMPs.
While Lightroom just removes "read-only" attribute and overwrites original file without shame'

Then use ACR..... LR's ethos is to start from scratch, not slavishly replicate earlier tools.

[RobertJG60]

Interesting. The reason is built in to the sentence that you commented on yet you ask why not anyway.

It can't take the place of what I'm using if I have to treat it differently.

Geez

[RobertJG60]

Writing the data "safely" has never been a concern of mine. I find the xmp files to be vital since I don't have the option of writing the metadata back to the dng (them being read only)

More than once, I've been able to recreate an edit of an event using the raws and the xmp file which allowed me to pick up where I left off.

If the originals were dng, this would have been impossible. I would have had to re-edit the job from scratch.

Fyi, Lightroom 3 doesn't remove the read only, it returns an error writing metadata.

And I haven't hit on a consistent set of changes to make and change back. Apparently, acr thinks some changes aren't worth saving.

[RobertJG60]

My point has nothing to do with jpgs or tiffs. Only with the fact that lightroom needs to have an option to use xmp files with dng at my discretion, not Adobe's.

[TheDigitalDog]

Now all you have to do is convince Adobe why.

[Noel Carboni]

Yes, an option that would FORCE the use of external storage for development settings is exactly what I'm requesting in this thread. And this option DOES need to affect all file types that can be opened by the raw processors (Camera Raw / Lightroom).

For those who seem to want to oppose this request, note that all I am requesting is an OPTION, not a permanent change to the operation of the software. Thus, you get to keep working as you do now, but the products become more useful for people like Robert and me who need to keep it from writing back to original files.

As far as convincing Adobe goes... What part of "applications shouldn't normally write back to input files" don't you understand? What they're doing now is inconsistent and wrong!

-Noel

[RobertJG60]

Yes, despite our requesting actual camera raw files, some photographers have been hoodwinked into adopting dngs on the off chance that some archeologists excavating his studio in a thousand years might want to look at his images.

[TheDigitalDog]

Or in less than 10 years, they might actually want to access the data (which today I can’t do with 10 year old PCD and DCS raws).

[Noel Carboni]

Can't open a PCD? Why is that? Because Adobe dropped support of the PhotoCD import plug-in from their latest products? Don't you have an older copy of Photoshop still running (or have done as I have and made the old plug-in work with the latest Photoshop)?

So the question becomes, when will Adobe drop support for DNGs (or just stop making new software or go out of business)? Less than 10 years from now? Open standard or no, it's not like it's been embraced by anyone else.

-Noel

[TheDigitalDog]

>Because Adobe dropped support of the PhotoCD import plug-in from their latest products?

The Plug-in was Kodak’s, it doesn’t run in Lion. The standalone Kodak software for DCS process is even older and requires OS9. So while its not impossible today, it requires hardware and OS’s that I don’t own any longer.

Even if Adobe did drop support for DNG, its an openly documented format. Anyone who wanted to continue to render that data could. DCS files and probably PCD were not.

[Noel Carboni]

Ah yes, I forgot about the discontinuity in the Apple legacy. Thanks for the info.

I suppose it's up to whomever has archival data to occasionally review capabilities and convert to a more modern, still-supported format as needed (e.g., when you still DID have PCD or DCS support, you should have converted all your images to TIFF).

I can see that it probably wouldn't hurt, if sensing an impending discontinuity, to make DNG in addition to keeping your proprietary camera raw files then. But that doesn't mean it has to be right now.

[TheDigitalDog]

Right on. And yes, TIFF is a smart move if only based on history. TIFFs (an openly documented Adobe owned format) and PSD (not an open format but one still around thanks to Adobe’s longevity) I created in 1990 in Photoshop 1.0.9 and can be opened today in CS5 among other products. The nice thing about DNG is at least there’s a pretty high quality JPEG embedded of the current rendering if so set and desired. And if the history of imaging is any indication, a betting man will probably find DNG will hang in there a long time like TIFF and PSD. But there are no guarantees.

I have lots of legacy, proprietary image files that are now saved on what amounts to drink coasters. FITs files from Live Picture, whatever Xres saved, whatever College saved etc. Of course I could render them as TIFFs and probably did if the image was important to me. I could do the same with raws but would rather not, considering them digital negs that improve with age. Improve in terms of new technology such as PV 2010 versus 2003. I fully expect to see Adobe improve their raw processing technology so my preference would be to have a raw format that has the best chances of being processed in the future. Again, based on history and having been burned more than once, DNG seems to be the good bet here.

[RobertJG60]

Because I'm the customer and they do things because of me, not themselves.