P: Add a Camera Raw Option to Prevent Writing Back into Input Files

[Noel Carboni]

Given:

Under some conditions Camera Raw writes data back into (overwrites) its input files. Example: A JPEG file.

Under other conditions Camera Raw maintains this same kind of information in a separate place (e.g., Sidecar XMP or central database). Example: A CR2 or NEF file.

Assuming one uses Camera Raw to open out-of-camera original files as many photographers do, depending on what mode one has captured the images in, Adobe is inconsistent about whether to keep its hands off them or overwrite them... This seems to be because some formats are proprietary and some are well documented. From a programmer's perspective, this makes perfect sense.

Trouble is, from a user's perspective, this behavior cannot be described as anything but inconsistent.

Personally, I do not want my original out-of-camera JPEGs updated/rewritten under any circumstances.

Camera Raw will not touch a proprietary raw file, such as a Canon .CR2 or Nikon .NEF. There's a whole process for remembering settings in a separate database or sidecar XMP files. So far so good.

However, if you open a JPEG, TIFF, or DNG through Camera Raw, data WILL automatically be written back into it to tell another run of Camera Raw in the future what settings you used - without the software ever having warned you it will do so.

It is true that some functions EXPLICITLY rewrite input files. You can ask the software to write new thumbnails back into DNG files, for example. This seems fine - the user has instructed the software to overwrite the file, and the user is in charge, after all.

Overwriting/rewriting an input file without being instructed to do so is NON-INTUITIVE BEHAVIOR for any application. Simply put, I would not expect an input file to be overwritten by Camera Raw.

And we do see that it causes people confusion and surprise from time to time. You may right now be reading this in disbelief. I recommend you go test it for yourself (on a copy of one of your original JPEG files).

The original file being overwritten is a chief reason why I don't configure Photoshop to open my JPEGs through Camera Raw.

Adobe:

Please give those of us who don't want our input files overwritten an option for using the database/XMP sidecar instead in EVERY case.

Thank you.

-Noel

[RobertJG60]

Noel Carboni asked:

Robert, have you actually TRIED using DNG files for the task, given your processes?

Yes, some photographers insist on sending us dng even though we've asked for camera raw direct for the camera without any changes.

I can either use the Bridge/ACR method of forcing xmps or burn new dvds using the edited dngs.

I'd rather just save the xmps.

[RobertJG60]

So, you don't' have any of the apps that you used on those files 10 years ago? Look around, I'm sure you can find them somewhere.

[TheDigitalDog]

>So, you don't' have any of the apps that you used on those files 10 years ago?

Sure, they just can’t be run! Are you a Mac user? Can’t run OS9 on Intel hardware. Can’t run Rosetta under Lion.

You have 8 track tapes? Great. You have a machine that can play them? No, SOL. Got any files on Syquest drivers or floppies but no hardware to access that data? SOL.

And yet, TIFFs I created in 1990 in Photoshop 1.0.7 can be opened TODAY on the latest hardware and OS. In dozens upon dozens of different software packages. Thanks to it being an open, non proprietary format. DCS files, PCD files, hosts of others? No such luck. Not good, especially when pixels I created 22 years ago the same way are accessible. Pixels I created 10 years ago are not. DNG is the TIFF of raw data (its a variant of TIFF). Based on the history, I’ll stick with that for archive of my precious raw data for obvious (to me) reasons.

Sticking with proprietary raw, at least in several examples for me, is like having a 4x5 B&W neg and no enlarger or silver paper to make prints.

[Guest]

My backup solution does not allow for differential backups... every file that is modified even one bit has to be saved again (which takes about a minute). That is a pretty big issue, I can't just go back and update a couple of tags. Sometimes even regenerating the thumbnails as I moved the catalog location meant I had to reupload a huge portion of my library (taking a few months, which meant had I lost my drive many files would have been lost!). In an ideal world we don't need sidecar files. In reality it is a useful thing for some of us, so why not give us that option?

[Guest]

I don't mind the default behavior, it's just that for some use cases it makes much much more sense to use sidecar files. A here be dragons warning that tells the user what the option is for, and what the risks are behind it should be sufficient. It's like Apple telling iPhone developers that they can not use the volume buttons to take photos, because it confuses the user. Yes, perhaps, but banning a very successful application from the App Store just because there is a hidden website that users of this program can visit, so that they can activate that feature? I assume when they go through such lengths they very well want that feature and know of the consequences! I don't even mind editing a config file hidden somewhere on the hard disk.

[FeedbackCommunityMember]

Camera Raw will write sidecar .xmp files for DNGs if the DNG file is readonly. (I use this in my workflow as my main camera is DNG native and I do not want my raw originals getting modified.) I'm not sure if this works in Lightroom, but if not, it could be regarded as a bug and just fixed. This is a small step toward giving users full control over where metadata is stored, but it can help quite a bit.

There is certainly awareness within Adobe of the workflow issues discussed here. I agree that embedded XMP causes pain for some workflows and the lack of user control is regrettable. Unfortunately the easy changes we can make look to introduce more problems than they solve and it is hard to do something winning in just Camera Raw and Lightroom as a strong selling point of XMP is that it interoperates well across a vast range of products from Adobe and other vendors. If the other apps don't also respect the constraints, the metadata ends up with the same field stored in multiple places and becomes ambiguous or inconsistent.

-Z-

[areohbee]

|> I'm not sure if this works in Lightroom, but if not, it could be regarded as a bug and just fixed.

It does not work in Lightroom. I like the idea of regarding it as a bug to be fixed. (there have been several forum discussions about it, but I don't recall any comment from Adobe)

There is a partial solution for Lightroom users:

xEmP

In the case of DNG and RGB, it depends on xmp being 1st written to the source file, therefore you can't make them read-only. xmp will then be extracted and written as sidecar if changed. So, it solves the problem of a smaller (sidecar) file for settings/metadata that is only modified when necessary. It does not solve the problem of writing back to originals, or storing in common database. It also writes xmp-like files for virtual copies so they can be backed up and restored if necessary.

[Noel Carboni]

Zalman,

Clearly you understand the concept of never wanting an original file overwritten...

The software is MALFUNCTIONING as it is now.

But if you're worried over consistency with past (incorrect) operation confusing existing users, just add the new "Never Overwrite Original Files" option I mention here and default it to off. You can still use embedded info if you find it, but this would prevent you from writing embedded info back into ANY FILE TYPE.

This is not hard to understand!

Adobe presents Photoshop is a professional product - and I'm sorry to be blunt - but honestly you folks treat operations on your users' files in a very amateur way indeed!

-Noel

[Noel Carboni]

Sigh. Now we have Camera Raw 7 and no fix for this.

I think we can see how well Adobe listens to its customers.

-Noel

[areohbee]

Hi Noel, I can relate. Consider casting a vote on this one too:

http://feedback.photoshop.com/photosh...

I think this issue qualifies as a longstanding issue of the "low hanging fruit" variety.

[Noel Carboni]

Yes, or in Adobe terms, JDI. They might argue that it's a design change, but it's a design error so that's what's needed!

-Noel

[FeedbackCommunityMember]

I want an option to force all filetypes to use xmp sidecar files only, and not to modify original file.

While xmp data is written inside of tiff and jpg files, this makes backup to add unnecessary media consumption as those files are changes (e.gg size and CRC do not match to previous copy). This is totally unnecessary use of media space, and very annoying especially with off-site backups.

Further, this unnecessary change of original file increases possibility of file corruption, should something go wrong.

I agree, that in some cases where e.g. raw+jpg are used, this will require special renaming measures during import, but should not be an issue.

For me, changes to original tiff-files are risking hight quality scans, and are unnecessaryli increating size of backups (incremental, and differential, running always full backup is not an option).

[Guest]

Maybe this is possible already, but I didn't see it. Can't Lightroom give us the option to write the metadata to a XMP sidecar file when using DNG? I know one of the advantages of DNG is the lack of sidecar files, but for those of us backing up our photos (especially when it is offsite, like Backblaze or Carbonite) it is a major pain in the... Every time I slightly modify a photo or add a tag it will have to reupload the whole file. Staying with the original RAW file is not an option, as they are completely uncompressed (twice the size).

[FeedbackCommunityMember]

Develop settings and metadata is saved to the database, and if the user chooses again "closer" to the actual file. For raw files to a xmp sidecar file, and to jpg files (and others) to the header of the file itsself.

This inconsistency ruins one huge advantage to saving the settings/metadata to a place other than the fast database: whereas backing up the work done on raw files, is a dream come true (as during an incremental backup or synch to a network storage only the small xmp files are copied/backed-up), this does not hold true for the jpgs, they must be backed up completely every time anew.

Please add support for xmp sidecar files for jpgs and other non-raw files, and thus offer the option of leaving the jpgs untouched, just like the raw files. Thanks!

[Noel Carboni]

Thanks for adding your thoughts, guys. It's good to see this idea getting some traction.

-Noel

[areohbee]

Bridge(/ACR?) will write xmp as sidecar if DNG is read-only, but Lightroom will not, ever. Lightroom *will* read xmp from DNG sidecar, *if* xmp in sidecar is newer than embedded xmp.

[Noel Carboni]

That's interesting... So there already is logic at least partly there to handle settings input from possibly different sources. Thanks for that tidbit, Rob.

-Noel

[areohbee]

Hi Noel,

Technically, it's cake to read/write from sidecar vs. embedded, *but* there is no spec for how to name files, in the case of RGB sidecars. In other words, the xmp spec says: "same basename, xmp extension", but then if one shot raw + jpeg, and imported separately, the xmp filename maps to same, and saving xmp for one would overwrite the other. One idea:

* include the extension for rgb files, thus asdf.nef would have asdf.xmp, but asdf.jpg would have asdf.jpg.xmp. But that has the problem that asdf.jpg.nef is a legal filename for a raw nef file (stupid, but legal), which would then have the name conflict problem again.

So, for 100% robustness, it almost demands a name change for the raw's too. In other words, for zero possibility of name conflict, one would need to change to asdf.nef.xmp for raw xmp files. But, then it would not be backward compatible.

Another idea, use different extension for rgb xmp files, e.g. keep asdf.xmp for asdf.nef sidecar, but have asdf.jpg use asdf.jxmp for it's sidecar name. But then asdf.tif would need a different one too, e.g. asdf.txmp. Then if there was a png, it would need a different one too, etc.

So, there are some tricky issues to be worked out. Not insurmountable, but tricky.

My sense is that we will *never* see this change. Hope I'm wrong, but I do not expect it. Why? Because I think these issues were already thought about, and decision made - not to look back... Adobe thinks of sidecars as a necessary evil when file format does not support embedded xmp, as opposed to a plus for people who prefer sidecars for various reasons... DNG supports embedded xmp, and that's the file-format Adobe is pushing...

Cheers,
Rob

[areohbee]

Technically, it's cake to read/write from sidecar vs. embedded, *but* there is no spec for how to name files, in the case of RGB sidecars. In other words, the xmp spec says: "same basename, xmp extension", but then if one shot raw + jpeg, and imported separately, the xmp filename maps to same, and saving xmp for one would overwrite the other. One idea:

* include the extension for rgb files, thus asdf.nef would have asdf.xmp, but asdf.jpg would have asdf.jpg.xmp. But that has the problem that asdf.jpg.nef is a legal filename for a raw nef file (stupid, but legal), which would then have the name conflict problem again.

So, for 100% robustness, it almost demands a name change for the raw's too. In other words, for zero possibility of name conflict, one would need to change to asdf.nef.xmp for raw xmp files. But, then it would not be backward compatible.

Another idea, use different extension for rgb xmp files, e.g. keep asdf.xmp for asdf.nef sidecar, but have asdf.jpg use asdf.jxmp for it's sidecar name. But then asdf.tif would need a different one too, e.g. asdf.txmp. Then if there was a png, it would need a different one too, etc.

So, there are some tricky issues to be worked out. Not insurmountable, but tricky.

My sense is that we will *never* see this change. Hope I'm wrong, but I do not expect it. Why? Because I think these issues were already thought about, and decision made - not to look back... Adobe thinks of sidecars as a necessary evil when file format does not support embedded xmp, as opposed to a plus for people who prefer sidecars for various reasons... DNG supports embedded xmp, and that's the file-format Adobe is pushing...

Rob

[Noel Carboni]

Cake to you and I seems to be an obstacle to Adobe, referencing how long we've been discussing this.

Regarding the name ambiguity, couldn't the very same XMP file carry multiple sections, one for each similarly named input file?

-Noel

[areohbee]

|> "couldn't the very same XMP file carry multiple sections, one for each similarly named input file?"

I suppose it could - but I haven't thought through the implications. - definitely not insurmountable...

~Rob

[JJMack]

Noel you wrote

"Personally, I do not want my original out-of-camera JPEGs updated/rewritten under any circumstances."

Then you should not use LR or the Bridge for they are metadata editors. I personally think that that good. Raw camera image data is not altered. Which is what I care most about. However when it come to Jpeg Image data other application will alter this data. Only Camera RAW data seems safe for now. I also feel the some metadata should not be changeable like some EXIF fields not all though. I also feel the best place to record RAW conversion setting is better placed there then in external locations like databases and sidecar files. I like the ability of keeping information about an image in the image files themselves. In standard formats like EXIF IPTC etc. My vote would be to do away with the ACR database and sidecar files in favor of keeping RAW setting in metadata and perhaps even supporting versions of raw conversions. I also feel more application should utilize this data like web galleries generators should get copyright, title, description, EXIF and other metadata information. I also have no objection of LR building logical database over this information quick indexed quires are useful for many applications.

[areohbee]

Nobody is suggesting the removal of the option for embedded metadata. But for those of us who understand all the ins and outs and would still prefer sidecars or database only, we would like the option. I can even imagine multiple checkboxes:

1. Embedded.
2. Sidecar
3. Database (if non-Lightroom environment).

That way, user can have his/her cake with frosting, or without, and eat it too...

Rob

[Noel Carboni]

John, you say that it's good that image data is not altered, that it all should be done via metadata, yet you advocate the Adobe software writing the metadata into the original file.

It's not a big leap to understand that some folks think ALL the data in the original file is something we don't want altered, is it?

Like Rob says, I'm not looking to change the product to avoid writing in any metadata-capable file for everyone. I'd just like the OPTION to do that. I'd even expect the default would be to work as it does today.

I don't understand your comment "However when it come to Jpeg Image data other application will alter this data". From my perspective, properly written computer applications in general have INPUT files (which they open) and OUTPUT files (which they save), and if one doesn't save over an input file one really doesn't expect it to be altered!

-Noel

[JJMack]

If a Program like Photoshop opens an as you call it an input file and you make updates to it and you use menu File>save or with CS6 have left the default autosave preference setting un changes does not the input file get over written with the exception of RAW files. For Photoshop can not write Camera RAW files. You have to got to be careful or defaults will kill you. No!