P: Photo integrity check

[quidquomodo]

Lightroom currently has the possibility to search for missing photos, which is a very important function. Unfortunately this does not warn about the case where some photos are still present, but corrupted. Given the large amount of data stored on most computers, and the long storage time, this corruption is bound to happen sooner or later (such examples show up regularly on forums). The user therefore needs to be warned as early as possible so that corrective action can be taken (e.g. retrieve from backup) before it’s too late.

Proposition: implement an automatic photo integrity check.

One way of implementing this would be to calculate the checksum of photos on import for all file types (I understand this is already done for DNG s). The verification against checksum could be done either during the regular catalogue optimisation, or at the user’s demand in a way similar to the command ‘Library’>’Find Missing Photos’. This could be then saved into a temporary collection for review by the user.

Lightroom, make me feel confident that my photo collection is sound and safe.

[FeedbackCommunityMember]

You pretty much have this if you force recreation of all your previews.

[areohbee]

It would be good not to have to re-render all photos from scratch, just for a quick integrity check.

[quidquomodo]

Could you please explain how this works? If one forces recreation of all the previews, does Lightroom – at the end of the process – provide a list of photos which are corrupted?

In this case it may be doable for a one-off check. But as Rob says, it is not practical to rebuild previews weekly or even monthly for a 100,000 photo collection.

[quidquomodo]

Hal P Anderson has recently highlighted the following link:
http://dpbestflow.org/data-validation...
It is possible to run the free DNG converter on folders containing raw files to check whether the conversion (and therefore the raw files health) is OK.

Pros: it's currently the only way to check a large number of raw files.
Cons: It can take a long time, requires disk space for the second copies, and is limited to raw files (no jpg or video).

[areohbee]

What are the advantages of this over recreation of previews?

[quidquomodo]

If the recreation of previews provides a list of corrupted files, then the advantage of this method is: none. But I don't know whether the preview regeneration will provide such a list.

If the recreation of previews doesn't provide a list of corrupted files, and instead requires manual visual inspection of each photo, then the advantage of this method is a lot of time saved.

[areohbee]

As it stands, *if* Lightroom has to render a photo from raw data, it will pin a corruption badge to it, but some (most/all) of us would probably prefer to know about this immediately, before happening upon the image down the road... I would think that comprehensive detection (e.g. when backing up catalog) would be a good idea, as well as putting corrupt files detected in a collection and notifying user of the problem when encountered during normal use.

[Babar_e]

Really needs this
Backing up the database isn't just enough, we need to trust our photos backup as well.
First we need to compute a rawimagedigest for all the photo types
It would be nice to store it into the metadata of all the photos. But storing it in the database only would already be a great step forward.
Then we need to be able (as Rob said) to run a check and create a collection of all the corrupted images.
That is the minimum.
The bonus would be to provide a ui to recover these images from a back up.
-Select image to recover
-Point to the backup folder
-Lightroom scan the folder and finds the backed up images
-Lightroom check the rawimagedigest of these back ups
-If rawimagedigest is Ok lightroom copy the back ups, to the original image location, and replace the original images. (LR would handle image name modification thanks to its database)
-Lightroom update the copied image with the metadata stored in its database.

[areohbee]

Y'all may be interested in the 'Raw File Integrity Checker' plugin (free, written by me):

http://www.robcole.com/Rob/ProductsAn...

[FeedbackCommunityMember]

Does anyone know if Adobe is working on this? You guys have already mentioned some good points why and how to care about image integrity and if the filesystem of my operating system doesn't care of it, I wish Lightroom does. Lightroom is perfect to take care of the integrity of my pictures and I'd be glad to here that they are working on it.

Any insider knowledge?

[areohbee]

Adobe sometimes labels stuff "under consideration" or "in progress", but only sometimes.. (they could surprise us by implementing something not so labeled).

Non-Adobe insiders are sworn to secrecy, so unless Adobe chooses to enlighten, we stay in the dark..

My guess: Adobe won't implement this feature. If you want integrity checking, convert to DNG! (in other words, it's on a relatively short list of advantages of DNG over other formats, so maybe best for DNG not to eliminate this perk). It would be really easy to implement, yet they haven't, so.. Note: in the beginning, non-raw DNGs did not enjoy integrity checking, now (@Lr5 I think): all DNGs do - when that change was made would have been a perfect opportunity to include other file-types, but they didn't...

Of course there is the "Raw (and non-raw) File Integrity Checker" plugin, but there is a draw-back in having such info handled in proprietary fashion - if the plugin (or equivalent) doesn't work 10 years from now, then the integrity check info has become useless. And anyway it's limited - check codes are over whole file, not just static data (like image data), so saving xmp internally kills the integrity checking.

Rob

[FeedbackCommunityMember]

Ohh man.. this topic is really annoying. Of course I can change everything to DNG, would be far more convenient if my camera supported it initially..

Probably Adobe is just not interested in this topic. They want to push their DNG format. It would be great if Adobe would support correcting errors on DNGs as well.

[JJ_Tiziou]

Yep, this sucks.

There should be a built-in way to easily:

– verify selected sets of folders of images (without having to export copies of them)

– verify the entire archive at a given time

– enable rolling background verification so that the entire archive is verified periodically

– access an archive verification report, that would tell us something like "All images are verified as completely readable as of X date"

– get warnings about corrupted files, and see them in a separate collection with a nice interface for retrieving batches of images from backups. (Aka if twenty files are corrupted, and I point LR to a source to retrieve a backup, the whole batch of images are matched and copied in place of the corrupted files. And of course verified again.)

Right now it sucks to run into this kind of thing, because it invariably happens when you're actually looking to access the files:

[bryn_f]

Having imported a batch of files with intermittent corruptions, only some of the corrupt ones were "unable to build preview" so maintaining a checksum to see if the raw files has changed at all is necessary to find corrupt files, as well as providing a way to help find missing files which is the number one problem in lightroom students I have taught. In terms of dollars I know students who get frustrated with missing files and stop using lightroom / cancel subscription (often just because drive letters have changed, or they accidentally reimported the folders and they got moved). Having the checksum stored would allow a possible future feature of "looking for my missing files anywhere in this folder tree"

[Linwood Ferguson]

Someone pointed me to this suggestion while I was ranting in another forum about the requirement to use DNG in order to get integrity checking in lightroom (and even then it's only half-way checking since it checks only image not metadata portions of the file).

With photographers with home-quality computers accumulating multiple terabytes of data, bit rot is a real and growing concern. 

The idea of trying to render an image to see if it is corrupt is flawed -- major corruption can occur in an image and it will still render successfully (meaning no error message).  It may be totally hosed visually of course, but no human can routinely look at 100,000 images to see if any are corrupt.

The answer really lies in integrity checking file systems like zfs, btrfs or refs if Microsoft finishes it, but that is a long way from most photographers' systems. 

Adobe could VERY simply handle this - pull a checksum of raw images on import, stash it with the image data in the catalog, not the image.  Periodically let a run check that the checksum remains unchanged, and collect the images failing for review.

It's pretty fast, easy, and solves the problem without requiring people convert to DNG.  For those not writing metadata to DNG, it also provides a more complete check of DNG files than the DNG checksum does.

This is not rocket science.  It's pretty easy.  I see one posting above hinting at a solution, but the link is broken.  I wrote my own if anyone is interested (https://lrvalidate.codeplex.com/ ), free, all source code provided.  But people will be reluctant (rightly so) to use anything that writes to the catalog not from Adobe.

Adobe could do this in a few man days' effort, and get in front of the encroaching epidemic of people discovering "I had not looked at my files in 5 years and now some of them are unusuable what happened?!?!" complaints.

Bit rot.  If you don't know the name now, you will eventually.  😞

[Christophe5DE8]

To help increase reliability, I'd like to have an option "verify files after copy" to compare hashes between source & destination.
It's a pretty standard feature in such tools, and i have seen multiples case of files corrupted but still opening

[johnrellis]

Moderators, @Rikk Flohr: Photography : Please merge with the existing Idea:

https://community.adobe.com/t5/lightroom-classic-ideas/lightroom-photo-integrity-check/idi-p/1224921... 

[giovannil15990362]

It has happened, that during some moving or coping actions in between harddisk or folder, LR crashed or the HardDisk got disconected for whatever reasons. I had some corrupted files.

I was wodering if it would be possible to have a CHECK TOOL to scan your catalog/HDD/SSD in search of corrupted RAW files.

This would be particularly usefull when backing up or copying photos to new HDD/SSD.

or maybe is there yet and I am not aware of it. I just could find Validate DNG but: Only DNG files created by Adobe software can be validated (camera-‐created DNGs cannot be validated because they do not contain the necessary checksum)

[JohanElzenga]

There is a plugin that can do this: http://bayimages.net/blog/lightroom/validator/

[giovannil15990362]

Thanks. It looks great. I will have a try at it.
It Would be nice to have an option native to LR.

[JohanElzenga]

Maybe, but the idea behind plugins is that each user can add functionality that they want, without getting a bloated application full of functionality others want, but they may not want.

[johnrellis]

Moderators, @Rikk Flohr: Photography, please merge with this existing Idea:

https://community.adobe.com/t5/lightroom-classic-ideas/lightroom-photo-integrity-check/idi-p/1224921...

[David Kotz]

The DNG file format includes an embedded hash that allows tools to verify the integrity of a DNG file when they read it.  I'm told Lightroom Classic (LRC) verifies this hash every time it loads a DNG file, and I know LRC includes a menu item "Validate DNG files" that validates the hash on every DNG file in the library.  Super!

But my Library also contains TIF, JPG, HEIC, PNG, MOV, MP4 and other files.  In principle, LRC should compute the hash on every file imported to the Library, record that hash in the catalog, and validate the hash whenever those files are loaded.  Then, the "Validate DNG files" menu item could become "Validate integrity of all files".  I care about my entire collection, not just the DNG files.

Relatedly, this Validate operation should also be available in a contextual menu for folders in the Folders pane, so such validation could be run over a subset of the collection rather than the entire Library.  (Important if some of the Library is not currently mounted!)  

[Ian Lyons]

See https://community.adobe.com/t5/lightroom-classic-ideas/p-photo-integrity-check/idi-p/12249210

I expect somebody will shortly merge your request into above thread.

[David Kotz]

Thank you! I did a search before writing my post, but did not find that post.