Files are corrupted by Lightroom: White rectangle

[_Matt_M_87876_]

Hi there,

I updated Lightroom to 1.4.1 yesterday and some time after the update I started experiencing a potentially disastrous problem:

When I click on any photo in the Library view (e.g. old ones from before the update), when the full resolution is loaded, sometimes a white recltangle appears in the picture as if the pixel information was missing there (its size varies between photos). The images were definitely OK beforehand.

I tried opening an affected photo in IrfanView and it reported a corrupted file header and couldn't open it. I'm slightly worried now that Lightroom is (maybe irreversibly) eating my precious pictures! Has anyone got the same problem? Any solutions?

Thanks,
Matt

[_Martin_Ottosen_]

Hi,
Thought I would join the chorus, as I am experiencing the same issues. Some details:

1. I have ~8K pictures in my catalog, some crw (Canon 300D) some cr2 (Canon 40D) taken over a period of ~5 years.
2. I was evaluating lightroom 1.4.1 when the problem occured
3. I first noticed the problem, after I decided to convert my catalog to dng format (using lightroom).
4. Looking at my catalog "post dng conversion" ~90% of alle my photos were corrupt (had a small heart attack).
5. The corruption affected both (former) crw and cr2 files.
6. Viewing the images in picasa2 shows variously colored rectanges in the lower right corner of the images.
7.Usually the colored rectangles are filled with two solid colored triangles, but not always.
8. Mostly, the images that have the colored rectangles in picasa corresponds to the images that are reported as corrupt in lightroom.
9. Some of the images that are reported as corrupt in lightroom, look fine in picasa (in .dng).
10. A few of the images with colored rectangles in picasa, are not reported as corrupt in lightroom. In which case the colored rectangles appear in lightroom as well.
11. Tried installing lightroom 2.0, created a brand new catalog, and imported a few of the .dng images that I knew were corrupt. No difference.
12. Before converting to .dng I had been evaluating lightroom for a few weeks, with no problems.
13. I opted to not delete my old raw files (thank god), so I have the corrupt .dng's as well as the original raw files.
14. Corrupted files originate from completely seperate sets of gear (different cameras, readers, memory cards, cables)
15. Photo's are stored on a HP mediasmart box using shadow copy
16. Would be happy to send someone example files.

btw. Also tried importing one set of photos directly from the memory card, as opposed to converting them on disk. Same problem. Unfortunately I do not have raw backups of those photos :(

/Martin Ottosen

[_Mike_Anthony_]

I have a similar problem with LR 2.0. Workflow: cards from 2 x EOS450D and 1 x EOS5D read in via USB 2.0 card reader using Finder copy, directly into a new folder onto my Mac Pro internal drive. I shot around 500 photos over 2 days in July on one particular job. On returning from the shoot I ran through them in Bridge - fine - then imported to LR 1.5 - fine - edited and exported some to PS and then jpeg - fine. Some have already appeared in magazines.

Today, re-opening the LR catalogue in LR2.0, as I'm flicking through them in the film strip, around one in 20 are acquiring white or psychedelically coloured blocks (these often seem to happen to adjacent shots), so the corruption appears to be happening as LR creates the high res previews. Perhaps it takes a couple of passes through LR before latent problems originating in the card or reader reveal themselves. Any thoughts?

[MadManChan2000]

What kind of computer/processor are you using?

[joc-an]

> The corrupted images either have no thumbnails & can't be opened, or look totally distorted or have .swp after the name.

If you look at this thread from last year
http://www.adobeforums.com/webx?128@@.3bc48213 , you will conclude that the most likely cause of your problem is hardware-related.

When I converted to Lr2, I had a similar occurrence on one of 2 PCs. There were many images without thumbnails, all .jpg (never raw files,well Lr never writes to raw files). For the .jpg files that had problems there was a corresponding .jpg.swp file. By renaming the .jpg and then removing the .swp from the .jpg.swp file all was well again. But I moved all my images to another drive and have not experienced any problems since. The drive that originally contained my images (external USB just to port images) is still in use and from time to time there are Warnings in System Error log. (Description: An error was detected on device \Device\Harddisk4\D during a paging operation. By the way \Harddisk4\D is a DVD reader containing no disc!.Microsoft has this article on the error: http://support.microsoft.com/kb/244780/en-us. Interestingly in the More Information section it says
>"For example, event ID 51 is logged when blank media such as CDR, CDRW, DVDR, and so on, is inserted into a writable drive while a USB device is plugged in. The system logs the event even though the disc is writable, and the USB device is still usable. In these particular cases, you can safely ignore the log entries, and no additional action is required.".

I wonder?

[_Mike_Anthony_]

Eric, I am using a Mac Pro 2.66Ghz Quad Core with 4Gb of RAM, running OSX 10.5.4.

Mike

[_Chuck_Sale_]

Same problem as many are reporting here, with some interesting diagnostics.

HOW THE PROBLEM PRESENTS
*Canon 40D
*Problem presents with use of three or more different CF cards (it is not the cards)
*Problem presents while importing directly into Lr2 from camera to computer via USB cable OR loading from computer CF temp drive (no camera connection)into Lr2
*Problem presents while importing from computer CF temp drive (no camera connection) into Ps Bridge AND viewing in Bridge viewing option #2. (No corruptions appear in thumbnails in viewing option #1, the default)
NOTE: In Bridge, the corruptions disappear when I double click the corrupt thumbnail and move into Camera Raw. Then I can work as normal. You would never know any corruption ever existed. However, nothing works to get around the corruption as it appears in Lr2. Even switching to edit in Ps(CS3)simply brings over the corruption.

WORKAROUND AND POSSIBLE DIAGNOSTIC
I can import all files without corruption if I do two to three imports of the same files from the same CF card into Lr2. This is the procedure, a true pain, but 100% effective so far:

*Import the first time and always get lots of corrupted thumbnails and no possible way I can find in Lr2 to fix this.
*Delete the corrupted files from disk
*Import a second time all files. Usually all corruptions fixed, i.e., the previously corrupted files appear just fine this second time around. If not, I repeat deletion of the few remaining corruptions AND
*Import a third time. Never had to go past three imports and have never lost a file when using this recovery procedure.

NOTE: The problem presents and this workaround works no matter how many different CF cards I use. But, of course, this shouldn't be necessary. Something is very wrong. It is a mystery to me why Adobe Camera RAW ignores the corruption while Lr2 does not, unless I import over and over again, as above.

Chuck Sale

Chuck Sale Photography

http://chucksalephotography.zenfolio.com

[Lee_Jay-7OQGJF]

> Import the first time and always get lots of corrupted thumbnails and no possible way I can find in Lr2 to fix this.

Always? Something is wrong in the hardware somewhere. Could be anywhere in the digital string all the way to the motherboard.

[Guest]

Chuck-

If you dump the card onto the DT, or directly where you'll want the image files to reside, do you still get anomalies? IAE, this method, till you get sorted, should be faster, safer.... unless it, too, is borked.

[_Chris_D_Miller_]

I have a similar problem with corrupt images, eg:

http://redyeti.net/Rome/content/IMG_6618_large.html

It doesn't appear to be the .cr2 file itself that is corrupt though - I'll explain why shortly.

Here's my setup:

* Canon 40D (I *think* I also saw this sometimes in the past with my old 350D too, can't confirm that though sorry).
* Multiple CF cards, choice & brand of card doesn't appear to be a factor.
* Computer is an AMD Athlon 64 X2 4200+, DFI LanParty NF4 motherboard, 2x1GB OCZP4001G RAM (PC3200). HDD containing photos is 300GB Maxtor 6B300S0. S.M.A.R.T analysis (using SpeedFan) indicates the HDD is in very good health.
* Graphics card is nVidia 6800GT with recent drivers (v163.75). I've just upgaded to v178.13, don't yet know if that makes any difference.
* Running Windows XP with latest patches. Lightroom 2.0, but I've seen the problem with 1.4.x too.

My workflow:

* Put CF card into a USB card reader, copy files off using Windows Explorer to a folder on my HDD.
* Import the files into Lightroom, leaving files at their original location.
* metadata & changes are made to sidecar XMP files.
* Typically the thumbnail previews all show up fine, and when I click on a photo in Develop mode the larger preview is fine too. It's not until the photo has finished "Loading..." that the corruption appears. I assume this is because the embedded JPGs are used for the previews, and the corruption is occurring when processing the RAW data.

I'd estimate I get 1-2 corrupt images per 100 photos. It's annoying because I don't discover which images are corrupt until I start working on a particular image in detail. But the really interesting thing is, I *do not* have to recopy the image off the CF card to correct the problem. I have found a couple of workarounds:

* Rename the file on disk, and reimport it. This tricks Lightroom into thinking it's a new image, and almost always this 'new' image renders fine!
* Delete the sidecar XMP file and force Lightroom to resync the metadata. Again, this often fixes the problem (though not as reliably as the first approach).

I've also tried 'touching' the cr2 file (updating the creation date of the file), which I think works sometimes once Lightroom resyncs it.

In other words, it appears as though the file on disc is physically fine, but something happens when Lightroom processes the cr2 raw file that causes the corruption. Forcing Lightroom to reprocess the file from scratch generally straighten things out. The question is, what's causing the problem? I have a few theories:

* RAM. Some time back I upgraded from 2x512MB to 2x1GB sticks, different manufacturer and specification. There were definitely some stability problems when I first did this, it took a few goes with different sticks in different slots, BIOS changes etc to get my PC booting again. I've run several memory tests since (including one built into my BIOS) without seeing any problems, but perhaps there is some corruption happening when using Lightroom. Unfortunately I can't remember whether or not the problems started appearing around the time of this upgrade.
* Some sort of HDD/bus issue causing data to be corrupted as it is read from HDD. Also unlikely I would think since no other apps or data have any problems with data integrity/corruption.
* Corrupt Lightroom catalog.
* Threading issue in Lightroom. It's possible that a bug in some multithreading code is causing data corruption in memory when the files are processed. Seems a little unlikely, but certainly possible.
* Lightroom bug when handling Canon (40D only?) CR2 files.
* Conflict with another running application or video drivers. Can't think of anything that might be to blame but who knows...

I hope the above information is of some use. I'm happy to send through example images or provide any other information you might want as I'd love to see this problem solved. I also plan on upgrading my PC in about 2-3 months' time, it'll be interesting to see if new hardware makes the problem go away then.

[_Susan_J_Roche_]

is this the hard drive as I am experiencing the same problem

[_Martin-S_]

In article <59b526e0.33@webcrossing.la2eafNXanI>,
Susan_J_Roche@adobeforums.com wrote:

> is this the hard drive as I am experiencing the same problem

This just happened again to me today. These were images I had been
working with for about a week with no problems and suddenly one
developed a blank rectangle, about one quarter in the lower right.

Attempting to open the problem image afterwards in any other editor
showed the blank rectangle, too. I then retrieved a copy from a backup,
re-imported it into LR and all was fine again.

Last time this happened, I ran a thorough check (took hours) on my CF
cards and hard drive and they all came up clean.

These are Canon raw files from an EOS 5d in LR 2.0 PPC Mac - what is
going on here?

--
Cheers Martin

[Chezbriand]

There would appear to be enough people posting of corruption issues here that we would have a solution by now. Yet, despite early response from Adobe, I haven't seen anyone from Adobe respond with a solution, and the number of people posting with identical corruption issues certainly points to Lightroom as the source of the corruption.

I am also experiencing corruption issues identical to those described on this thread. In particular the corruption I am seeing is just like the corruption reported by Chris D Miller in this thread on Sep 26, 08 PST (His image was here http://redyeti.net/Rome/content/IMG_6618_large.html )

I am willing to supply sample files if that would help. I have an example where a Canon Rebel XT .CR2 file that was imported from a CF card via Adobe Lightroom has corruption, yet the same file copied from the CF card to the HD does not have the corruption. The file sizes are the same, but a compare shows differences in the data. The .CR2 file has been corrupted by Lightroom in exactly the same way as others on this thread have posted about. I have many cases now where files that have otherwise gone undamaged for months, begin to appear as corrupted by Lightroom.

The corruption seems to be permanent: Opening with Photoshop CS3, Canon ZoomBrowser, and Lightroom all show a corrupted RAW image.

This is likely not a hardware issue as there have been many postings with this identical issue all with very different setups.

The only two things I can seem to find in common is that everyone is using Lightroom and I have only seen posts from folks with Canon RAW images. Is Lightroom having a problem with Canon RAW files?

If anyone has found a solution to this madness please let us know. Adobe are you still listening?

[_Martin-S_]

In article <59b526e0.35@webcrossing.la2eafNXanI>,
Briand_Sanderson@adobeforums.com wrote:

> The only two things I can seem to find in common is that everyone is using
> Lightroom and I have only seen posts from folks with Canon RAW images.

In general only Canon users seem to be affected, but I think one person
encountered the problem with Nikon raw files. It also occurs on both
platforms.

I don't import straight from the card into LR, but rather use Image
Capture (Mac) and then drag&drop labelled folders into LR. The bizarre
thing in my case is that files can be fine for months and then suddenly
become corrupt. In every case restoring from the backup fixes it.

That seems to suggest that the files *are changed* by something.

--
Cheers Martin

[Guest]

To be clear, Martin, this was one image that came up corrupted? It's almost a certainty it wasn't corrupted by Lightroom, as LR only renders from the RAW file, not rendering the file itself.

I'd also go to v 2.1 if I were you. My 5D files look a treat in there.

[_Martin-S_]

In article <59b526e0.36@webcrossing.la2eafNXanI>,<br /> "John McWilliams" <jpmcw@comcast.net> wrote:<br /><br />> To be clear, Martin, this was one image that came up corrupted?<br /><br />In this particular catalogue, yes, but it's happened several times <br />before through all versions of LR as far as I recall.<br /><br />-- <br />Cheers Martin

[Chezbriand]

Can you site the reference to the Nikon raw file corruption? I hadn't seen that one. I would love to rule out Canon RAW from the equation.

My workflow only includes Lightroom, which is why I find it hard to believe Lightroom isn't responsible for the corruption. I use Lightroom to import photos from the CF card, and also have Lightroom create a backup copy upon import. In my most recent example of a corrupted file, the file in the imported directory is corrupted while the copy in the backup directory is not corrupted.

It is highly unlikely that this is a hardware issue as there have been a significant number of people reporting the exact same type of corruption, all with very different setups and platforms.

I have seen this corruption happening for some time, but until yesterday I believed it to be isolated to the import process. In cases where I saw a corrupted file I had a backup copy to replace it and I could keep working.

However, yesterday I had a set of much older photo files (9 months old) that were previously uncorrupted that became corrupted while using Lightroom and this has lead me here and raised a very significant concern that my photos are being corrupted well after the import process.

Since so many have reported this, yet I have seen no one find a solution, it is very worrisome.

[_Martin-S_]

In article <59b526e0.38@webcrossing.la2eafNXanI>,
Briand_Sanderson@adobeforums.com wrote:

> Can you site the reference to the Nikon raw file corruption?

Clive Moss, post #20
http://www.adobeforums.com/webx?14@@.59b526e0/19

Seems an isolated case though.

--
Cheers Martin

[Ian Lyons]

>> The only two things I can seem to find in common is that everyone is using
> Lightroom and I have only seen posts from folks with Canon RAW images.

That might be true, but the sample size is miniscule when compared with the number of Lr/Canon users not experiencing a problem with file corruption. A quick check on the Apple Aperture board would suggest that Lr isn't the only application that folk point their fingers at.

>That seems to suggest that the files *are changed* by something.

A change should also cause the file modification date to be altered to reflect the time at which the change occurred.

Do your corrupt files display such a date/time change? Do you write XMP metadata and develop settings back to your files? Do you open them in any other software?

You should also be aware that corruption can occur during transfer from the flashcard to the hard disk. USB is the usual culprit on Macs. However, more often than not the corruption is as result of an application or OS crash that occurs when the file is open in memory. Do you recall whether either occurred during an editing session (with Lr or any other application)? Do you recall "anything" untoward occurring during an editing session that could have caused the corruption?

[_Martin-S_]

In article <59b526e0.40@webcrossing.la2eafNXanI>,
Ian_Lyons@adobeforums.com wrote:

> Do your corrupt files display such a date/time change?

I don't have any corrupted ones at the moment, but I'll keep an eye on
the dates next time.


> Do you write XMP metadata and develop settings back to your files?

No.


> Do you open them in any other software?

No, only later to check the corrupted ones when the problem occurs.


> You should also be aware that corruption can occur during transfer from the
> flashcard to the hard disk. USB is the usual culprit on Macs.

I have never noticed the corruption *immediately after* importing and,
for what it's worth, I use a FW reader.


> Do you recall
> "anything" untoward occurring during an editing session that could have
> caused the corruption?

Not really, LR has always been very well behaved for me. I usually run
it as the only application.

It happens rarely, but if I remember correctly, the white rectangles
popped up either upon starting LR or switching to the library's grid
view from develop mode.

--
Cheers Martin

[_benedict_johnson_]

Hi there, I'm also experiencing corruption issues.
I'm shooting canon 1ds MKI and 1dMKIII. I'm running three pc; 2 with LR1 and 1 with LR2 (recently upgraded from LR1).

I shoot a lot and LR1 works/ed fine regards import with the same card readers and cards and still does. However since upgrade to LR2 and after successful testing period I have lost 2 out of 4 batch imports - the files have come up as corrupt on the disk and any image rescue software that I own. The CF card is no longer readable in normal use - using image rescue software only recovers corrupt files.

Just adding my voice here but I'd really appreciate any pointers on good image recovery software/services and solving this issue . I used (amongst others) Photorescue software which found the corrupt files.

thanks, benedict

[_Martin-S_]

In article <59b526e0.43@webcrossing.la2eafNXanI>,
benedict_johnson@adobeforums.com wrote:

> The CF card is no longer readable in normal use - using image rescue software
> only recovers corrupt files.

Are you experiencing the white rectangle corruption?

If not, please, open a new thread.

--
Cheers Martin

[Chezbriand]

Does anyone who is monitoring this thread who has seen corruption of this type (white rectangle, or pink rectangle, or similar corruption mentioned in this thread) have the ability to compare a corrupted version with a known copy that is not corrupted?

In other words, if you open up a command prompt and use the "comp" or "fc" command to compare the corrupt version and the non corrupt version of the same file, what are the results?

This information may be useful in trying to determine the root cause of this corruption.

[_Chris_D_Miller_]

This thread may be of interest:

http://www.adobeforums.com/webx/.59b6fe68

Also, I have a batch of new images that I will be processing in Lightroom shortly and I'll try to keep an eye out for this problem to see if I can spot any patterns/triggers. I'll also diff files etc to see if there is a problem with the images themselves, or the corruption is purely in the Lightroom previews/catalog (I suspect the latter). I'll try and report back with my results in a week or two.

Chris

[_Chris_D_Miller_]

It looks like the corruption is purely within Lightroom. I just had an image (IMG_0414.CR2) that appeared corrupt, but running a binary file compare against the original copy on the CF card showed no differences. Additionally, I did the following:

* Made a copy of the image called IMG_0414b.CR2 in the same directory.
* Right-clicked on the folder in the library navigator and selected "Synchronize Folder...". This picked up the copied file which did NOT appear corrupt. The IMG_0414.CR2 image still appeared corrupt.

Next, I removed and re-added the corrupt image as follows:

* Right-clicked on the imagine and selected "Delete Photo...". Click "Remove" to remove it from the catalog.
* Ran "Synchronize Folder..." again, which picked up the IMG_0414.CR2 file that I had just removed. This time the corruption did not appear, everything looks fine.

Given the above, it's very clear that the file itself is NOT corrupt, and it is in fact the Lightroom catalog that has the corruption. Whether this is caused by a bug in Lightroom or hardware problems is still unclear. Regardless, the quick workaround seems to be to remove and re-add the corrupt image(s) to Lightroom to fix the corruption. Can anyone else verify that this works for them too?

Chris

[Guest]

I'd be happy to, but I haven't experienced an image that doesn't display properly in LR that wasn't in fact corrupt.

It'd certainly seem to be a problem in the Previews file. Did you try deleting (creating a new one) the Preview for that file?

How did the apparent corruption manifest?