D750 "unexpected end of file" for NEF that loads fine into Capture NX-D

Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

I have a NEF that produces an "unexpected end of file" error in both Lightroom 5.7 and PS CC (both with Camera Raw 8.7). The same NEF loads and converts fine in Nikon's Capture NX-D V1.0.2. The NEF also converts fine into a JPEG using the D750's in-camera NEF processing function. I can provide the NEF to Adobe for analysis.

Views

23.0K

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
replies 111 Replies 111
Community Expert ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

If you used Capture NX to download the images from your camera, that could be the problem. Older versions of Nikon software to do something to the raw images so they cannot be read by Camera Raw or Lightroom. If I am describing your situation, here is a link to a utility that can fix your NEF files:

Fix Corrupted Nikon NEF Images

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Thanks for the reply. I did not use Capture NX to transfer the image. I first transferred the NEF from the camera via USB and then after seeing the error tried again by mounting the SD card in a reader and transferring the NEF again which produced a binary identical file (ie, the problem is not in the transfer). And again, the same file in the computer can be successfully converted by Nikon's own raw processor (Capture NX-D).

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

This is a user-to-user forum. Adobe staff people frequently check here, but most of the responses will come from other users like yourself. If you can provide a dropbox link to one of your NEF files then perhaps some of the more seasoned users here can take a look. Personally, I don't have an answer. I haven't encountered the problem you describe, and I'm not as technically oriented as some of the other regulars here.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Btw the same NEF also converts without error with RawSpeed Developer Information | Rawstudio, http://www.rawdigger.com/, and http://www.fastrawviewer.com/. The problem appears limited to ACR/LR.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

The problem isn't being mentioned by other D 750 users. You must be something unique to your system.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Dec 24, 2014 Dec 24, 2014

Copy link to clipboard

Copied

I just received a D750 for Xmas preent of the first 21 images I took, 13 of them registered the file:///.file/id=7035116.25file:///.file/id=7035116.25unexpected end of file error when I tried to off load them from the memory card to LR5 and also to PSE 13 on my iMac. 

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 24, 2014 Dec 24, 2014

Copy link to clipboard

Copied

@papajohn, Can you send me a few of your raws for me to analyze? You can use Transfer Big Files Free - Email or Send Large Files up to 20 GB to send the files to yourself and then PM the link. No registration is required and it's free for that service. Thanks!

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Dec 26, 2014 Dec 26, 2014

Copy link to clipboard

Copied

Haven’t been able to send a file but I’ve been doing some experimenting and found that if I set the D750 to 12 bit raw, the issue does not occur. On 14 bit raw some files display the “unexpected end of file”

John

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Nov 16, 2015 Nov 16, 2015

Copy link to clipboard

Copied

Having the same issue here. I have the latest version of LR (5.7, Camera Raw 8.7) and a small number of unexpected end of file occures. Importing into Capture One Pro never has a problem with those files.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Nov 16, 2015 Nov 16, 2015

Copy link to clipboard

Copied

Hello,

As far as we know there was a file truncation issue with the D750. I'm not certain if that was fixed in a firmware update, but nevertheless I recommend making sure you are up to date with your camera's latest firmware. As for why some images would not open in ACR/Lr, it's due to how conservative the software is in preventing corrupt or potentially corrupt files from being opened. We made a fix on the Camera Raw/Lightroom side in 9.0/6.0 respectively that works around the D750 file truncation issue. The latest version available is 9.2/6.3.


Hope that helps,


- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Nov 16, 2015 Nov 16, 2015

Copy link to clipboard

Copied

Hi Chris,

Thanks for the info. I'll check the latest firmware for my camera.

Regarding Lightroom, I have the non-cloud version, which I believe is at

version 5.7. Not sure if Adobe are going to make more updates for that

version as their cloud subscription seems the way they want to go.

Cheers,

Dayve

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Nov 16, 2015 Nov 16, 2015

Copy link to clipboard

Copied

Hello Dayve,

You can choose the upgrade option for the standalone (non-CC) LR 6.x from 5.7.x by going to the following page.

https://www.adobe.com/products/catalog/software._sl_id-contentfilter_sl_catalog_sl_software_sl_photo...

Regards,

- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Nov 16, 2015 Nov 16, 2015

Copy link to clipboard

Copied

Thanks Chris. That's great. I'll have a look tomorrow.

Best wishes,

Dayve

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Please provide a sample as Jim suggested earlier and I can investigate whether this is an issue with ACR/LR or something else.

Cheers,

- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Thanks Chris, I just PM'd you a link to the NEF.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Does this "end of file" error happen with only this single image or with multiple images?

Do any of your D750 images load correctly in Lightroom?

Are you using unique camera settings for the images that do NOT open?

Have you tried transferring to a different machine?

Thanks,

- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Also, if you are seeing this consistently with more than a single image, what about trying a different memory card to see if you can reproduce it?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

This is the only NEF I've seen this issue on, out of 250 NEFs I've ingested into LR so far. All my NEFs have used the same camera-settings, including being 14-bit lossy compressed. In fact I took an identical photo of that same scene/lighting/exposure a few minutes after discovering it wouldn't ingest and the new photo ingests into LR fine. I'll PM a link to that second NEF to help in your troubleshooting. I haven't tried a different machine but I did send the NEF to someone else and they were able to reproduce the EOF error on their setup as well.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

OK. ACR/LR is conservative when encountering potentially corrupted image data. It's possible that there was a file truncation issue writing the image to the SD card in-camera or during transfer. Since you transferred it a couple of times, I'm leaning towards an in-camera glitch or perhaps an issue with the SD card. In any case, it's good to know you are not seeing this as a widespread issue.

- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

Thanks Chris. If you and your engineers examine the huff segment information in the EXIF of the NEF ("strip offsets" and "strip byte counts"), you'll see they match the actual starting offset and resulting length of the file, indicating that all of the encoded data is there. Specifically, the strip offset is 3,155,968 and the strip byte count is 22,501,973. That results in an ending file offset/size of 25,657,941, which exactly matches the file size. If anything it appears ACR/LR's decoder ran off the rails somewhere. Another explanation is that some of the encoded huff data is corrupt but again, since Nikon's Capture NX-D can decode the entire file (along with other raw processors and the camera's internal NEF converter as well), that doesn't appear to be the case.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 12, 2014 Dec 12, 2014

Copy link to clipboard

Copied

I've been digging deeper into this issue. I downloaded and compiled the code to http://rawstudio.org/blog/?p=800 to see why it doesn't get an EOF on the NEF. I walked through the code handling this NEF and found that it is rounding up the size of the encoded NEF segment by 4 bytes, as called by NikonDecompressor::DecompressNikon():

BitPumpMSB::BitPumpMSB(const uchar8* _buffer, uint32 _size) :

  buffer(_buffer), size(_size + sizeof(uint32)), mLeft(0), off(0) {

  init();

}

The NEF's strip byte count per EXIF is 22,501,973. The above logic rounds that up to 22,501,977, which is just enough extra bits to deliver the encoded data needed for the last few pixels on the final row of the image data. RawSpeed doesn't initialize those extra bytes to anything and they happen to be 0xCD (MSVC C-Library alloc init pattern), so it would likely generate garbage data for those last pixels. The mystery is why Nikon's Capture NX-D is handling the file correctly - I examined the image including the last pixels and everything  looks correct. I know that everyone including Adobe had to reverse-engineer the NEF format; I wonder if part of Nikon's huff encoding is to presume zero-extension of the final word of data when/if it compresses to a certain known set of values.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Dec 12, 2014 Dec 12, 2014

Copy link to clipboard

Copied

Thank you for the detective work.

I'll add this to the internal bug I logged on the topic.

Regards,

- Chris

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 26, 2014 Dec 26, 2014

Copy link to clipboard

Copied

I got the same issue since I got the D750 (and the latest version of Adobe Camera Raw).

Here is a file that has the problem: http://stuff.inxsoft.net/tmp/DSC_3816.NEF

I found DCRAW has also issues with the files:

DSC_3816.NEF: Corrupt data near 0x1e502a7

Camera Raw reports unexpected end of file.

Versions:

Camera Raw 8.7.1

Operating System: Windows 8.1 64-bit

System architecture: Intel CPU Family:6, Model:5, Stepping:1 with MMX, SSE Integer, SSE FP, SSE2, SSE3, SSE4.1, SSE4.2, AVX, AVX2, HyperThreading

Capture NX doesn't complain about it at all and the image data looks fine...

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 26, 2014 Dec 26, 2014

Copy link to clipboard

Copied

Thanks @engerim. I did an initial analysis of your D750 NEF and I see the same symptom as mine - the size of the encoded (compressed) image data as describe the EXIF fields matches the actual file size. Specifically for your file, the strip offset to the encoded data is 3,228,160 (start of raw image data) and the strip byte count is 28,557,561 (size of raw image data); that results in an ending file offset of 31,785,721, which exactly matches the NEF file size of 31,785,721. For a "real" underrun we would expect the actual file length to be shorter than the size indicated by the EXIF data, which is the typical case for a corrupted/truncated NEF resulting from SD/CF corruption. However in this case all of the NEF data described by the EXIF is there, the same as in my file. I haven't dug deeper into your NEF file yet but I imagine I'll see that the last few columns of the last row of image data are underrepresented in the encoded data, meaning that the decoder runs out of compressed huff bits before reaching what it believes is the end of the image data. A few possibilities here:

1) There is a bug in the D750's ASIC/firmware encoding of NEF data that creates an underrun of huff-encoded bits for certain scenes

2) There is a new behavioral aspect in how Nikon is encoding the data and non-Nikon raw processors aren't accounting for this

3) Non-Nikon decoders like ACR/LR/DCRAW are using/assuming too large an image dimension vs the actual dimension of usable data from the file, thus trying to decode more of the image data than they should

The fact that Nikon's decoders handle these EOF D750 files without error indicates #2 or #3 are more likely. #1 is still possible if Nikon's processors just happen to decode less of he image data (use a slightly-smaller image dimension for the file).

Btw @engerim, any chance you were using both SD card slots on your D750 when you got this NEF and had them configured in backup mode? If so I'd like to see the same NEF from the other card if you have it.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines