Skip to main content
Horshack
Horshack
Legend
December 11, 2014
Question

D750 "unexpected end of file" for NEF that loads fine into Capture NX-D

  • December 11, 2014
  • 15 replies
  • 33389 views

I have a NEF that produces an "unexpected end of file" error in both Lightroom 5.7 and PS CC (both with Camera Raw 8.7). The same NEF loads and converts fine in Nikon's Capture NX-D V1.0.2. The NEF also converts fine into a JPEG using the D750's in-camera NEF processing function. I can provide the NEF to Adobe for analysis.

    This topic has been closed for replies.

    15 replies

    Show previous replies
    Lattefarsan
    Lattefarsan
    Participating Frequently
    December 27, 2014

    I just bought a D750 and have had about 6-7 images with the "unexpected eof" error in LR 5.7 and PS cc 2014. Have been trying to reproduce the error with two memorycards to see if the error occurs on both cards, but haven't been able to generate any files with the eof-error... yet.

    engerim
    engerim
    Participating Frequently
    December 26, 2014

    I got the same issue since I got the D750 (and the latest version of Adobe Camera Raw).

    Here is a file that has the problem: http://stuff.inxsoft.net/tmp/DSC_3816.NEF

    I found DCRAW has also issues with the files:

    DSC_3816.NEF: Corrupt data near 0x1e502a7

    Camera Raw reports unexpected end of file.

    Versions:

    Camera Raw 8.7.1

    Operating System: Windows 8.1 64-bit

    System architecture: Intel CPU Family:6, Model:5, Stepping:1 with MMX, SSE Integer, SSE FP, SSE2, SSE3, SSE4.1, SSE4.2, AVX, AVX2, HyperThreading

    Capture NX doesn't complain about it at all and the image data looks fine...

    Horshack
    HorshackAuthor
    Legend
    December 27, 2014

    Thanks @engerim. I did an initial analysis of your D750 NEF and I see the same symptom as mine - the size of the encoded (compressed) image data as describe the EXIF fields matches the actual file size. Specifically for your file, the strip offset to the encoded data is 3,228,160 (start of raw image data) and the strip byte count is 28,557,561 (size of raw image data); that results in an ending file offset of 31,785,721, which exactly matches the NEF file size of 31,785,721. For a "real" underrun we would expect the actual file length to be shorter than the size indicated by the EXIF data, which is the typical case for a corrupted/truncated NEF resulting from SD/CF corruption. However in this case all of the NEF data described by the EXIF is there, the same as in my file. I haven't dug deeper into your NEF file yet but I imagine I'll see that the last few columns of the last row of image data are underrepresented in the encoded data, meaning that the decoder runs out of compressed huff bits before reaching what it believes is the end of the image data. A few possibilities here:

    1) There is a bug in the D750's ASIC/firmware encoding of NEF data that creates an underrun of huff-encoded bits for certain scenes

    2) There is a new behavioral aspect in how Nikon is encoding the data and non-Nikon raw processors aren't accounting for this

    3) Non-Nikon decoders like ACR/LR/DCRAW are using/assuming too large an image dimension vs the actual dimension of usable data from the file, thus trying to decode more of the image data than they should

    The fact that Nikon's decoders handle these EOF D750 files without error indicates #2 or #3 are more likely. #1 is still possible if Nikon's processors just happen to decode less of he image data (use a slightly-smaller image dimension for the file).

    Btw @engerim, any chance you were using both SD card slots on your D750 when you got this NEF and had them configured in backup mode? If so I'd like to see the same NEF from the other card if you have it.

    M
    Mr Papajohn
    Participant
    December 27, 2014

    I have been experimenting and have found that setting the D750 to 12 bit raw eliminates the issue. When set to 14 bit raw, some files produce “unexpected end of file”

    John

    Horshack
    HorshackAuthor
    Legend
    December 12, 2014

    I've been digging deeper into this issue. I downloaded and compiled the code to http://rawstudio.org/blog/?p=800 to see why it doesn't get an EOF on the NEF. I walked through the code handling this NEF and found that it is rounding up the size of the encoded NEF segment by 4 bytes, as called by NikonDecompressor::DecompressNikon():

    BitPumpMSB::BitPumpMSB(const uchar8* _buffer, uint32 _size) :

      buffer(_buffer), size(_size + sizeof(uint32)), mLeft(0), off(0) {

      init();

    }

    The NEF's strip byte count per EXIF is 22,501,973. The above logic rounds that up to 22,501,977, which is just enough extra bits to deliver the encoded data needed for the last few pixels on the final row of the image data. RawSpeed doesn't initialize those extra bytes to anything and they happen to be 0xCD (MSVC C-Library alloc init pattern), so it would likely generate garbage data for those last pixels. The mystery is why Nikon's Capture NX-D is handling the file correctly - I examined the image including the last pixels and everything  looks correct. I know that everyone including Adobe had to reverse-engineer the NEF format; I wonder if part of Nikon's huff encoding is to presume zero-extension of the final word of data when/if it compresses to a certain known set of values.

    ccastleb
    Adobe Employee
    ccastleb
    Adobe Employee
    December 12, 2014

    Thank you for the detective work.

    I'll add this to the internal bug I logged on the topic.

    Regards,

    - Chris

    Horshack
    HorshackAuthor
    Legend
    December 11, 2014

    Btw the same NEF also converts without error with RawSpeed Developer Information | Rawstudio, http://www.rawdigger.com/, and http://www.fastrawviewer.com/. The problem appears limited to ACR/LR.

    JP Hess
    JP Hess
    Inspiring
    December 11, 2014

    The problem isn't being mentioned by other D 750 users. You must be something unique to your system.

    M
    Mr Papajohn
    Participant
    December 24, 2014

    I just received a D750 for Xmas preent of the first 21 images I took, 13 of them registered the file:///.file/id=7035116.25file:///.file/id=7035116.25unexpected end of file error when I tried to off load them from the memory card to LR5 and also to PSE 13 on my iMac. 

    JP Hess
    JP Hess
    Inspiring
    December 11, 2014

    If you used Capture NX to download the images from your camera, that could be the problem. Older versions of Nikon software to do something to the raw images so they cannot be read by Camera Raw or Lightroom. If I am describing your situation, here is a link to a utility that can fix your NEF files:

    Fix Corrupted Nikon NEF Images

    Horshack
    HorshackAuthor
    Legend
    December 11, 2014

    Thanks for the reply. I did not use Capture NX to transfer the image. I first transferred the NEF from the camera via USB and then after seeing the error tried again by mounting the SD card in a reader and transferring the NEF again which produced a binary identical file (ie, the problem is not in the transfer). And again, the same file in the computer can be successfully converted by Nikon's own raw processor (Capture NX-D).

    JP Hess
    JP Hess
    Inspiring
    December 11, 2014

    This is a user-to-user forum. Adobe staff people frequently check here, but most of the responses will come from other users like yourself. If you can provide a dropbox link to one of your NEF files then perhaps some of the more seasoned users here can take a look. Personally, I don't have an answer. I haven't encountered the problem you describe, and I'm not as technically oriented as some of the other regulars here.

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices