• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

D750 "unexpected end of file" for NEF that loads fine into Capture NX-D

Enthusiast ,
Dec 11, 2014 Dec 11, 2014

Copy link to clipboard

Copied

I have a NEF that produces an "unexpected end of file" error in both Lightroom 5.7 and PS CC (both with Camera Raw 8.7). The same NEF loads and converts fine in Nikon's Capture NX-D V1.0.2. The NEF also converts fine into a JPEG using the D750's in-camera NEF processing function. I can provide the NEF to Adobe for analysis.

Views

25.6K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
replies 111 Replies 111
New Here ,
Dec 26, 2014 Dec 26, 2014

Copy link to clipboard

Copied

I have been experimenting and have found that setting the D750 to 12 bit raw eliminates the issue. When set to 14 bit raw, some files produce “unexpected end of file”

John

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

"Btw @engerim, any chance you were using both SD card slots on your D750 when you got this NEF and had them configured in backup mode? If so I'd like to see the same NEF from the other card if you have it."

Tried the following this morning:

Menu -> Photo shooting menu -> Role played by card in Slot 2 = Backup (second option)

Then I started an interval shooting with 200 images, and went for breakfast 😉

Normally one would expect the images are 100% the same in both slots (checked with md5sum) but they are not (this makes this a bit more difficult).

Anyhow when opening the images in Camera Raw the images affected by this issue ("unexpected end of file" error) were the exact same files in both slots. All other images were fine in both slots. Files are here: http://stuff.inxsoft.net/tmp/d750-raw-with-issues.zip

All images included in the zip have the problem except DSC_4078.NEF (which I included for reference and it will also allow you to see the difference in checksums in both slots). Anyhow we can rule out its an SD Card issue or buffer issue.

Contents of the zip file:
$ find . -type f|xargs md5sum

d10ca1603b10820a8018afbf9a7b09a1 *./slot1/DSC_3992.NEF

a0394903a03b413202d7c6d0e971cc8b *./slot1/DSC_4047.NEF

02f54769845fc86890d1791950d14324 *./slot1/DSC_4065.NEF

2a35398be7298407c649a57d610d82b0 *./slot1/DSC_4077.NEF

^^ problematic files

879e94509d9fbb1bc19e96c7694850ba *./slot1/DSC_4078.NEF <-- file which has no issues for reference

94f542df6f3f85dec01919f4535910cd *./slot2/DSC_3992.NEF

eacb286d896d8282079d273812f27e9d *./slot2/DSC_4047.NEF

a70f9535ea180ce37fda81d05a723119 *./slot2/DSC_4065.NEF

11898b2c6fecf320af5fa1d234844598 *./slot2/DSC_4077.NEF

^^ problematic files

01995ec69995c2d72a8c4993ecefda30 *./slot2/DSC_4078.NEF  <-- file which has no issues for reference

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

engerim, thanks, I'll take a look at the files in the morning. Btw Nikon stores the SD slot number in the EXIF so that's probably why the checksums aren't matching.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

True, seems to be stored here (only file difference is this byte):

slot1/DSC_3992.NEF: @0x0007f90: 0000 0000 0000 0000 3031 3030 0000 6400  ........0100..d.

slot2/DSC_3992.NEF: @0x0007f90: 0000 0000 0000 0000 3031 3030 0100 6400  ........0100..d.

Here's the dcraw output (converting to PPM):

tom@box:~/d750-raw-with-issues/slot2$ for i in *.NEF;do dcraw $i;done

DSC_3992.NEF: Corrupt data near 0x1c200b2

DSC_4047.NEF: Corrupt data near 0x1c300b6

DSC_4065.NEF: Unexpected end of file

DSC_4077.NEF: Corrupt data near 0x1c1810b

tom@box:~/d750-raw-with-issues/slot1$ for i in *.NEF;do dcraw $i;done

DSC_3992.NEF: Corrupt data near 0x1c200b2

DSC_4047.NEF: Corrupt data near 0x1c300b6

DSC_4065.NEF: Unexpected end of file

DSC_4077.NEF: Corrupt data near 0x1c1810b

tom@box:~/d750-raw-with-issues/slot1$ md5sum *.ppm

f71b766cc72ccdd39e95b2173212f3eb  DSC_3992.ppm

40b00390c91cb44c896da52470f536fe  DSC_4047.ppm

155d9da9ae91c2ea396163e7a424a102  DSC_4065.ppm

b4382e67ded87946f7140461f0d9e44f  DSC_4077.ppm

a9f205a0ff8b3f788758780177cae110  DSC_4078.ppm

tom@box:~/d750-raw-with-issues/slot1$ cd ..

tom@box:~/d750-raw-with-issues$ cd slot2/

tom@box:~/d750-raw-with-issues/slot2$ md5sum *.ppm

f71b766cc72ccdd39e95b2173212f3eb  DSC_3992.ppm

40b00390c91cb44c896da52470f536fe  DSC_4047.ppm

155d9da9ae91c2ea396163e7a424a102  DSC_4065.ppm

b4382e67ded87946f7140461f0d9e44f  DSC_4077.ppm

a9f205a0ff8b3f788758780177cae110  DSC_4078.ppm

tom@box:~/d750-raw-with-issues/slot2$

Checksums of the resulting ppm's are the same.

I may dig into the dcraw source code some day... 🙂

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

OK, so all above files fail here (in dcraw):

unsigned CLASS getbithuff (int nbits, ushort *huff)

[...]

  if (vbits < 0) derror();

And also in:

void CLASS nikon_load_raw()

[...]

      if ((ushort)(hpred[col & 1] + min) >= max) derror();

Seems indeed related to decoding 14 bit raw (and would explain not everyone complains about it).

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

engerim, I analyzed one of the NEFs from the set of dual-card NEFs in your ZIP. The corruption signature is almost identical to what I see on my Sony card - a block of data in the huff stream near the very end of the file that is a replica of data before it. I have analyzed lots of SD/CF corruption from lots of different cameras and I have never seen this type of corruption before; the fact that we see it on two totally different card models and both near the very end of the image from two different D750's (in addition to you seeing it on two cards at the same time in both slots) tells me that camera firmware/ASIC is involved here. It's either a bug in the D750's firmware/ASIC or its a marginal timing instigated by the D750 that the SD card is violating for this last set of media write(s).

Here's an analysis of your DSC_3992.NEF file. I found the corrupt offset by loading the NEF into LibRaw under a debugger and setting a breakpoint on when an error in the huff decode was detected, then viewing the file position at that point.

i-n9pxq9r.png

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

I don't know if the data is really corrupt. Capture NX, View NX2 have no issues.

dcraw/libraw/Camere Raw are based on the same code (and many others).

I even get proper ppm files out of dcraw (with those error messages).

It could just be that Nikon added some additional things/compounding, etc. 🙂

I sent the files to the dcraw author. More debug needed...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

It's possible that the replicated image data at the end of the NEF is intended by Nikon although not sure why we'd only see it on certain images, esp when shooting the same static scene via the intervalometer, plus the replicated data corrupts the huff stream and this area is included in EXIF-described compressed extent (offset+count) so it's intended to be deocded. The reason Nikon's software is not complaining is probably because they're not decoding to the very end of the NEF. For example on your DSC_3992.NEF the huff error was detected by Libraw at column 5327 of row 4031. Libraw is decoding the full NEF to an image dimension of 6032x4032. In contrast, both ACR/LR and Nikon's software are decoding to a smaller area of 6016x4016. Even though ACR/LR are only using 6016x4016, they appear to be decoding the entire NEF/huff stream, thus encountering the "bad" huff data beyond those dimensions. Nikon's software is probably decoding only to 6016x4016 and thus not decoding beyond those dimensions.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

I just bought a D750 and have had about 6-7 images with the "unexpected eof" error in LR 5.7 and PS cc 2014. Have been trying to reproduce the error with two memorycards to see if the error occurs on both cards, but haven't been able to generate any files with the eof-error... yet.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

I was able to reproduce the EOF on my D750. This time I populated both SD slots - the first slot had my original Sony 32GB 94 MB/s card that produced the first EOF NEF from two weeks ago, the second SD slot had a Sandisk 32GB 45 MB/s. Unlike engerim setup, my two NEFs didn't match when I reproduced the EOF - the Sony card produced another EOF NEF but the Sandisk's equivalent/"backup" NEF in the other slot could be read by ACR/LR without error. To isolate the problem to the card vs the specific D750 slot, I then swapped the cards between the two slots and reproduced the EOF NEF again - the EOF issue followed the Sony card to the other slot. To isolate the problem to the specific copy of this Sony card vs this model of card, I then replaced the Sony card with another of the same make and the new card did not produce an EOF NEF. So it looks like my original Sony SD card is bad.

I then analyzed the EOF by comparing the EOF NEF from the Sony card to to the equivalent good NEF from the Sandisk and found that the Sony card has data corruption near the end of the file. Because the corruption is at the end of the file it appears Capture NX-D (and the D750's in-camera NEF->JPEG conversion) is not encountering during its decoding, perhaps because it's converting a smaller image dimension from the NEF vs ACR/LR.

Here is an analysis of the data corruption. Based on the data-replication I see it appears one of the parallel NAND channels within the card is incorrectly sourcing data from another NAND channel.

i-RDG2W5h.png

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 27, 2014 Dec 27, 2014

Copy link to clipboard

Copied

I've continued playing around with the suspect Sony card and have reproduced the EOF NEFs a few more times. What's interesting is that each repro has the same signature - corruption near the very end of the file (within last 1KB), with corrupt data replicated across the corrupt area. This implies that whatever is wrong/marginal with my Sony SD copy is sensitive to the D750's media timing for the last write(s) of the image data. Because of this I'm still interested in analyzing other users' D750 EOF NEFs, to see if there's a unique timing sensitivity of the D750 with certain SD cards. When you attempt to reproduce please use both card slots because both NEFs are required to do the differential analysis to find the corruption. Please also report which model cards you are using.

I also tried the suspect Sony card in my similarly-configured D800 (14-bit lossy compressed raw) using the same 100-shot intervalometer sequence I used on my D750 and did not see an EOF NEF. Not conclusive by any means but an interesting data point.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

Any particular settings that reproduces the EOF?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

14 bit raw in my case. What's your setting?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

14 bit. Upgraded the firmware yesterday evening. I haven't managed to reproduce the Eof today either. Maybe it's time to move this thread to Nikonsupport based on your findings

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

I'll try updating my D750 firmware as well - I'm still running V1.00. Before we move this to the Nikon forum I think we need to establish if the problem is with the camera or Adobe's decoding of the NEF. engerim, I just checked your EXIF and see you're running V1.00 as well.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

I'll try upgrading too

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 28, 2014 Dec 28, 2014

Copy link to clipboard

Copied

Just tried the 100-shot intervalometer sequence on the original suspect Sony card with D750 firmware V1.01 and didn't see any EOF errors Also did a binary compare with the backup NEFs on the Sandisk and all are identical sans the EXIF-encoded slot number.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 29, 2014 Dec 29, 2014

Copy link to clipboard

Copied

Tried with 1.01 and still the same problem (did an interval shooting with 75 frames, one image with the error in both slots).

http://stuff.inxsoft.net/tmp/d750-raw-error-v101.zip

And yes, output is: ViewNX 6,016px × 4,016px vs. DCRAW 6,032px × 4,032px Not sure yet if the error happens within those pixels.

Wasn't sensor data bottom to top?

I could try to decode only 6016 rows on all concerned files and see how it goes.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 29, 2014 Dec 29, 2014

Copy link to clipboard

Copied

engerim, that sucks. My successful 100-shot sequence with V1.01 was probably just dumb luck then. I'll analyze your NEF today to verify it's the same issue. Lens -> sensor projection is bottom-up but the NEF data is top-down. Reducing the dimensions decoded to the actual output dimensions would definitely avoid the EOF.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 29, 2014 Dec 29, 2014

Copy link to clipboard

Copied

ok, added some debug to dcraw. It always fails in the last row (different columns) (note row is 0 based) on the files I shared for testing:

dcraw errors:

DSC_3992.NEF:row:4031 col:5327 - Corrupt data near 0x1c200b2

DSC_4047.NEF:row:4031 col:5329 - Corrupt data near 0x1c300b6

DSC_4077.NEF:row:4031 col:5461 - Corrupt data near 0x1c1810b

DSC_4244.NEF:row:4031 col:5795 - Corrupt data near 0x1fa8212

DSC_4065.NEF:row:4031 col:6026 - Unexpected end of file

So a solution is to fix it to 4016 rows (as per camera spec) in Camera Raw (imho).

I can't imagine Nikon would care about this.

http://imaging.nikon.com/lineup/dslr/d750/spec.htm

Spec here: FX (36x24) image area: 6016 x 4016

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Dec 29, 2014 Dec 29, 2014

Copy link to clipboard

Copied

I agree, a solution for Adobe is to modify ACR/LR to only decode to the image dimensions. But this still appears to be a bug in the D750 firmware; the extra columns/rows beyond the output image dimensions are useful for certain raw processing techniques and image analysis, such as from the optically-masked black pixels, and thus are included in the full NEF data stream by Nikon; the fact that that area has replicated/corrupt data is an issue. If Nikon didn't intend for them to be decoded/decompressed they would have left them out of the stream.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Dec 29, 2014 Dec 29, 2014

Copy link to clipboard

Copied

Ticket with Nikon opened: [Incident: 141229-001985]

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 03, 2015 Jan 03, 2015

Copy link to clipboard

Copied

Yesterday I finally got an eof in slot 1 (same memorycard I got the other eof). The copy in slot 2 was OK when I imported it to LR.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Jan 03, 2015 Jan 03, 2015

Copy link to clipboard

Copied

Thanks for the update Lattefarsan. Based on yours and engerim post-firmware update results this is not an issue corrected with the latest firmware. engerim, any word back from Nikon on your open ticket?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 03, 2015 Jan 03, 2015

Copy link to clipboard

Copied

In their system:

Status: Unresolved

Date Created: 12/29/2014 06:01 PM

Date Updated: 01/03/2015 09:53 AM

I don't expect a response before end of this week and will update the status here accordingly.

Not sure if anyone else is using 14 bit RAW and could test it (using interval mode, etc)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines