P: Jpg exposes bugs in QImage and ZoomBrowser

Report · Jun 22, 2011

Since a recent upgrade to CS5 (I guess 12.0.4 and certainly the current version) JPG files I've saved take about 1000x times longer to open in Canon ZoomBrowser 6.7.2.33 (the latest version). When I say 1000x I mean 1000x. Recent jpg are taking over 30s to open a single image! I raised this with Canon sending then old and new jpg (created using an older version of CS5 and the latest) from the same CR2 file. They said:

Extracting the EXIF data from both the good and bad images we found that the JPEGInterchangeFormatLength (JpegIFByteCount) value is bigger in the bad files.
JPEGInterchangeFormatLength shows the number of bytes of JPEG compressed thumbnail data.

We believe that this higher number is causing the problem as the ZoomBrowser EX application is trying to use the EXIF data to generate the thumbnail images, and to display the files. We were able to reproduce the issue in our test environment.

We would recommend you to contact the Adobe support in order to find out if there were any related updates released in the last few weeks that possibly was installed on your computer manually or automatically.

Please can you investigate what changed recently in CS5. And how I rescue my recent jpg images that I've needed to create for my clients. If you need the images that I sent Canon for your investigation then please let me know.

Report · Sep 13, 2011

Chris,

You say "Yes, the files differ. But I doubt that the presence or absence of profiles in the thumbnails is going to hurt anything other than adding 6324 bytes to the total image size of 6.8 Meg."

OK, at least you admit there is a problem. CS5 should not be producing different output depending on the phase of the moon and the alignment of the planets (what other files might be present in the same folder).

If Adobe is okay with the fact that CS5 inserts leftovers from other images inside saved files and it does so inconsistently (sometimes but not others), I'll find a way to work around the Adobe bug. Personally, I don't believe you should just accept "randomly" garbaged-up headers even IF parsing the JFIF tree avoids those areas, but if that's what you want to do, fine. The JPEG spec specifies an APP2 (FF E2) tag followed by "ICC_PROFILE" as the proper tag for an embedded profile. Look at the file offsets I posted and you will see that there are three APP2 tags in that JPEG file and three ICC profiles inside the file: each properly formed JPEG should only have one APP2 tag whether it is in the parse tree or not! The fact that some programs "skip around" the garbage is fine, but you still are producing garbage headers and that needs to be fixed.

Qimage is not the only program that recognizes the extra APP2 color space tags. Other programs are having issues with the extraneous data in the files which is why you have people reporting that files saved that way take ages to load in some programs, files being rejected as invalid images in other programs, etc. As I said, I can work around the problem so this has never been an issue for me and my software, but that's like fixing the effect, not the cause. I just thought Adobe might want to fix the *fact* that under the conditions I specified, CS5 is inserting chunks of JFIF headers from previously processed files into some saves. Whether that data gets parsed as part of the JFIF tree is not really relevant to the bug: it's still a bug. There's more than one way to parse a JFIF header particularly if you are searching the header for a particular tag (like APP2, of which there should only be one).

Mike

Report · Sep 13, 2011

Chris, I would LOVE to have the answers to the above three questions. Ask another aunt or uncle in the "Photoshop Family". Not trying to return the condesending attitude, but could your method of trying to discover the problem be erred? Can you give us a detailed procedure we should try to see if we can get the same results you are getting? What is your definition of a bug in the program?

Report · Sep 13, 2011

Where is "the color wrong every time"? Is it QImage that displays the files incorrectly? Photoshop appears to read the "bad" file and displays it correctly with the AdobeRGB profile in the file (ignoring the extras in the thumbnails like it should). Photoshop prints the good and "bad" files identically. (yes, I just did that)

Shouting won't help the fact that you are leaving out important details and ignoring the findings from the evidence provided.

Yes, you have problems. But that does not mean that you understand the problems, or are assigning the blame correctly. Again, there is apparently a minor bug in Photoshop that causes profiles to be included with the thumbnails sometimes (possibly randomly). Again, that should be completely harmless because the profiles in the thumbnails have no relation to the full resolution document.

Yes, insulting the senior developer who is taking his own time to investigate the problem and help you is really going to help your case. Oh, and I am the developer most knowledgable about file formats, standards, parsing, profiles, and all the related bits.

So far, we can't reproduce your results. Maybe there is a random bug, but we haven't found the triggering factor yet. Now that we know that something is writing profiles in the thumbnails, we'll investigate that code more closely.

But the claims of corrupted files, wrong color, and incorrect printing are still completely unproven. And it still sounds like you are leaving out important details.

You aren't getting any condescension -- but I'm having a hard time determining the facts of the case because you're making a variety of wild claims but providing scant evidence to back up those claims.

Report · Sep 13, 2011

The original issue was about Canon's ZoomBrowser software being slow to read files. Brett requested samples, which I don't recall ever seeing (but maybe he forwarded them to Canon without sending them to me).

Also, that still sounds like a bug in Canon's software which Canon would need to debug. I can't do much for Canon's software because I don't have the source code for their software. Even a minor change in metadata (ie: changing the date) could cause a problem if they have a bug in their code (and yes, dates have caused crashes in other software before).

Report · Sep 13, 2011

The profiles are not leftovers, and are correct for the thumbnails (which according to EXIF standard, are converted to sRGB). There is no garbage, and no corruption -- just the inclusion of the profiles. Parsing the JPEG tag tree does avoid the thumbnails, unless you're extracting only the thumbnails. Those are correctly embedded profiles in the thumbnails, completely parsable within the thumbnails. Um, no, embedded JPEG thumbnails are valid JPEG streams within the main JPEG file. That's how the EXIF standard works, plus another thumbnail/preview is embedded in a private tag for Adobe data. Again, there is no garbage, and no need to "skip around". The file parses just fine.

I hope you are mistaken about the way QImage reads JPEG tags - professional software should never be that sloppy, and should be parsing the file structure correctly. That sort of bug could lead to lots of mistakes in file parsing, especially when working with EXIF standard JPEG and TIFF images.

So, the bottom line appears to be that Photoshop has a minor bug that sometimes includes profiles in thumbnails. And QImage has some major bugs in the way it attempts to read JPEG images and picks up the thumbnail profiles by mistake.

Report · Sep 13, 2011

Brett told me on 29-Jun that he'd passed the files over to the engineering team.

OK let's start again. here's what Canon told me:
Extracting the EXIF data from both the good and bad images we found that the JPEGInterchangeFormatLength (JpegIFByteCount) value is bigger in the bad files.
JPEGInterchangeFormatLength shows the number of bytes of JPEG compressed thumbnail data.

We believe that this higher number is causing the problem as the ZoomBrowser EX application is trying to use the EXIF data to generate the thumbnail images, and to display the files. We were able to reproduce the issue in our test environment.

Report · Sep 13, 2011

So please ask me the questions you need the answers to. Do you need a detailed procedure with settings on my workflow? I am willing to tell you every step I do, I just need to know what you need know. As far as unproven, what do you need us to send you to show you the results we are getting? I can send you prints, files, screen shots, just let us know what you want to see and where to send it.
And, lastly, can you please tell me if there is anyway to "un update" and let us see if we get any better results. It may be coincidence, if that would make my workflow go smoothly again, I can live with that and turn off Adobe updates.

Report · Sep 13, 2011

What Canon told you translates to: the files have a different size. That's all it means. It doesn't tell us anything about why they take so long to read the files.
And it really sounds like that came from someone, um, not technically savvy. The issue really needs to go to Canon's developers so they can debug their code and find out why one file takes longer.

I'll check with Brett (and check the SPAM trap to see if I missed his email).

Report · Sep 13, 2011

See Mike's comments below. I think we've found the root cause: bugs in the way QImage parses JPEG files.

Report · Sep 13, 2011

OK, I did get the files from Brett back in June and just forgot about them.

Yeah, what you're looking at is definitely a bug in ZoomBrowser's code.
We've tried to talk to Canon independently, and haven't gotten anywhere.
So it's in Canon's court - if they want help identifying the bug in ZoomBrowser, they need to contact Adobe. We've done all we can.

Report · Sep 13, 2011

FYI - we got sample files, there was nothing wrong with them. This appears to be a bug in Canon's ZoomBrowser code, but we haven't been able to talk to Canon's developers to narrow it down.

Report · Sep 13, 2011

Follow up - Mike's guess was wrong. The JPEG file is just fine, but QImage's parsing code is broken and it picked up the wrong profile.

Report · Sep 13, 2011

Turns out QImage was the problem -- bugs in the QImage JPEG parsing caused it to read the wrong color profile.

Report · Sep 13, 2011

Chris,

You're welcome! Took 10 days but you do finally see that there is a bug in CS5. We started at "deny, deny", went to "that's impossible", and we've now ended up with "oh yes, there ARE three profiles in the file, but two of them are attached to thumbnails". You and I both know that those two extra APP2 profile tags don't belong in there attached to thumbnail(s) or not, which is why no previous version of PhotoShop (or any other software) has ever done that, nor should 12.0.4. That's probably why you never thought of looking there even when I pointed out the exact file offsets: because they don't belong there.

And no, it's not a minor bug. If you want to define sloppy, the fact that CS5 12.0.4 cannot consistently save the same file the same way twice, randomly inserting extraneous data into the header is a MAJOR bug. I'm sure people who regularly use file cataloging/archiving/backup software are pleased that CS5 will create completely different files on two different runs even though the image, settings, and software version are the same. Kinda makes it hard to tell if your latest conversion is really any different from the previous! Professional software can at least save the same file the same way twice.

And yes, now that you've finally looked in the file where I've been pointing for the last 10 days and you realized CS5 was embedding color spaces into thumbnails (sometimes), you found the CS5 bug and pointed out a minor issue in Qimage as a result of that info. Headers are not parsed correctly if they have undefined data chunks not defined in the EXIF specs (embedded color spaces should not be in thumbnails: thumbnails are sRGB as you pointed out above). Easily fixable and that's a lot less of a problem because I've been doing this for 13 years and Qimage does not misread an embedded profile unless it encounters a malformed header: that combo did set off a bug which I need to fix. So thank you for pointing that out. See, help can go both ways so in the future I hope we can do this without the condescending words and the Adobe pedestal.

Report · Sep 13, 2011

The extra tags are harmless to any correctly written JPEG parser. This entire issue has been about a major bug in QImage that just happened to be triggered by a very minor inconsistency in Photoshop. Inaccurate and incomplete descriptions of the problem didn't help. Someone who jumped to incorrect conclusions about the cause didn't help. Not providing the examples requested didn't help. Repeatedly insulting the people trying to help you didn't help. Etc.

I couldn't find anything at the file offsets because you posted the wrong ZIP archive and had me looking in the wrong JPEG file. You only posted the "bad" file 23 hours ago, and I responded about 2 hours later with the diagnosis. Before that you only gave us source files that failed to reproduce the problem.

Failing to parse JPEG images correctly isn't exactly a minor issue. That's like claiming to read a language because you recognize 4 or 5 words, most of the time.

Next time, consider asking for help debugging your software instead of making inaccurate claims about other companies and dragging out what should have been a 5 minute debugging session.

Report · Sep 13, 2011

Looks like you were talking about the QImage bug. Check with Mike to see when he'll have a version out that correctly parses JPEG files with color profiles.

Report · Sep 14, 2011

Follow up - the JPEG file is not "just fine". Sometimes CS5 produces JPEG's with the correct header and sometimes it produces a header that has inapplicable profiles attached to multiple thumbnails in the header. While the aberrant files *will* parse, embedded profiles are not applicable to thumbnails and the inconsistent saving behavior needs to be fixed so that the same file converted with exactly the same settings will produce the same output: not different output depending on a "leftover" state based on what was converted previously. The Qimage bug, which only surfaced due to the Adobe bug, has already been fixed. Please focus on consistent file saving with the next CS5 release.

Report · Sep 14, 2011

Already fixed and released. How are you doing on the CS5 bug?

Report · Sep 14, 2011

Qimage had a bug, yes, but it only surfaces when the CS5 bug produces JPEG's with inapplicable profiles attached to multiple thumbnails in the header.

Report · Sep 14, 2011

Which surfaced due to the CS5 bug that inconsistently produces a header that has inapplicable profiles attached to multiple thumbnails in the header.

Report · Sep 14, 2011

I see you've spent extra time replying to each individual thread trying to shift blame to Qimage and away from the bug that we both know exists in CS5. Without the CS5 bug, I would never have found the Qimage bug so yes we both have bugs. Have you fixed yours yet? I've fixed and released mine.

Bottom line: there is a CS5 bug in 12.0.4 and there *was* a bug in Qimage Ultimate v2012.103. To outline the bugs we've found in this thread (as I will not be back here because it serves no further purpose for either side at this point):

CS5 bug: Exact same file and program settings can cause a bug depending on what was converted prior. The bug is that under the right (inconsistent) conditions, CS5 12.0.4 will embed sRGB profiles into all embedded thumbnails. When it works properly (again depending on what was converted prior, even in a separate session), the thumbnails don't have the inapplicable (extra) profiles.

Qimage bug (fixed): A bug in the algorithm used to extract the embedded profile was inappropriately following the parse tree and parsing both the original image and the thumbnail images when looking for an embedded profile. The bug has never surfaced because the condition simply has never been encountered in order for it to surface: there really should only be one profile embedded in any file (including the thumbnails within). Until CS5 12.0.4, no program has ever tried to embed color spaces into embedded thumbnails as they should be sRGB by definition. With the extraneous data, the bug surfaced. The bug is fixed in Qimage Ultimate v2012.104.

BTW, I never posted the wrong file. If you follow the directions in the original ZIP file which were detailed and WILL reproduce the CS5 bug (call it a minor bug if you want, doesn't matter any more), this whole thing could have been solved 9 days ago. Instead, you're looking in the JPEG I included in the ZIP which the instructions clearly say is only there because that is a JPEG that (when present), causes CS5 to inappropriately start embedding color spaces in thumbnails. The instructions very clearly say to use that JPEG as test conditions and *examine the output files from CS5*. Had you done that, this would have been so much easier. Instead, you were so convinced there couldn't possibly be a problem in CS5 that you completely ignored the readme file in my original ZIP. I know you ignored it: or you wouldn't have been examining the sRGB profile embedded in the (superfluous) JPEG that was included. And that would have prevented you from posting the comments like all you've seen are "anecdotes and confused descriptions": a post I see you've now deleted.

Anyway, over and out. No more gained from posting here.

Report · Sep 14, 2011

Well I ;have jsut fininshed working 20 files for a client and when sent to Qimage to print only 2 of them were messed up and had re-open them in raw as jpegs and re save as. this fixed the problem and they are now ready to print. Please stop being little kids and fix the problem. I have been printing and loving Qimage for many years and this problem has only been happening for a short time now. As mentioned but not noted by me with the latest release.

Report · Sep 14, 2011

You're contradicting yourself. The JPEG images parse correctly, and doesn't affect applications that read JPEG correctly. Only QImage has problems because it failed to parse JPEG images correctly.

Yes, we will work on the very minor Photoshop issue.
But the problem discussed here is a more major QImage issue of failing to read JPEG images correctly.

Report · Sep 14, 2011

Supposedly QImage has a new version out that fixes the bug.

Report · Sep 14, 2011

I don't want to belabor this and I promise this will be my last post, but you really do need to get more familiar with the JPEG spec because the Qimage parsing issue is ancillary to the header corruption caused by the intermittent CS5 bug. The Qimage bug will never surface unless the file header is corrupted and this CS5 bug results in undeniable corruption of the header. A file corruption bug is worse than an "error handling" bug in my opinion.

And yes, those files *are* corrupted. You have a thumbnail embedded in the JPEG stream itself and as such, the spec calls for that embedded thumbnail to be free of JFIF segments. Simply put, embedding a profile in a thumbnail image IS NOT ALLOWED by the spec yet the CS5 bug is causing the APP2 JFIF segment to appear there: where it shouldn't. I suspect that might be why BreezeBrowser and other software are having issues with the files. Yes, with the right error handling, they can work around it, and maybe it can even be argued that they SHOULD work around it because embedded profiles in thumbnail images are not allowed. But they really shouldn't be there in the first place so I suspect it is possible that other software engineers have not accounted for that non-spec condition either.