JPEG and color management

[_Mark_J_Peterson_]

Hi. Is it impossible to get 100% identical colors (if we leave aside the jpeg compression artefacts) when saving a .jpeg file with Photoshop using the highest quality setting (12) ?

[_Mark_J_Peterson_]

They are certainly experts.
They made comments in another thread with another topic.

This thread is about color management issues with JPEG-encoding. The other thread is about differences in JPEG-decoding and has nothing to do with color management.

At least, that's the status quo of what we found out so far.

[Ramón_G_Castañeda]

>This thread is about color management issues with JPEG-encoding.

I don't see the connection

[_Mark_J_Peterson_]

... ok how else can it be explained that the majority of patches have wrong colors even if stored with the highest possible jpeg quality settings (12) in Photoshop ?

And why are calibration targets available as jpeg files, if the colors cannot be accurate ??

[Ramón_G_Castañeda]

>And why are calibration targets available as jpeg files

NEVER came across one I would use myself!

[Lou Dina]

Mark,

JPG files ARE color managed properly, which you can see by assigning or converting a JPG file inside Photoshop. I'm not intimately knowledgeable about exactly how the compression algorithm works, but I am guessing you are seeing the effects of rounding, compression, squeezing pixels that are 'nearly identical' to take on identical values, etc (sort of a gentler version of Indexed Color mode).

As I read throughout this dialog, you are talking about very small differences from the original, most of which will be unseen visually, at least at high quality compression levels. You may be able to see these differences at high magnification levels, but that isn't what JPG is for.

I agree with Ramon...I never use JPG for high quality work, only TIFF or PSD. Having said that, you would be hard pressed to see a difference between a high rez, high quality jpg (level 10 or higher) and a TIFF, either on screen or in print, at normal magnification levels. I only use JPG's for web, email, and those situations where someone just cannot accept a large TIF.

Lou

[_Mark_J_Peterson_]

Lou Dina, Thanks a lot for your take on it.

>I agree with Ramon...I never use JPG for high quality work, only TIFF or PSD.

me neither, I want to know how it works though.

> you are talking about very small differences from the original, most of which will be unseen visually, at least at high quality compression levels. You may be able to see these differences at high magnification levels, but that isn't what JPG is for. ... you would be hard pressed to see a difference between a high rez, high quality jpg (level 10 or higher) and a TIFF, either on screen or in print, at normal magnification levels.

You know, that is exactly what I've thought for all my life ... until some days ago. I used to think that JPEGs only suffer from the typical artefacts along hard edges or in areas with high local contrasts and that with the hightest possible quality settings these artefacts could only be seen at very high magnification levels. That's problem (A) of JPEGS.

But as it turns out, aside from problem (A), JPEGs also suffer from another problem (B ), which has nothing to do with magnification level. From the demonstration above, you can see that the majority of color patches have wrong colors! (all white pixels in the picture above denote different pixels). You can see that the differences not only occur along the borders between color patches and around letters and numbers (problem A), but that the affected color patches have a wrong color for their entire area respectively (problem B ). These color patches could be 1 meter (or a couple of feet), too if you want, and they would still have the wrong color. So it has nothing to do with magnification.

Try the detailed steps above for yourself and see the results. Compare the colors of the color patches in the JPEG file and in the PNG file.

[Lou Dina]

Mark,

I followed the discussion and believe that your results are valid. But, if a color is changed from 0,0,0 to 1,0,0 or 0,1,0, you won't see the difference. The colors that are most likely to show any visible changes are light neutrals and skin tones, and even in those, changing one digit will be inconsequential. If the numbers change too much, of course, it will be noticeable. Well calibrated and profiled monitors and printers are not 100% precise, and they all have to deal with rounding, imprecise quantities of ink, paper differences, ambient viewing light, imperfect calibration, etc. There is no guarantee that the same exact file sent to the same exact paper and printer will reproduce exactly the same two times in a row, (in fact, I'd be surprised if they did), or that the custom printer profile will render colors perfectly, even if the colors are in gamut.

If you find large color shifts, then we have a problem and some of the colors may really look different to our eyes. Have you found significant shifts? If so, I'd be interested in knowing.

But miniscule shifts are essentially meaningless, especially when you consider the purpose of JPG's, which is to greatly shrink file size for continuous tone images. It's a lossy format. It's like converting a high quality audio file to MP3....data is lost, but I don't seem to notice it, even though it is there. Every time you make an edit in Photoshop, (even on a TIF or PSD file) data is lost, but this is considered acceptable if the overall image benefits.

I don't mean to discourage your investigation as to exactly how jpg compression works and how it alters images. This is especially relevant if you are developing software or doing something esoteric. But, it doesn't surprise me that there are minor shifts in the numbers. My bigger concern is how far I can go before it becomes noticeable to my eye or I create ugly artifacts.

Let me know if you find some significant color shifts. That would be interesting.

Thanks,

Lou

[_Mark_J_Peterson_]

> Well calibrated and profiled monitors and printers are not 100% precise, and they all have to deal with rounding, imprecise quantities of ink, paper differences, imperfect calibration, etc.

Yes of course. But the more varying factors you add, the worse your outcome will get. Hence the desire to limit differences whenever possible.

> But, if a color is changed from 0,0,0 to 1,0,0 or 0,1,0, you won't see the difference. The colors that are most likely to show any visible changes are light neutrals and skin tones, and even in those, changing one digit will be inconsequential. If the numbers change too much, of course, it will be noticeable.

You can easily verify it. Take original PNG source file. Save it as JPEG with highest quality settings (12) and compare the colors in square C9. Even with the very highest quality settings, you get a considerable differnce:

PNG file: RGB=(198,180,163)
JPG file: RGB=(198,181,165)

--> difference=(0,+1,+2)

A difference of (0,+1,+2) is certainly visible, especially if the original color is pure gray, because then, the jpeg's color would have a considerable cyan color cast.

And don't forget, this is just the very first image that "crossed my way". Other images might lead to even bigger differences!

(Somewhat unrelated, but if you read the abovementioned thread, you will see an example in which JPEG-decoding leads to differences of (+16,-2,-30))

[_Mark_J_Peterson_]

That's why I still wonder why there are calibration targets in JPEG format, if JPEG colors cannot be trusted even for the highest quality JPEGs ???

[Lou Dina]

Mark,

I hear what you are saying, but I doubt most people would notice that small a difference. I guess if you were doing an A:B comparison, you might see some difference, but even that is questionable.

Take a series of perfectly neutral gray blocks of color (r=g=b), convert to high quality jpg (10-12 level) and see what happens. If you get some serious differences, I'm interested. Otherwise, it seems pointless for daily usage. People will continue using compression algorithms to make images sized for web and email use. Maybe one format is slightly better than the other, but I have heard no complaints regarding jpg format, except for the artifacts cause by too high a compression level. JPG is best used for continuous tone images (photographs). There are other formats better suited to illustrations and graphics that have large blocks of solid colors.

For the best print, press, lightjet and high end work, TIF and PSD are the way to go.

What exactly is the point of this exercise?

Lou

[_Mark_J_Peterson_]

> Take a series of perfectly neutral gray blocks of color (r=g=b), convert to high quality jpg (10-12 level) and see what happens. If you get some serious differences, I'm interested.

I have no example handy, but if a pixel like (140, 107, 130) can suffer from JPEG compression, nothing suggests a gray pixel like (140, 140, 140) would somehow be exempted from JPEG compression.

> What exactly is the point of this exercise?

No offense, but you already asked that question. Is it illegitimate for me to simply wish to understand ?

> People will continue using compression algorithms to make images sized for web and email use.

Yes, but calibration targets have to be 100% accurate. So if it's really true, that colors in JPEG files cannot be accurate even at the highest quality level, than I ask myself why calibration targets exist in JPEG format.

[Ramón_G_Castañeda]

Calibration targets in JPEG format are practically always compressed from the already existing TIFF or PSD original file.

Their existence is a useful clue to the low quality and low standards of whoever is offering or using them. In that sense, their existence does fulfill a useful purpose.

Another common use for such JPEG target files is to allow for a quick and dirty eyeball verification of the approximate state of calibration on things like laptop screens and inexpensive secondary palette monitors. JPEG deficiencies such as the one Mark has exposed would never be a factor in such rough and tumble eyeball exercises.

The JPEG format in those instances also allows for slow dialup connections.

[_Mark_J_Peterson_]

I see. Thanks for pointing that out Ramón.

[_Peter_Figen_]

Just out of curiosity, what profile targets are being supplied as jpegs? Every one I've ever seen has been a zip compressed tiff, which, because of the nature of the color patches, compresses as much as most jpegs. There would no need for patch target files to be anything else. Even profile targets uprezzed for vendors who think they need 300 dpi for such things are still only a meg or two.

The stock imaging world seems to get along just fine on number 8 quality jpegs. That's what the vast majority of stock images are and you see them everywhere you look, in every newspaper and magazine you look at, both in editorial and advertising images. If there was any real problem with them aside from a jpeg artifact once in a while, you'd hear about it and maybe even see it. Small things you see while pixel peeping almost never are visible in print.

[_Mark_J_Peterson_]

Thanks for your comments Peter Figen. I don't have a link handy right now, but I can post it here the next time one crosses my way.

Peter, reading your post, I'd say the bottom line of this thread is:

* When converting an image to JPEG there can be small color shifts even on very large areas, as demonstrated with the color patches turning white in post #2.

* Color management still makes sense, even for JPEG files, because it assures a relatively high color fidelity for quality-12-jpegs (although with slight color shifts compared to tiff)

* If a user would be...
... okay to accept the JPEG artefacts, because at the print resolution and with quality-12-jpegs the human eye can't see them anyway
... but not okay to accept (even slight) color shifts on large areas
==> this user would have to stick to lossless formats, for example tiff.

Right ?

[Lou Dina]

Mark,

I'm using PSCS2. I just created a 200 ppi 21 step grayscale in sRGB color space (black to white gradient, followed by 21 step posterization). I duplicated the image, leaving the original untouched. Then I saved the duplicate file as a JPG, quality 10. Just to be certain, I closed the JPG file and reopened it, so I was working with previously saved, previously compressed data. The sRGB tag was saved with the file.

I compared the two files using the sampler tool. I tried both 3x3 and point sampler. The files are the same. I don't get any skewing of data or changes in color in any grayscale patch. They measure and look the same.

I also dragged the JPG on top of the original (holding the shift key to guarantee perfect alignment), set the top layer to Difference, and got a solid black image. No differences at all. JPG seems to work fine for its intended purpose. No color shift and color management works as intended.

I then RE-saved the jpg file a second time (opening and resaving jpgs is usually a no-no) but with a quality 6 setting, then closed and reopened it. Comparison showed zero difference in any of the 21 patches, even along the edges. Works fine. BTW, both were saved as "baseline standard" jpgs.

I agree, for satisfying intellectual curiosity, it's a great exercise, and worth doing. Nothing wrong with digging in and learning. I do it all the time and I'm glad I did the above. Now I can relax and get on with processing high quality TIFs and PSDs, and compressing them for web and email using jpg.

Lou

[Lou Dina]

Mark,

To correct my previous post, there were some visible artifacts along the edges of my 21 step grayscale patches, noticeable at jpg quality 6, but I had to increase magnification above 100% to see it. At very high magnifications it can be seen fairly easily.

Lou

[_Mark_J_Peterson_]

Thanks a lot for your tests Lou.

> I also dragged the JPG on top of the original (holding the shift key to guarantee perfect alignment), set the top layer to Difference, and got a solid black image. No differences at all.

No, this is wrong. There are differences. If there were no differences, the JPEG compression was lossless, which is certainly impossible (at least for the image you used). You can easily verify that. Carry out the steps described in post#13, starting at step7 (as in your example you have already done steps 1-6 from what I understand).

In a nutshell: Just select the magic wand with tolerance 0, anti alias:no, contiguous: no, sample all layers: yes
and click anywhere in the image. You will see JPEG compression patterns. Your selection will not cover the whole image, as it is not entirely black.

But you are certainly right when saying the differences are negligible. And what's important, is that the large areas keep its color.

[Gernot_Hoffmann]

Lou,

Gray remains gray in the JPEG workflow - even for
the worst quality.
The luminance channel Y is treated much better
than the (residual) color channels Cb,Cr.
Except for the best quality (this depends on the
program), the colors are handled by subsampling:
averaging 4 boxes 8*8 for one result 8*8.
This reduces the file size already by 50%.
The most important size reduction happens by
Quantization (see tutorial below). This is much
stronger for Cb,Cr than for Y.
http://www.fho-emden.de/~hoffmann/jpeg131200.pdf
According to older tests, the gray content of an
average photo consumes about 75% of the file size.

All this (including your test) isn't directly related
to Mark's observations: different results for decoding
the same JPEG by different programs.

The color target is here:
http://www.fho-emden.de/~hoffmann/targetrgb.txt
Replace the extension txt by eps, load in PhS and
assign sRGB. The numbers are accurately in sRGB.
Target patches with small squares were out-of-gamut
for sRGB, but these patches are consistent in the file.
The numbers are here on p.5:
http://www.fho-emden.de/~hoffmann/a3gencolorhigh.pdf
The target contains a step graywedge, R=G=B.

Best regards --Gernot Hoffmann

[_Mark_J_Peterson_]

Thanks for looking into this thread, too, Gernot Hoffmann. Of course the two threads are related, but "different results for decoding the same JPEG" (no saving to jpeg), that's the other thread. This here is about color distortion when saving to jpeg.

I don't know if you read my comment earlier on, but some graypatches are not gray:

>I have opened the .eps file you posted above in three different programs (as PS CS3 can't open it) and there are two patches in the grayscale row, that they don't show as pure gray. The grayscale patch in column 7 has the RGB values (150,150,149) and the grayscale patch in column 8 has the RGB values (139,138,138). All other grayscale patches have equal amounts for R,G,B. Are these rounding errors from the LAB to RGB conversion? If not, what else ?

[Gernot_Hoffmann]

Lou and Mark,

sorry for the Forum confusion.

If the gray patches are not accurately reproduced
by R=G=B, then we might have had rounding errors
in the inverse set of this conversion:
Y =+0.299 R +0.587 G +0.114 B
Cb=-0.169 R -0.331 G +0.500 B
Cr=+0.500 R -0.419 G -0.081 B

But it's hard to believe, because the coefficients
should be accurate in this sense: R=G=B delivers
Cb=0 and Cr=0. The DCT for zeros delivers zeros,
and so on.

Lab is not involved.

Of course I've done immediately a test: Save for Web
in PhS CS2 with quality 50% delivers equal numbers
R=G=B in each gray patch.

Best regards --Gernot Hoffmann

[_Mark_J_Peterson_]

No propblem.
Here is the the lossless conversion to png, so you can see for yourself:

http://www.imageboo.com/files/dupil5jjanyti9wfzvik.png

grayscale patch in column 7: RGB=(150,150,149)
grayscale patch in column 8: RGB=(139,138,138)

all other grayscale patches have equal values for R,G,B

[_Mark_J_Peterson_]

> Just out of curiosity, what profile targets are being supplied as jpegs?

Peter Figen, I just happened to stumble across some. Look here:

http://www.colourmanagement.ca/resources.shtml

[Lou Dina]

Thanks, Gernot....your technical and mathematical analysis confirms what I have seen with my eyes and my 'low tech' approach.

Mark,

I will continue to watch this thread and see what everyone turns up.

For my purposes, however, I am completely satisfied with JPG compression (for its intended purpose) and the fact that colors are reproduced accurately (except in artifact areas). My testing shows that patches of color remain unchanged. If this is the case with neutral grays, then it is a moot point (at least for me) as we stray farther from the neutral axis.

If I have to work THAT HARD to show any color shift due to rounding or compression, it seems irrelevant, at least for routine work.

I don't see why pasting a JPG version of the file over the uncompressed TIFF original in "difference" mode wouldn't show all the different pixels? When I do this and set magnification to 400% or higher, you can see the artifacts. The eye dropper, when set to "point sample", does not detect ANY differences except along the borders of two adjacent patches, where artifacts clearly exist. This can be done at maximum magnification and I don't see ANY shifting of pixel values.

Have fun!! I'll check in from time to time to see what everyone turns up.

Lou

[_Mark_J_Peterson_]

> I will continue to watch this thread and see what everyone turns up.

great, thanks

> For my purposes, however, I am completely satisfied with JPG compression (for its intended purpose) and the fact that colors are reproduced accurately (except in artifact areas).

Hm, not accurately in the sense of 100% identical but maybe we could say barely visible to the human eye ?

>My testing shows that patches of color remain unchanged.

But I showed you above a clear test case (it was actually the very first picture that crossed my way, so it wasn't specially chosen), where I showed you a color shift of (0,+1,+2) ( post #32) No disrespect, but it is enough to provide a single counter-example to prove your statement wrong.

>I don't see why pasting a JPG version of the file over the uncompressed TIFF original in "difference" mode wouldn't show all the different pixels? When I do this and set magnification to 400% or higher, you can see the artifacts. The eye dropper, when set to "point sample", does not detect ANY differences except along the borders of two adjacent patches, where artifacts clearly exist. This can be done at maximum magnification and I don't see ANY shifting of pixel values.

I'm not sure if I understand right what you did. I just meant that pasting the jpg over the original file, setting its blending mode to difference, using the magic wand with abovementioned settings on it and filling the selection with white clearly shows you which pixels are 100% identical and which are not.