Weird Crop tool behaviour with 16-bit files

[erickellerman]

Shadow areas of my 16-bit files show strange patterns if I use the Crop tool to rotate. I can also replicate the problem by doing the following:

Make a new empty 16-bit file in PS6 in RGB mode (File>New). Fill it with 100% black. Use the Crop tool and rotate to taste. Press Enter to accept the crop and zoom in to enjoy the black and grey noise patterning reminiscent of a man's suit. The Straighten tool will achieve the same result. Your suit pattern will vary according to the degree of rotation.

The problem goes away if you subsequently convert the file to 8-bit, or convert to any other mode (Greyscale, Lab, CMYK).

This patterning is easily visible at 100% view or less, and appears whether the View>Show>Pixel Grid option is on or off. It is decidedly NOT the pixel grid I am seeing.

This effect can be created with files from both my Canon and Hasselblad cameras.

I'm using a Mac Pro with OS10.6.8.

[Noel Carboni]

Chris, when I look at the pattern.psd file Eric posted, there are no artifacts.  Not even in the transparency channel.

Honest!

It must be something that's happening to the file when being opened on your particular system.

I did a Layer - Layer Mask - From Transparency, copied the mask, and pasted it as a new layer.  It's pure white.  The histogram says so and two layers of extreme curves adjustments say so.

The artifacts are NOT present in that file!

Edit:  Scratch that.  I went through it again and saw something different.  Eric, did you update the file at the link posted above?

-Noel

[Chris Cox]

It's a 16 bit file, and the errors are small.

Open patterns.psd, Layer -> Layer Mask -> From Transparency, option/alt click on the layer mask to show it, Levels and change the black point to 245, commit the levels adjustment, now repeat the same levels adjustment and you should see the artifacts.

The errors appear to be in the least significant 2 bits of the result.

But resampling solid white should always result in solid white -- so something when wrong before we saw that document.

[Noel Carboni]

Chris I've been able to reproduce the issue in a new document.  Here's how:

• File - New, 400 x 300 px x 96 ppi, 16 bits/channel, Transparent
• Set foreground color to pure black
• Bucket fill
• Choose Crop Tool
• Uncheck [  ] Delete Cropped Pixels
• Click the Straighten Button
• Drag a line across the image right and upward until 5.0 degrees is indicated.
• Complete the crop.

This reproduces the problem whether a Grayscale or RGB image is created.

I'm thinking another inaccuracy is making this ultra slight transparency error visible on some GPU-accelerated modes with some drivers.

-Noel

[Chris Cox]

It looks clean because the error is very small (4 / 32768).

But something is making it more visible on your system, and so far I haven't gotten it to be visible without using two extreme levels or curves adjustments.

[Noel Carboni]

FYI, the transparency channel gains the same minor transparency glitches with just Image - Image Rotation - Arbitrary.

Note that these glitches don't really look like what Eric and Peter are seeing.  That *could* be another issue entirely.

-Noel

[Paulo Skylar]

I will add my observation on the unexpected behavior of the transparency channel. While I do not see the behavior noted in the original posting, I do see the transparency error reported by Chris and Noel. It is only present in 16-bit mode and only with  my graphics processor turned on ( Nvidia 9600 GT, CS6). 

In my case I see someting not yet mentioned. I can readily see the error on the display, which should be displaying pure black (Fill canvas with black, crop tool, rotate). It appears to be an error of a few percent on the display so if the calculation error is in fact only two LSBs in the mask data, then it appears as if the higher & lower order bytes are switched in writing the transparency channel ....... or some other error.

Paulo

[Chris Cox]

Paulo - can you get a screenshot of what you're seeing?  I'm starting to suspect this is a display pipeline error, but it may only happen in certain circumstances (which I haven't yet reproduced).

[Paulo Skylar]

Here are  screen shots at 100% magnification.  The canvas was 100 x 100 pixels and using the crop tool I "straightened" the image by 40 degrees (if I remember correctly) - different angles yield different patterns, of course.

Here is another view with a levels adjustment

I had a medium, blue transparency  grid, which can be resolved.

Paulo

[Noel Carboni]

I'm only barely seeing the pattern you are seeing in the first of the two images you posted, Paulo - it looks almost perfectly black from here.  If I work at it I can just barely see the pattern, especially if I zoom in.  Looks like just a 1 or 2 level shift above pure black in that image.

It occured to me the appearance / absence of this might just be differing responses to very dim level changes with different display systems, so I purposefully reproduced the issue and did a screen grab, but nope - I'm still getting a perfectly black display.  I think Chris must be right - something not right in the display pipeline on only some systems.

-Noel

[Chris Cox]

And byte order reversal was actually my guess as to the cause - but I need to reproduce it and then figure out where in the pipeline it gets messed up.

[Paulo Skylar]

If you want a dramatic visual of the effect Noel, set the blend mode of the "all black" layer to Vivid Light and place a color fill layer beneath. Something like this:

Paulo

[Noel Carboni]

Good trick.  But here, no red.

Only with extreme enhancement does the issue show.

-Noel

[Paulo Skylar]

Noel,

Indeed, the effect on your system is much weaker. I also see no pattern in the unadjusted image .

Strange!

Paulo

[Noel Carboni]

Summarizing the two (or more) issues...

Problem root: 

• In 16 bits/channel mode, something in the pixel math during an arbitrary canvas rotation appears to be causing a loss of precision in the transparency mask for a layer, resulting in fully opaque pixel values (0x8000) being changed for some pixels to be one or several levels smaller.

• The error/inaccuracy appears as a repeating noise-like pattern or something that looks a bit like suit pin-stripes (on different systems).

• It's not a certainty that the noise-like pattern and the pin-stripes pattern are one and the same issue, but it seems likely.

Visibility of miniscule transparency differences:

• On some systems, but not all, the slight inaccuracy in the transparency mask becomes visible, where it clearly shouldn't be seen.  It needs to be verified whether transparency works otherwise normally on such systems.

• It's interesting that we've heard from ATI users and nVidia users, Mac users and PC users.  Amongst the ATI PC users, some see it and some don't (possibly a video driver version issue).

-Noel

[erickellerman]

This is fascinating, but has actually become too technical for me. My question is, is this now a Photoshop problem that needs solving, or is it just my and others' bad luck?

[Noel Carboni]

The first of the two is almost certainly a Photoshop problem.  The other may be rooted in how Photoshop is using particular video cards, or maybe a bug in your display driver that may be able to be solved by seeking an updated version and downloading/installing it from the web site of the maker of your video card.  It certainly seems that some people have display subsystems that don't show the problem.

In either case you have the attention of one of the more senior Adobe engineers, so there's not much else you need to do to get Adobe working on it.

Something to try as a test:  Go into your Preferences - Performance dialog, select the [Advanced Settings...] button, change the Drawing Mode to Basic, then quit and restart Photoshop.  Do you still see the problem, and do you still have all your GPU-accelerated functions?

-Noel

[erickellerman]

Thanks for all your trouble, Noel. And thanks to Chris too.

I did your test, and find that I do not get gentlemen's suiting patterns using Basic Mode.

[Noel Carboni]

Hm, that may imply that the GPU-resident color-management code is the place for Adobe to look for the root of the bug.

-Noel

[Chris Cox]

I'm more worried about the GPU transparency visibility issue than the small errors in interpolation (which we'll get to later).

But I'm running the same machine, video card and OS as one of the users, and can't reproduce the problem.  What are we missing?

[Noel Carboni]

Chris Cox wrote:

But I'm running the same machine, video card and OS as one of the users, and can't reproduce the problem.  What are we missing?

Advanced Photoshop GPU settings? Different display resolution (a stretch, I admit)?

I was going to say double check the display driver version, but assuming you're talking about a Mac, that's covered by the OS version being the same.

-Noel

[Paulo Skylar]

Two pieces of information I failed to mention previously: The transparency visibility problem occurs on my system (PC, Nvidia, CS6) with either Normal or Advanced GP drawing modes enabled. It does not occur with the Basic mode setting. Using the same system but running CS4, the problem does not present.

Paulo

[Noel Carboni]

FYI, I've been testing in Advanced mode and don't see the problem.

-Noel

[Chris Cox]

OK, that means that the issue is related to the color transform shaders in OpenGL. (converting from document colorspace to display colorspace)

[conroy]

There seems to be a problem with transparency in relation to image resampling. Not only image rotation is affected. Up-sizing is, too.

Up-sizing 16-bit images with Bicubic Smoother (or Bicubic Sharper) and Bicubic Automatic (which will use Smoother when up-sizing, of course) causes a grid pattern in the transparency of the layers' pixels (except Background which has no transparency, of course).

The Info panel's Opacity readout is in whole percentage units, so it is no aid in examining this phenomenon. Use "Layer > Layer Mask > From Transparency" then copy the mask's pixels and paste as a new layer. That'll provide the transparency values as an examinable and adjustable image.

Invert that transparency image then boost exposure, e.g. +16, to see the pattern clearly.

Different scaling factors produce different patterns. The example below was up-sized to 900%.

Background isn't affected, of course, since it never has transparency.

[Chris Cox]

You're showing low bit errors -- that is a minor problem, made visible only by extreme adjustments.

We'll get that fixed when we can.

This topic is talking about a major problem that makes low bit errors visible on some GPUs when they should not be visible.

We need to get a reproducable case of this so we can figure out where the bug is (in Photoshop, in the driver, etc.) and get it fixed.