Anyone knows the solution for stacking problem?

[MrHackSaw]

I'm trying to solve the problem with stacking. I am going to describe the example that you can reproduce to see what is the problem.

Create a new image in RGB color space. Let it be 10x10 for example. Make there two layers, one of them should be complete white, another complete black, what gives us layers of 255,255,255 and 0,0,0 in rgb in every pixel. Make sure that every of your pixels in a layer are equal to each other.

Then make a smart object of your two layers. Then in Layers->Smart object->Stacking mode use "Variance" or "Standard deviation" mode.

Now check the image in pallete. You will see that:

1) you image values are not equal, some of values are a bit higher and a bit lower than least average pixel values. You can see it applying autocontrast filter as an example or in Levels menu.

2) your Variance pixel value results will be 186 - does not match the formula.

3) your STD pixel values will be 219 - does not match the formula.

The problems are:

1) Photoshop official tutorial shows the formula for variance and std stacking and this formula equals to statistical variance and std. But if you use this formula then you will see that the result differs from photoshop's one. Example for first pixel of two images (x=0, y=0):

pix1=[255,255,255]

pix2=[0,0,0]

normalized1=[1.0,1.0,1.0]

normalized2=[0.0,0.0,0.0]

mean=[0.5, 0.5, 0.5]

variance=[((1.0-0.5)2 + (0.0-0.5)2 )/(2-1), ...] = [0.5, 0.5, 0.5]

std=[sqrt(0,5), ...] = [0.7, 0.7, 0.7]

back to [0:255]: variance=[127.5, 127.5, 127.5] ; std=[178, 178, 178]

You can see that the result differs from photoshop's one.

2) the problem of random rounding of mean stacking mode when some values are 127 and some are 128.

So I tried to find the correct formula which is used by photoshop to have the same values for STD and VARIANCE stacking modes but I could not.

Anyone knows why do I get wrong values and maybe can help me to find a mistake in my computations?

[c.pfaffenbichler]

»1) you image values are not equal, some of values are a bit higher and a bit lower than least average pixel values. You can see it applying autocontrast filter as an example or in Levels menu.«

Have you tried working in 16bit? 

[MrHackSaw]

Yes, this solves only a rounding problem of MEAN stacking mode and does not solve the problem of wrong Variance and std values.

[c.pfaffenbichler]

Actually it should have solved the noise issue you mentioned and on which I specifically quoted you. 

As for the formula … what sense does Standard Deviation make for two Layers/values? 

edited 

[c.pfaffenbichler]

I do apologize, in the example I posted I had made a mistake that lead to an apparantly meaningdul result – but it was indeed not.

So I cannot get a correct result with the formula either … so far at least.