how to handle transparent parts of layers in filter plugin?

[_rgreene62_]

What is the right way to handle layers in a filter plugin so as to preserve the colors at the semi-transparent edges of the opaque parts of a layer?

It seems the inData my plugin gets from Photoshop already has the image composited (based on the alpha channel) with a white background. So even if I just copy all the RGB and alpha data from inData back to outData, I end up with light-colored edges around the non-transparent parts of the layer. Simply setting inHiPlane=outHiPlane=3 (since all I really want to modify is the RGB data) doesn't seem to work either.

[Chris Cox]

In most cases, treat it as if there was no transparency.

And you can specify how the data should be matted.

[_rgreene62_]

Chris,

I'm not sure what you mean by treating it as if there was no transparency. Simply copying the RGBA planes from inData to outData results in those light-colored edges. Likewise if I set inHiPlane=outHiPlane=3 and just copy the RGB planes.

Can you tell me how to specify how the data should be matted? It seems to default to a white matte. Ideally, I would have thought I'd want to get from inData the color value at each pixel in the layer _before_ any matting is applied, along with the alpha values. Then for example, a red pixel at the edge that's 50% transparent would be RGBA = (255, 0, 0, 127). It seems like that's what my plugin needs to hand back to Photoshop (when I don't modify that pixel), in order for it to be able to correctly matte it against any background. Instead, what I get from inData is (255, 127, 127, 127), which then shows up when I hand it back as a pink pixel of 50% transparency on the edge. Can you tell me what I'm missing here?

[_Ivan_Kharin_]

ChannelPort suite is your friend

[_rgreene62_]

thanks, I'll check it out.

[qu1j0t3]

The normal way of dealing with this in a filter is to define the FilterCaseInfo in the PiPL, for each of:

filterCaseFlatImageNoSelection
filterCaseFlatImageWithSelection
filterCaseFloatingSelection
filterCaseEditableTransparencyNoSelection
filterCaseEditableTransparencyWithSelection
filterCaseProtectedTransparencyNoSelection
filterCaseProtectedTransparencyWithSelection

This defines how input data is matted, or whether you wish to use the transparency data directly (see PIFilter.h and PiPL.r).

[_rgreene62_]

Many thanks Toby! Simply changing outStraightData in those filterCases to outWhiteMat, makes it work as expected.

[Chris Cox]

If you matted the data on the way in, and do not unmatte the data on the way out -- then yes, you will get fringes.

You have to be careful to specify the correct matting behavior (or no matting).

[_rgreene62_]

yes, I hadn't known where to specify the matting behavior till Toby pointed it out.