how to handle transparent parts of layers in filter plugin?

yes, I hadn't known where to specify the matting behavior till Toby pointed it out.

If you matted the data on the way in, and do not unmatte the data on the way out -- then yes, you will get fringes.

You have to be careful to specify the correct matting behavior (or no matting).

Many thanks Toby! Simply changing outStraightData in those filterCases to outWhiteMat, makes it work as expected.

The normal way of dealing with this in a filter is to define the FilterCaseInfo in the PiPL, for each of:

filterCaseFlatImageNoSelection
filterCaseFlatImageWithSelection
filterCaseFloatingSelection
filterCaseEditableTransparencyNoSelection
filterCaseEditableTransparencyWithSelection
filterCaseProtectedTransparencyNoSelection
filterCaseProtectedTransparencyWithSelection

This defines how input data is matted, or whether you wish to use the transparency data directly (see PIFilter.h and PiPL.r).

thanks, I'll check it out.

ChannelPort suite is your friend

Chris,

I'm not sure what you mean by treating it as if there was no transparency. Simply copying the RGBA planes from inData to outData results in those light-colored edges. Likewise if I set inHiPlane=outHiPlane=3 and just copy the RGB planes.

Can you tell me how to specify how the data should be matted? It seems to default to a white matte. Ideally, I would have thought I'd want to get from inData the color value at each pixel in the layer _before_ any matting is applied, along with the alpha values. Then for example, a red pixel at the edge that's 50% transparent would be RGBA = (255, 0, 0, 127). It seems like that's what my plugin needs to hand back to Photoshop (when I don't modify that pixel), in order for it to be able to correctly matte it against any background. Instead, what I get from inData is (255, 127, 127, 127), which then shows up when I hand it back as a pink pixel of 50% transparency on the edge. Can you tell me what I'm missing here?

In most cases, treat it as if there was no transparency.

And you can specify how the data should be matted.