Enable Selection and Mask Support for a Plug In

Report · Dec 22, 2018

Hello,

I would like to add Selection / Masking awareness to my Plug In.

The problem is I can't get the information about the Selection from Photoshop (At least not according to my understanding of the documentation).

Here is a simple test case.
Lets say we have an image (Background Layer only) of size 400 x 400 (Width x Height).
The user used Rectangular Marquee Tool to select a rectangle area of 50 x 50 (Width x Height) which it top left corner is on pixel 51, 51 (Row Number, Column Number). So it masks pixels [51:100, 51:100].

The question is, how can I get the coordinates of this selection.
Here is the relevant documentation in PIFilter.h (Assuming using non BigDocument for the simplicity):

/** @ingroup FilterModule

*/

/** This structure is passed to the plug-in module through the parameter block.

* See @ref PGPluginEntryPoint "Plug-in Entry Point" for an explanation of how the

* host calls a plug-in.

*/

typedef struct FilterRecord

{

Point imageSize; /**< \deprecated Use \c BigDocumentStruct::imageSize32. The width and height

of the image in pixels. If the selection is floating, this field instead

holds the size of the floating selection. */

Rect filterRect; /**< \deprecated Use \c BigDocumentStruct::filterRect32.

The area of the image to be filtered. This is the bounding

box of the selection, or if there is no selection, the

bounding box of the image. If the selection is not a perfect

rectangle, Photoshop automatically masks the changes to the

area actually selected (unless the plug-in turns off this

feature using autoMask). This allows most filters to ignore

the selection mask, and still operate correctly. */

Rect inRect; /**< \deprecated Use \c BigDocumentStruct::inRect32.

The area of the input image to access. The plug-in should set

this field in the \c filterSelectorStart and

\c filterSelectorContinue handlers to request access to an

area of the input image. The area requested must be a subset

of the image’s bounding rectangle. After the entire

\c filterRect has been filtered, this field should be set to

an empty rectangle. */

Rect outRect; /**< \deprecated Use \c BigDocumentStruct::outRect32.

The area of the output image to access. The plug-in should set

this field in its \c filterSelectorStart and \c filterSelectorContinue

handlers to request access to an area of the output image.

The area requested must be a subset of \c filterRect. After

the entire \c filterRect has been filtered, this field

should be set to an empty rectangle. */

void * inData; /**< A pointer to the requested input image data. If more than

one plane has been requested (see \c inLoPlane and \c inHiPlane), the

data is interleaved. */

int32 inRowBytes; /**< The offset between rows of the input image data.

The end of each row may or may not include pad bytes. */

void * outData; /**< A pointer to the requested output image

data. If more than one plane has been requested

(see \c outLoPlane and \c outHiPlane), the data is interleaved. */

int32 outRowBytes; /**< The offset between rows of the output image data.

The end of each row may or may not include pad bytes. */

Boolean isFloating; /**< Indicates if the selection is floating. Set to TRUE if and only if

the selection is floating. */

Boolean haveMask; /**< Indicates if the selection has a mask. Set to true if and only if

non-rectangular area has been selected. */

Boolean autoMask; /**< Enables or disables auto-masking. By default, Photoshop

automatically masks any changes to the area actually selected.

If \c isFloating=FALSE, and \c haveMask=TRUE, the plug-in can

turn off this feature by setting this field to FALSE. It can

then perform its own masking. <br><br>

If the plug-in has set the PiPL bit \c writesOutsideSelection, this

will always be FALSE and the plug-in must supply its own mask, if

needed. */

Rect maskRect; /**< \deprecated Use \c BigDocumentStruct::maskRect32.

Provides a mask rectangle. If \c haveMask=TRUE, and the

plug-in needs access to the selection mask, the plug-in

should set this field in your \c filterSelectorStart and

\c filterSelectorContinue handlers to request access to an

area of the selection mask. The requested area must be a

subset of \c filterRect. This field is ignored if there is

no selection mask. */

void * maskData; /**< A pointer to the requested mask data. The data is in the

form of an array of bytes, one byte per pixel of the selected

area. The bytes range from (0...255), where 0=no mask

(selected) and 255=masked (not selected).

Use \c maskRowBytes to iterate over the scan lines of the

mask. */

int32 maskRowBytes; /**< The offset between rows of the mask data. */

int16 imageMode; /**< The mode of the image being filtered, for example, Gray Scale, RGB Color,

and so forth. See @ref ImageModes "Image Modes" for values. The \c filterSelectorStart

handler should return \c filterBadMode if it is unable to

process this mode of image. */

Point floatCoord; /**< \deprecated Use \c BigDocumentStruct::floatCoord32.

The coordinate of the top-left corner of the selection

in the main image’s coordinate space. */

Point wholeSize; /**< \deprecated Use \c BigDocumentStruct::wholeSize32.

The size in pixels of the entire main image.

*/

///@name New in 3.0.

//@{

int16 filterCase; /**< The type of data being filtered. Flat, floating, layer

with editable transparency, layer with preserved transparency,

with and without a selection. A zero indicates that the host

did not set this field, and the plug-in should look at

\c haveMask and \c isFloating. See @ref FilterCaseIdentifiers for values. */

Fixed inputRate; /**< The sampling rate for the input. The effective input

rectangle in normal sampling coordinates is

<code> inRect * inputRate. </code> For example,

<code> (inRect.top * inputRate,

inRect.left * inputRate, inRect.bottom * inputRate,

inRect.right * inputRate). </code> The value for

\c inputRate is rounded to the

nearest integer in Photoshop 3.0.1+. Since the scaled

rectangle may exceed the real source data, it is a good

idea to set some sort of padding for the input as well. */

Fixed maskRate; /**< Like \c inputRate, but as applied to the mask data. */

//@}

///@name Reserved Space for Expansion

//@{

char reserved [46]; /**< Reserved for future use. Set to zero. */

//@}

}

FilterRecord, *FilterRecordPtr;

Looking at this I would assume the data would be available in filterRect as its description says:

This is the bounding box of the selection, or if there is no selection, the bounding box of the image.

Yet it always holds the size of the image itself.

Anyone could comment on how it should be done?First for Selection and later for Masks (Soft Selections).

Thank You.

Report · Dec 22, 2018

Have you tried setting `autoMask` to false?

Report · Dec 22, 2018

qu1j0t3 wrote
Have you tried setting `autoMask` to false?

I haven't tried it (Yet).
The first try I do a rectangle selection hence I thought anything with Mask in it are less relevant.

Do you suggest that setting `autoMask` to false might make Photoshop update the filterRect data?
Do you have any experience with it?

Thank You.

Report · Dec 27, 2018

OK,

It seems we have figured it out.

The trick is to set the mask rectangle and then do the advance routine.

So it would be something like:

VRect filterRect = GetFilterRect(); // Basically the image size unless something weird
SetInRect(filterRect);
SetOutRect(filterRect);
SetMaskRect(filterRect); // Setting the rectangle the plug in needs
gFilterRecord->advanceState(); // This makes Photoshop look at the data and update it according to the rectangle
pMaskImage = gFilterRecord->maskData;
maskRowStep   = gFilterRecord->maskRowBytes;

Basically one should define the zone of the mask one needs.
Then one should make "Advance State" routine which then makes Photoshop update the data in `gFilterRecord->maskData`.

If someone from Adobe could add more information (Official) it would be great.

Thank You.

Enable Selection and Mask Support for a Plug In

Explore related tutorials & articles