After Effects Mask Handling for SmartFX Plugins

I just got out of a wrestling match with an AE Plugin trying to get my mask to work "after" my plugin. I'm leaving my findings here A) In case it helps and B) In case someone has a better way of doing this.

Say thanks to Claude Code for this -->

# After Effects Mask Handling for SmartFX Plugins

This document explains how to properly handle masks that are applied AFTER a SmartFX plugin renders. When a user draws a mask on a layer with an effect, After Effects crops the output buffer to optimize rendering. This creates coordinate system challenges that must be handled correctly.

## The Problem

When a mask is applied to a layer with an effect:
1. After Effects crops the output buffer to the bounding box of the mask
2. The effect must render to this cropped buffer
3. The mask alpha must be applied at the correct position within the cropped buffer
4. **The rendered content must appear in the correct location** - not stretched or offset

Without proper handling, two bugs occur:
- **Content appears in wrong location**: The rendered content gets squeezed into the upper-left of the visible area
- **Mask applied at wrong position**: The mask alpha gets applied at (0,0) instead of its actual position

## Key Coordinate Systems

```
+--------------------------------------------------+
|                                                  |
|   COMPOSITION SPACE (comp_width x comp_height)   |
|   - Full composition dimensions                  |
|   - Where 3D content is projected to             |
|                                                  |
|      +------------------------------------------+|
|      |                                          ||
|      |   LAYER SPACE (in_data->width x height)  ||
|      |   - Usually same as composition          ||
|      |   - Where UV coordinates map to          ||
|      |                                          ||
|      |   output_origin (x,y)                    ||
|      |      +---------------------------+       ||
|      |      |                           |       ||
|      |      |   OUTPUT BUFFER           |       ||
|      |      |   (output->width x height)|       ||
|      |      |                           |       ||
|      |      |   mask_pos (x,y)          |       ||
|      |      |      +---------------+    |       ||
|      |      |      |               |    |       ||
|      |      |      |   MASK AREA   |    |       ||
|      |      |      |               |    |       ||
|      |      |      +---------------+    |       ||
|      |      |                           |       ||
|      |      +---------------------------+       ||
|      |                                          ||
|      +------------------------------------------+|
|                                                  |
+--------------------------------------------------+
```

## Key Values from After Effects

| Value | Source | Meaning |
|-------|--------|---------|
| `output_origin_x/y` | `in_data->output_origin_x/y` | Where output buffer (0,0) sits in layer space |
| `masked_rect_left/top/right/bottom` | `GPUPreRenderData` from PreRender | Where the mask bounds are in layer space |
| `output->width/height` | Output buffer | Cropped buffer dimensions |
| `comp_width/height` | `in_data->width/height` | Full composition/layer dimensions |

## The Critical Calculation

```cpp
// mask_pos = where mask sits in OUTPUT BUFFER coordinates (not layer space!)
int mask_pos_x = masked_rect_left - output_origin_x;
int mask_pos_y = masked_rect_top - output_origin_y;
```

This is the key insight: **separate the rendering offset from the mask application offset**.

## Flowchart: Mask Handling in SmartFX

```
+------------------------------------------------------------------+
|                    PF_Cmd_SMART_PRE_RENDER                        |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 1. Store mask bounds in GPUPreRenderData                         |
|    gpu_data->masked_rect_left = in_result.result_rect.left       |
|    gpu_data->masked_rect_top = in_result.result_rect.top         |
|    gpu_data->masked_rect_right = in_result.result_rect.right     |
|    gpu_data->masked_rect_bottom = in_result.result_rect.bottom   |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
|                    PF_Cmd_SMART_RENDER                            |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 2. Retrieve mask data from pre_render_data                       |
|    GPUPreRenderData* gpu_data = extra->input->pre_render_data    |
|    masked_rect_left = gpu_data->masked_rect_left                 |
|    masked_rect_top = gpu_data->masked_rect_top                   |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 3. Dual-checkout input layers                                    |
|    UNMASKED: PF_CHECKOUT_PARAM(in_data, 0, ...)                  |
|              -> Full texture without mask cropping               |
|    MASKED:   checkout_layer_pixels(0)                            |
|              -> Texture with masks applied (cropped buffer)      |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 4. Get output_origin from in_data                                |
|    output_origin_x = in_data->output_origin_x                    |
|    output_origin_y = in_data->output_origin_y                    |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 5. Calculate mask position in BUFFER coordinates                 |
|    mask_pos_x = masked_rect_left - output_origin_x               |
|    mask_pos_y = masked_rect_top - output_origin_y                |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 6. RENDER: Map coordinates to LAYER space, then offset           |
|                                                                  |
|    // Map UV/screen coords to full layer dimensions              |
|    float layer_x = uv_coord * comp_width;                        |
|    float layer_y = uv_coord * comp_height;                       |
|                                                                  |
|    // Convert to buffer coordinates                              |
|    float buffer_x = layer_x - output_origin_x;                   |
|    float buffer_y = layer_y - output_origin_y;                   |
|                                                                  |
|    // Write to output buffer at buffer coordinates               |
|    output[buffer_y][buffer_x] = rendered_pixel;                  |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 7. APPLY MASK ALPHA at correct position                          |
|                                                                  |
|    for each pixel (x, y) in output buffer:                       |
|        // Convert to mask buffer coordinates                     |
|        mask_x = x - mask_pos_x                                   |
|        mask_y = y - mask_pos_y                                   |
|                                                                  |
|        if (mask_x < 0 || mask_y < 0 ||                           |
|            mask_x >= mask_width || mask_y >= mask_height):       |
|            // Outside mask - make transparent                    |
|            output[y][x].alpha = 0                                |
|        else:                                                     |
|            // Inside mask - apply mask alpha                     |
|            output[y][x].alpha *= mask[mask_y][mask_x].alpha      |
+------------------------------------------------------------------+
                              |
                              v
+------------------------------------------------------------------+
| 8. Return rendered output with mask applied                      |
+------------------------------------------------------------------+
```

## Example: Before and After Fix

### Before (Incorrect)
```
Mask at (748, 513) in layer space
Output buffer: 1467x1533 at origin (0, 0)

WRONG: UV (0,0) -> (1,1) maps to buffer (0,0) -> (1467,1533)
       Content fills entire buffer but should only fill part of comp
       Mask applied at buffer (0,0) - clips wrong region
```

### After (Correct)
```
Mask at (748, 513) in layer space
Output buffer: 1467x1533 at origin (0, 0)
mask_pos = (748-0, 513-0) = (748, 513)

CORRECT: UV (0,0) -> (1,1) maps to layer (0,0) -> (3840,2160)
         Then offset: layer coords - output_origin = buffer coords
         Mask applied at buffer (748, 513) - clips correct region
```

## Code Implementation

### 1. PreRender: Store mask bounds

```cpp
static PF_Err PreRender(PF_InData* in_data, PF_OutData* out_data,
                        PF_PreRenderExtra* extra) {
    // ... setup code ...

    // Allocate pre-render data to pass mask info to SmartRender
    GPUPreRenderData* gpu_data = new GPUPreRenderData();

    // Store the result rect (mask bounds in layer space)
    gpu_data->masked_rect_left = in_result.result_rect.left;
    gpu_data->masked_rect_top = in_result.result_rect.top;
    gpu_data->masked_rect_right = in_result.result_rect.right;
    gpu_data->masked_rect_bottom = in_result.result_rect.bottom;

    extra->output->pre_render_data = gpu_data;

    return err;
}
```

### 2. SmartRender: Retrieve and use mask info

```cpp
static PF_Err SmartRender(PF_InData* in_data, PF_OutData* out_data,
                          PF_SmartRenderExtra* extra) {
    // Retrieve mask data from PreRender
    A_long masked_rect_left = 0, masked_rect_top = 0;
    if (extra->input->pre_render_data) {
        GPUPreRenderData* gpu_data = (GPUPreRenderData*)extra->input->pre_render_data;
        masked_rect_left = gpu_data->masked_rect_left;
        masked_rect_top = gpu_data->masked_rect_top;
    }

    // Dual-checkout: get both masked and unmasked versions
    PF_ParamDef unmasked_param;
    PF_CHECKOUT_PARAM(in_data, 0, ...);  // Full texture

    PF_EffectWorld* masked_input = NULL;
    extra->cb->checkout_layer_pixels(in_data->effect_ref, 0, &masked_input);

    // Get output origin
    int output_origin_x = (int)in_data->output_origin_x;
    int output_origin_y = (int)in_data->output_origin_y;

    // Calculate mask position in buffer space
    int mask_pos_x = (int)masked_rect_left - output_origin_x;
    int mask_pos_y = (int)masked_rect_top - output_origin_y;

    // Render with offset
    RenderWithOffset(output, unmasked_input,
                     in_data->width, in_data->height,  // Full layer dimensions
                     output_origin_x, output_origin_y);

    // Apply mask at correct position
    ApplyMaskAlpha(output, masked_input, mask_pos_x, mask_pos_y);

    return err;
}
```

### 3. Render function: Use layer dimensions and offset

```cpp
void RenderWithOffset(PF_LayerDef* output, PF_LayerDef* texture,
                      int layer_width, int layer_height,
                      int output_origin_x, int output_origin_y) {

    for (each_element) {
        // Calculate position in LAYER space (using full layer dimensions)
        float layer_x = element.normalized_x * layer_width;
        float layer_y = element.normalized_y * layer_height;

        // Convert to BUFFER space (subtract origin)
        int buffer_x = (int)layer_x - output_origin_x;
        int buffer_y = (int)layer_y - output_origin_y;

        // Bounds check against buffer dimensions
        if (buffer_x >= 0 && buffer_x < output->width &&
            buffer_y >= 0 && buffer_y < output->height) {
            // Write to output
            WritePixel(output, buffer_x, buffer_y, color);
        }
    }
}
```

### 4. Apply mask alpha at offset position

```cpp
void ApplyMaskAlpha(PF_LayerDef* output, PF_LayerDef* mask,
                    int mask_pos_x, int mask_pos_y) {

    int mask_width = mask->width;
    int mask_height = mask->height;

    for (int y = 0; y < output->height; y++) {
        for (int x = 0; x < output->width; x++) {
            // Convert output position to mask position
            int mask_x = x - mask_pos_x;
            int mask_y = y - mask_pos_y;

            float mask_alpha = 0.0f;  // Default: outside mask = transparent

            // Check if inside mask bounds
            if (mask_x >= 0 && mask_x < mask_width &&
                mask_y >= 0 && mask_y < mask_height) {
                // Read mask alpha
                mask_alpha = GetAlpha(mask, mask_x, mask_y);
            }

            // Apply to output
            MultiplyAlpha(output, x, y, mask_alpha);
        }
    }
}
```

## Common Mistakes

### Mistake 1: Using buffer dimensions for content mapping
```cpp
// WRONG: Maps content to cropped buffer size
float x = uv * output->width;

// CORRECT: Maps content to full layer, then offsets
float layer_x = uv * layer_width;
float buffer_x = layer_x - output_origin_x;
```

### Mistake 2: Using masked_rect directly for mask position
```cpp
// WRONG: Uses layer-space coordinates in buffer
ApplyMask(output, mask, masked_rect_left, masked_rect_top);

// CORRECT: Converts to buffer-relative coordinates
int mask_pos_x = masked_rect_left - output_origin_x;
ApplyMask(output, mask, mask_pos_x, mask_pos_y);
```

### Mistake 3: Assuming buffers align at (0,0)
```cpp
// WRONG: Assumes mask and output start at same position
for (int y = 0; y < min(output->height, mask->height); y++) {
    output[y].alpha *= mask[y].alpha;  // 1:1 mapping
}

// CORRECT: Accounts for offset between buffers
for (int y = 0; y < output->height; y++) {
    int mask_y = y - mask_pos_y;
    if (mask_y >= 0 && mask_y < mask->height) {
        output[y].alpha *= mask[mask_y].alpha;
    } else {
        output[y].alpha = 0;  // Outside mask
    }
}
```

## Summary

The key principle is **separation of concerns**:

1. **Rendering offset** (`output_origin`): Tells you where the buffer sits so you can render content at the correct layer-space position
2. **Mask position** (`mask_pos = masked_rect - output_origin`): Tells you where to apply the mask alpha within the buffer

Always render to layer/composition coordinates first, then convert to buffer coordinates by subtracting `output_origin`. Apply mask alpha at the buffer-relative mask position, not the layer-space mask bounds.