"Refine Soft Matte" on tracked pre-comp results in black shadow

[benjamin Philipp]

I'd like to use a soft matte on a pre-comp layer, based on its existing intermediate opacity map; i.e. starting from what's visible as-is, and give that soft edges.

So I thought I could use the "Refine Soft Matte" effect to do just that, but this results in a black shadow around the layer contents, as if they're composed of a foreground pasted on a black background, and the background areas masked out by the mask that gets softened.

The strange thing is, it's still a black shadow when I give the pre-comp a white background, I still get a black shadow.

How do I get rid of that?

The pre-comp uses a tracked 3D layer, but I turned all material options off so I think that should not be the problem.

It feels like this should be solved by clicking one button, but I couldn't for the life of me find it 😅

[benjamin Philipp]

When I shift the edge inwards, I can get past the black area. But I feel like I shouldn't have to do that; such a black shadow is much more jarring than a jump in opacity over the regular background 🤔

[Rick Gerard]

Without seeing the timeline and the modified properties of the problem layer, it's pretty hard to figure out where the problem is coming from. 

Your lights should be in the Main Comp, and Collapse Transformations should be turned on in the nested comps (pre-comps). The material options should be properly set up. Only the Classic 3D rendering engine fully supports lights and shadows. I don't know if you are just fiddling with the alpha channel or if you have a track matte setup. I have no idea how the layers and lights are arranged in the comps. I also don't know if you are using Layer Styles/Drop Shadow or casting a shadow from a light. 

Give us some more workflow details and screenshots showing the modified properties of the problem layers, and we can probably help you figure it out.

[benjamin Philipp]

Thank you!
Sorry, to my mind that was all the relevant info; Attaching more screenshots and the project files now.
There are no layer effects, no lights. In fact, all the 3D-ness I need of that layer is the transformation of the tracked floor texture.

It's not actually a shadow, it's just surrounding blackness creeping in through the soft matte. I'm just calling it a "shadow" because that's what it ends up representing visually.

Ugh, .zip files are not allowed 🙄
Guess it would make things too neat to have a properly collected project 😛

[Rick Gerard · Accepted solution]

If there is no soft edge (feathered edge) when you apply "Refine Soft Matte," the Matte modifies the alpha channel but not the color. If you expand the matte, the RGB values in the expanded area are going to be [0, 0, 0], so you get a black soft edge.  Expanding the Refine Soft Matte effect will reduce the feathering if there is already a soft edge. 

If you want to soften the edge, you need to either reduce the shift edge or use another matte tool. 

[benjamin Philipp]

Thank you!
This pointed me in the right direction, I think. Applying Channel > Set Matte to the pre-comp layer before (underneath) Refine Soft Matte seems to do the trick!

Left at their defaults, it takes the matte Source from 1. Pre-comp 1, using its Alpha Channel to create an explicit matte.

(Correct me if I'm misunderstanding anything)

Looks like that's doing what I thought should be the implicit default behavior.

[Rick Gerard]

You have got it.