Layer style, stroke has faceted edges

[PotatoLauncher]

This happens on particularly large canvas sizes (I'm authoring something at a print-able size), so the stroke is set to 200 pixels or so.

The darker orange in the attached image is created using a stroke on a layer, set to "inside". I created the shape that has the stroke with the pen tool, so its very smooth. I am wondering why this is happening, and if there is a way to fix it.

[michelew83603738]

Based on what I see, the lighter orange shape is not very smooth. I know you said that you created it with the pen tool, but was anti-aliased turned off when you made the path from the pen tool into a selection? Something is eroding that edge and so the stroke that you are then applying is just showing it.

[PotatoLauncher]

this is the shape with the stroke turned off, just to be clear. It is antialiased. To me, it seems like it is a resolution problem with the stroke itself. 

for instance, the same thing happens on a perfect point:

[JJMack]

Stroking a path is more complex than ones first intuition would be.  Your 200 PX stroke clearly shows for its not as smooth as you thought it would.   If you look at a normal 200 wide px stroke about a path. You will see the outside edge Path and the inside edge pate are different shapes and different then the stroked path. Your Stroke is not a normal 200PX stroke. Its more like half of a 400PX stroke. One edge of your strike is the stroke path shape.  A normal stroke shape more or less the shape of a selection the has been expanded and contracted.  The stroke center path is the path stroked. Abrupt changes in path direction effect the inner and outer path.  Photoshop has shape stroke options to deal the these abrupt changes. Along with three stroke option Centered Normal, Inner and outer.  Layer styles Stroke has the three stroke options inner, outer and centered but nor the smoothing options. So a Layer Style path can be quite different then the same Shape stroke path. You could remove your layer style Duplicate your shape layer then  edit the top shape set ts fill to none and stroke to 200 px orange  align inner and corners to rounded.  It will be a better inner stroke than the layer style stroke.

[PotatoLauncher]

thanks for the reply, though having the stroke on the shape as opposed to the straight pixels, with rounded corners set, did not solve this issue.

[JJMack]

Can you upload your example PSD and post a link to it.  A Inner stroke on a vector path should work better then inner stroking a selection or adding an inner stroke layer style. For Photoshop is basically a raster pixels editor. While Text and shape layers are vector layers many layers are not. So it would not surprise me if Photoshop Layer style feature treated all layers as raster layers.  Transforming a vector should work better than transforming a raster.   On my machine stroking a Shape Path worked better then stroking a selection and better than inner layer style stroke.

[PotatoLauncher]

OK, I see where I was going wrong with fresh eyes on this stuff this morning! I was still using a layer style. I find it strange that the stroke on a "shape" is a different option/ set of tools from the layer-style stroke. Im pretty destructive with my paths and dont really save them once I convert them to pixels, so I never really used those shape-specific options before.

So, my original question about the layer style being faceted, the answer is: there is no way to make this work, and the stoke layer style will always have this problem at a larger scale. The solution is more of a work-around. 

Good info to know for the future, though. Maybe Adobe will figure out a way to better extrapolate large strokes and other layer styles at a larger scale on rastorized pixels, since, like you said, that is the primary output of Photoshop as a program. But they havnt added anything I find useful to this program in many years so I'm not holding my breath, haha

Thank you very much!