Outside stroke with square corners

Another option
Create New Layer. Make a Selection (if needed). Create Work Path (if needed)
Select Path. Fill Path (w/ White or Black). Right Click Filled Layer
Select: Blending Options
Under > General Blending (Bottom Option)
Blend If: Grey • 'This Layer' Move White Slider to Left or Right Slider Right to remove previous fill of White or Black
Check: Stroke > Size: whatever you want. Position: INSIDE
*If you make is into an action can be pretty quick. Also remains editable via Transform Tool/Option

I think that you need to redraw the raster layers requiring a stroke as a vector shape layer, where you have proper control over the stroke – don't use layer styles!

It might be possible to convert a raster layer to a vector shape layer with a script, however, I'm not sure if I'm the person to make such a script.

Good luck.

You may have to play with the action a bit...

Your off-list sample file had a stroke of 39 px.

I had to set a Maximum filter of 28 px. When I used 39 px the result was too thick, which was unexpected, so YMMV.

Then I made the stroke 56 px. It doesn't have to be double, just large enough that the rounded corners extend past the mask.

As you have rotated/angled content, I don't think that this will work for you as the corner is chamfered, not sharp:

@Kaj35945084nura 

The following action works fine in my test setup, but yours may differ. Presuming a transparent layer with a raster fill with a stroke layer style applied and does not have any existing layer mask or clipping group applied. The following animation has a 2 second frame delay on the loop showing the round to square corner that the action delivers:

* Run the action on the selected layer.

* When prompted, set the Maximum filter to the size of the required stroke thickness in pixels (i.e. 8 px)

* Edit the stroke layer style and double its thickness (i.e. from 8 to 16 px)

Download link:

https://www.dropbox.com/scl/fi/n40xme9q3el2c85f5oiq6/Mask-Stroke-Style-to-Square-Corners.atn?rlkey=l6fpgukbfvj803u2i3ktv5a4e&dl=0

@Kaj35945084nura 

Can you provide a sample PSD file?

As I previously wrote, vector shape layers avoid the problem. Otherwise, a workaround using a layer mask or clipping group layer with an oversized stroke is also possible.

Vector shape layers have these miter options and avoid these issues.

I have added my vote for layer styles on a raster layer. Until such a thing is available, depending on the use case, I can recommend a work around using a layer mask or clipping group layer with an oversized stroke – it is multi-step so best put into an action or script.

Stohlmeyer shows a great example of how to accomplish this.

Yes! This drives me absolutely crazy. I wish Adobe would give us greater control over the outside corners like Kevin points out in Illustrator.

So far my workaround has been to go ahead and set my outside stroke for however many pixels (ex. 20px) and let it do it's rounded-corner thing. Then with the Marquee tool (set at 0 for the corners) select the area of the outside stroke and create a solid color layer matching stroke color. Put it directly below the layer with the outside stroke and link the two layers together. It's a PITA, but so far seems to get me the results I wanted.

@familybees perhaps what would be helpful is to allow the same stroke controls as InDesign or Illustrator which have join, corner and bevel controls: