Layer masks introduce dither?

DESTRUCTIVE

Work-Around to reduce noise in general (old school):

Make a selection of the original image.

Save selection.

Bring selection to your "dithered" layer.

Filter - tiny motion blur at 90•

Filter - tiny motion blur at 180•.

Filter - Sharpen - Unsharp mask.

(selection keeps it inbounds)

---------------------

Non Destructive

Make original layer a smart object.

Create adjustment layers on original art to taste, then drop same adjustments onto your dithered pic.

Delete adjustments from the original.

Reopen the original in a separate doc. (only because you can't copy and paste it without dither)

Make it a Smart object.

Desaturate it completely.

Increase contrast to max (b&w)

Select darkest spot, Select Color Range (use slider for all darks) - create adjustment mask.

Click art layer.

Select lightest spot, Select color range (all lights) - create adjustment mask.

Pull new art with adjustment layers into original document.

Blend if to clean up any strays or mess.

Smooth your selections and adjust layers for desired affect.

------------------------

SO SIMPLE AN IDIOT COULD DO IT.

Open it. Rename it. Save it. Open both. Pull one onto the other.

Another issue that may be involved here is that Photoshop Selections are only 8bit, even in a 16bit image.

Do you mean banding? Dither is usually the brute force solution to break up banding.

There is indeed LSB dither when dropping from 16 to 8 bpc, however I wouldn't have called it terrible, at least in the context of photography (except if you require a truly uniform flat area of solid tone, as opposed to gradations). Truly synthetic "pure" images may show the LSB dither as more unsightly compared to a natural photograph. Can you change a copy down to 8 bpc in the other software? If so how does it compare? You could turn off the dither or merge two copies together with dither on/off in flat areas.

Please post meaningful screenshots taken at at least View > 100% including all pertinent Panels (Options Bar, Layers, Channels, …) on this Forum directly.

Lets assume I take a black and white image layer within an 8-bit RGB PSD file. I copy the layer, hit quick mask, paste, quick mask again. Then use that selection to fill a layer mask.

If you are in 8 bit anyway you could just load the luminance of the composite channel in the Channels Panels (even including hiding other Layers and showing them again if necessary that would still seem more efficient to me).

Also I want to bring up something else. I noticed that if I import a 16-bit image from substance painter, then tell photoshop to drop the bit depth to 8 bit, i get horrible dithering. If I however open a new, 8-bit file, then copy all layers over, I have zero dithering.

Can you provide a file for testing?

What are your Edit > Color Settings?