Photoshop; pixelation of transformed smart objects

[josh tusjak]

I've converted a floor swatch to a smart object and am applying it to a room scene. The floor swatches are much higher resolution than is what's needed for the scene, yet when transformed, they don't seem to be anti-aliasing properly. Additionally the "anti-alias" check box is grayed out (and unchecked while grayed out) when attempting to free-transform. Attaching screenshots below (left is the swatch applied in the room scene, right is the swatch embedded within the smart obejct, both screenshots taken at 100% zoom).

I've tried adjusting the interpolation method, upscaling the swatch resolution (and a few other weird workarounds like doubling the layer with a slight offset to the transform and applying transparency to mitigate the pixelation) but nothing looks right. I can't upscale the overall image much larger as it's already difficult to modify due to mem and scratch disk limitations.

If anyone has ideas please let me know! I haven't noticed this happening in other work before; I have a feeling if the "anti alias" check box issue can be figured out then it may solve the issue.

[josh tusjak]

In addition to the above, with a little tinkering, the same pixelation occurs even if free-transforming the swatch without it being a smart object (ie transforming as fully raster layer); in those cases I've tried all interpolation methods and all results are quite poor, and functionally identical in result to transforming the swatch as a smart object.

[Stephen Marsh]

I'm posting this as it might help someone else:

https://community.adobe.com/t5/photoshop-ecosystem-discussions/resize-smart-object-via-selectable-in...

[c.pfaffenbichler]

Please try the following: 

Convert the Smart Object to a Smart Object, open that, double its Resolution and save/close it. 

Could you provide the file for testing? 

[josh tusjak]

Unfortunately I can't provide the file as it's related to client work, company policy etc etc.

That said your suggestion does help mitigate the problem; it doesn't solve it entirely but since quadrupling the resolution of the nested smart object it does help! increasing the resolution any more seriously kills my system (.psb is approaching 2GB, whoof) but I think I can manage thanks to your suggestion. Thank you!

[josh tusjak]

Issue has been resolved with a combination approach, including c.pfaffenbichler's suggestion above. posting steps below in case anyone stumbles on this issue themselves someday:

- create a new smart object of the existing smart object, thereby nesting a smart object within a smart object

- double-click into newly nested smart object, increase image dpi 3x or 4x (i went from 300 to 1200 dpi), confirm

- double-click into the original smart object within the nested SO, use Quick Mask and Gradient tool to create a gradient selection so that the parts of the swatch furthest away (ie compressed the most by the vanishing point) from viewpoint are selected, fading to the foreground with 0% selection

- apply Lens Blur (for me depending on image, between 4px and 12px blur)

confirm all changes, save to update smart objects at each nested layer

Hope that helps anyone reading this in the future.