Scaling images without losing quality while compositing

In practice, even without using Smart Objects, this should not be a huge problem. I am a product photographer and composite many of my images, and as long as I'm careful I don't see pixelation.

I use free transform to make the part smaller, and this reduces the quality of this part significantly

By @fierce_teacher98A1

Just to be safe, check what Interpolation is set to for Free Transform.  If it has become set to Nearest Neighbor, then that is going to have a significant negative impact on image quality, (not sure why I suddenly started talking in double bollox in that last sentance 😞  ).  With very sharp images, I like to use Bilinear, but don't be afraid to apply some sharpening even when downsizing.

If you do a lot of this, then apps like Topaz Photo Ai are very good at upscalling.  This is a 4X upscale of your screenshot to 6240 X 7208

it becomes pixelized and blurry if I try to scale it back a bit. 

By @fierce_teacher98A1

As Dag says, that is completely normal.  I use Smart Objects so often I begrudge the time it takes to right click the layer and hunt out Smart Object, so I have F12 set to create Smart Objects.

You should covert the pasted layer to a smart object before transforming smaller and then back to original size. This isn't magic, but it will preserve the original pixels.

Edit - looks like @D Fosse @posted while I was typing!

Pixels are pixels. If you go down, you can't go back up. Then you've lost those pixels.

Which is why smart objects were invented. This very cleverly embeds the original into the file, and keeps that original untouched. When you transform a smart object, you transform a representation, an "image", of that original. You're not transforming the original itself.

This way you can do repeated transforms, upscaling and downscaling, without ever damaging the original.

Until crunch time. Sooner or later you have to rasterize the smart object, and then it will be degraded according to the scaling factor. But the point is - that happens only once. You're not accumulating damage as you normally would.