Smooth edges in Photoshop

Quick addendum:

Aside from resolution, the concept of anti-aliasing plays an important role here as well.

anti-aliasing

(in computer graphics) a technique used to add greater realism to a digital image by smoothing jagged edges on curved lines and diagonals.

Photoshop treats the anti-aliasing of transformed (rotated) vector shapes very differently compared to transforming (rotating) a bitmap (rasterized) version.

Here is a quick comparison:

(view this image at 100% to see the difference between version).

If a vector shape is rotated, the anti-aliasing applied to edges is quite sharp. This is the main reason why you see the 'steps': the anti-aliasing is a smite insufficient to reduce those steps. A shallow rotation angle is harder to hide with anti-aliasing as well (generally). The -21.6 degree rotation needs slightly more anti-aliasing applied for it to become less grainy looking.

Compare the two other version on the right with the first version: these were transformed AFTER I rasterized the rectangle first. Rasterize, then transform/rotate.

In Photoshop's transformation properties you can choose a different image interpolation (bicubic, bicubic sharper, etc.). Check out the result in the demonstration image, and notice that the two results on the left are noticeably more anti-aliased. The one on the far right is actually a bit too smooth looking.

As a result, when viewed at 100%. these two will seemingly look less grainy compared to the first one.

Anyway, if you need a smoother result, rasterize the vector shapes first, and then transform the bitmap version. Select an interpolation method to adjust the anti-aliasing.

Another option is to use filters (sharpen filters, for example) to subtly adjust the anti-aliasing afterwards.

Alternatively, I always rasterize vector work at much higher resolutions, and scale down the bitmap version to a lower resolution one to achieve the best quality edge anti-aliasing.