HELP! pixelation issues...

To expand on what Doug and Ton said, the vector smart object that you pasted from Illustrator into Photoshop has the advantage of being completely vector inside the smart object. You can resize and transform the vector smart object in Photoshop without loosing quality. You can edit it in Illustrator and update the edits in Photoshop.

However, eventually, when the Photoshop document is flattened, the smart object will be rasterized to the resolution of the Photoshop document. That is why it appears pixelated when you zoom in.

By the way, a vector smart object layer is not the same as a vector shape layer created in Photoshop itself. Under certain circumstances a Photoshop vector shape layer can remain vector and output as vector. Not so with a vector smart object from Illustrator.

Like Doug said, it will look pixelated when zoomed in higher than 100%.

But with a smart object you can choose: Image > Image Size and change the resolution to something higher than 72 pixels per inch.

Make sure Resample is checked.

What is the resolution of your Photoshop document?

Anything in Photoshop will look pixellated at higher than 100% zoom. Once your Illustrator drawing is in Photoshop, it is rendered at the document resolution.