Can't create vector shape when tracing a jpeg image

It look like you are trying to create a vector version of the background layer which was made with graphic and text.  So you use text layers for the text and to create as shape layers for the graphic.  Shape layers are fill layers with a vector layer mask. The graphics in you case is  one color so you only need to create one shape layer.   You want the shape layer to have as few  control points as possible to have a smooth well defined shape.  When the graphic has a geometric  items it is helpful to  drag out guide lines you can snap pen points to so things will line up. A shape can haves many closed vector paths in the layer mask and combine is different ways.  These Paths can be edited.  Paths can be added, deleted, split transformed etc.  Here is how I would go about creating your Star Shape layer.

Mylenium is correct. You are misinterpreting how Photoshop works.

Your screenshot shows you are zoomed in to 400%. That means 16 screen pixels are being used to display 1 image pixel.

The smallest display unit in Photoshop is an image pixel. Even a vector shape is displayed using image pixels so if you zoom in to 400% you are going to see those pixel edges regardless of whether the shape was made using raster or vector tools.

The advantage of vector in Photoshop comes when you resize the image or the shape (which is different to zooming the display).  Resizing a pixel shape means combining or interpolating pixels, which softens it. Resizing a vector means the application redrawing at the new image pixel size which keeps it sharp but still with the restriction that it can only be redrawn using image pixels.

Illustrator works differently, redrawing at screen pixel size as you zoom in/out, as there are no 'image pixels'.

Dave

PS is not Illustrator. Content doesn't get re-rasterized dynamically when zooming in, so of course you only see the pixels as they would be at 100%. Feel free to scale them up later without any loss of quality. Shape layers are recolored by double-clicking on them. You may want to read the onlinhe help on some of these basics.

Mylenium