is it possible to make round custom corners like in Illustrator using Pen tool?

In Photoshop, the rounded corner handles are available only for Live Shapes, which are drawn with the shape tools, not the Pen tool. When you draw a Live Shape, you can see one or more rounded corner handles on the shape path, and also a rounded corner field in the Properties panel.

But Photoshop does not yet offer those handles for arbitrary paths you draw with the Pen tools. If you need that level of control, draw the path in Illustrator and then paste or import it into Photoshop.

It is not clear whether Adobe plans to add those in the future for the Pen tool in Photoshop, but in the Ideas section of this community you can check to see if someone else has already requested it (if it exists, you can vote it up), or submit your own request.