I need to cut out a shape without defined edges, like a tree out of forest background

You can use Select and mask from the upper control bar… Please post a pic to evaluate it better

You've had some good answers, but I am going to add that it depends on the particular situation, like the contrast between the tree and its background.  So if you have an example image you can share, I think you might get more relevent help.  Add it to the thread at full resolution for best result.

what you are trying to do is sort of an advanced thing in Photoshop. But there is a new tool called the  Object Selection tool.  All you have to do  is drag your mouse so that  you select around your object (tree) then Photoshop AI does the work for you and selects the object. See image

Or simply select the layer with your tree/forest and from the Menu select Select>Subject. It doesnt always work 100% but you  could be lucky! Depends on how much contrast there is between the shapes and colors. Then you can press Command +J (Ctrl+J on PC) to jump that selection to  its own layer. 

hope that helps!

mark

Depends on the complexity of the tree. You'd basically start out with any of the Selection Tools and then move on with Select and Mask to refine the edges of the tree. For more info:

https://helpx.adobe.com/photoshop/using/select-mask.html

(Take it all in slowly if it seems overwhelming; there's lots to take in.)