Photoshop CS5 Subtract shape from smart object

A Smart Object is not directly editable. But you can double-click on the layer thumbnail to open the psb connected to the Smart Object. There you can substract a shape!

Hello, do you want to use a custom shape to mask a smart object, or remove a shape from within a smart object?

In the first case, CMD(mac)/Ctrl(Pc)+click the add layer mask icon, to add a vector mask, then use the pen tool in path mode, or add a shape in path mode in the Option bar.
If it is the second case, double-click the smart object thumbnail in the Layer panel to be able to edit its contents.

Let me move this to the Photoshop forum for you, where you are more likely to receive an answer to your question.

The Community Help forum is for help in using the Adobe Support Community forums, not for help with specific programs. Product questions should be posted in the associated product community.