You can create that effect with a layer mask and a brush.
If your photo is open in Photoshop as a background layer, in the Layers panel click the lock on the right side of the Background layer to turn it into a regular layer.
Add a new layer (click the 2nd icon from the right at the bottom of the Layers panel or go to the panel menu and select New Layer...).
Drag the new layer to the bottom of the layer stack.
Fill the new layer with white, if it isn't white already.
Select the photo layer and click the 3rd icon from the left at the bottom of the Layers panel to add a layer mask.
Select the Brush Tool and go to Windows > Brushes to select a brush to make the ripped edge effect. (You might try something like tiger scratch in the Dry Media Brushes. You may need to adjust the size or opacity to get the effect you want.)
Set the foreground color to black, make sure the layer mask is selected, and draw where you want the rough edge to be.