How to do this tearing effect

Add a layer mask to the flag layer.

Find a really rough textured brush. I used this one from Kyle's mega pack, but modified by adding 50% angle jitter in  Brush settings.

And brush around the dge with black

Similar but I increased Scatter jitter to 100% and Count to 3

I am sure you get the idea.  Angle jitter keeps changing the brush shape and scatter moves it about giving you the ragged outline.

Make sure you have the search window turned on in the right click brush panel so you can use search words like rough, splatter, texture etc.  It only looks for presets in the main window, so Kyle's megapack makes a good source because it has hundreds of presets and they are all very good.