after effect crashes soon as start using roto brush

Use Rotobrush 1 instead of 2. Here's the workflow. Open the footage in the footage layer by double-clicking on the source file in the Project panel. Set in and out points in the footage panel, then create a new composition from the trimmed footage. Set the Composition Resolution to Full.

If your footage is already trimmed and in a comp, select the footage, duplicate the footage, select the top copy, and pre-compose moving all attributes to the new comp and open the new comp. Set the comp resolution to Full. You are now at the same starting point you would be at if you had followed the first set of instructions.

Open the Paint workspace because workspaces make your life easier and the system works better. Double click on the trimmed footage layer to open it in the layer panel. You now are ready to start using Rotobrush. 

Pay close attention to these steps. Hold your finger over the Caps lock key, select the Rotobrush tool and get ready to click and not drag anywhere inside the area you want to mask. Now Click with Rotobrush and immediately press Caps Lock to prevent AE from trying to render anything else or propagate. 

You should now be able to go to the Effects Control Panel, see Rotobrush applied to the footage layer, and change the Version from 2.0 to 1.0. Once you have done that you can release the Caps Lock key and proceed with the regular Rotobrush workflow. If your first click resulted in a bad selection you select the layer, press the 'u' key twice to reveal all modified properties and delete the Foreground 1 stroke from the timeline. I use this technique to remove bad strokes or edge refinements all the time. When you have checked all frames carefully, freeze Rotobrush, then take a good look at the matte and make adjustments to the Rotobrush Matte and Refine Edge settings after Robrush has been frozen. You can even change the size of the strokes that were added. The only thing you can't do is change the path shape of a path, but you can move it, change the size, and fine-tune it to your heart's content. 

I hope this helps. I am afraid that we are stuck with this bug and version 1 until Apple fixes their OS. It was the update to OSX 11.5.1 that caused this problem. It doesn't have anything to do with Adobe.

Moved from the Premiere Pro forum to the After Effects forum.