I have been having an issue with After Effects for the last several days now, where I have been trying to learn how to use the rotobrush tool. The clip I'm working on is a 15 second video filmed on a mobile phone, so is not particularly high fidelity. However, when propogating, After Effects consumes all available memory to the point where it either runs out of memory to use and the program crashes or my whole computer crashes. From people I have consulted with, my system should be able to handle this. But not being a particularly computer savvy person, I don't know if this is the case or not. I have watched a number of tutorials on how to use the rotobrush and I believe I'm using it correctly. But I'm not completely sure, as it is a new tool to me, so I could easily be missing a step.
I made a video explaining what I'm trying to do, as well as detailing my computer specs and After Effects settings as there may be something in there that is causing the issue.
Too many fine details and the RB simply chews through them. As a start you may want to switch to the legacy version mode. This may work better than the AI-bolstered modes in v2. Other than that you could likely easily mask out the scene manually after simply stabilizing it with a conventional 2D track stabilization (position and rotation).