Have tried updated and previous versions and a bunch of different kinds of videos with the same issue. Ae will let me select a subject with the rotobrush 2 but crashes immediately when trying to move a frame forward or backward.
Other stacks of effects with plugins on full res work just fine so this seems to be specific to the rotobrush :'(
This has been reported a number of times but since you haven't provided any system information or specific details about your project nobody can advise on fixes outside the usual "Update your graphics driver, check your hardware acceleration settings and clean out your caches."
I'm on a MacBook Pro M1 13" using the latest macOS Big Sur 11.5 and have Ae updated to the latest version. The project is only one mp4 layer in 1080p but I've tried other types of videos with the same result.
Ive seen others mention the recent macOS update has caused issues with the rotobrush so I will try running a previous version to see if it is a workaround. I use roto all the time so I will provide an update if it works!
I was running my MacBook Pro 13" on the recent Big Sur 15.1 update. Downgrading to any of the previous versions allowed the use of rotobrush with no issues, large and small projects. Here's the YouTube link to the tutorial video on how to downgrade MacOS with USB: https://youtu.be/TDUYiMIT4jM