I've been trying to use Rotobrush 2.0 on 'best' settings but it doesn't propagate past around frame 100, an error message comes up "After Effects Error: Unknown Exception" Afterwards the playback source is a black screen.
I have tried emptying and relocating the Disk Cache, and using the 17.7 beta. I'm using 17.5.1. but have the exact same issue.
I had less of a problem using the standard version of rotobrush but the error still comes up sometimes.
Have you properly set up the Span for Rotobrush in the Layer panel? How long is your clip? What is the frame rate? Does the clip frame rate match the comp frame rate? What is the frame size of your clip/comp? What are your Preferences/Memory? 8GB is barely adequate for HD comps and you need to have at least 2 to 4 GB for other applications.