Well, that's a pretty clear indication that something isn't working right with your graphics card and in turn AE's hardware acceleration. i.e. the buffers are filling up without ever being freed up again. Start by updating your graphics driver and check al lthe relevant hardware acceleration settings in AE. Also flush your caches and verify their status.