Could you give us all the possible details of the footage you are working with or what resolution your composition has, to get a clearer idea. Also in what resolution factor you have adjusted your composition panel when working and in the Preview panel. Also what GPU does your system have and how much VRAM. All the details are important.
I can also suggest that you first try to update to Big Sur 11.5 so that all the macOS drivers and system are updated.