AE only caches a few frames despite plenty of free space

I'm on AE 24.0.2 right now, but I've been having this issue ever since I got my M2 Pro Mac Mini (including on AE v22 and 23).

The first time I preview a comp, it will play like butter.

But then if I make a change and try to preview it again, it will only preview a tiny bit and then give up.

I've set memory reserved for other applications to the minimum (3GB) and max cache to 300GB (on a dedicated internal SSD for cache).

I've seen other Apple Silicon users make similar complaints on this forum, but no solution that works for me (usually just people telling them to update to a newer version of AE).

Any suggestions? Or is this just a bug for Silicon users?