There are hard limits to certain values and numbers of items in AE and while many of them have been raised again and again to accommodate certain workflows, some of them persist. There's only so much you can squeeze out of 16 bit integer math or certain string operations in strict computer terms. Latest when you reach that magic 9999 folders mark you'll be in trouble. And even if that weren't the problem, AE is simply not that great at dealing with some image formats such as Raw/ DNG, so for performance reasons alone you wouldn't want to have too many such assets in a project at the same time. They will also create enourmous caches, which could be another performance bottleneck. So by all means straighten out your project and divide it into sensible segments. AE is simply not meant to handle such complex projects on its oiwn nor is there any sensible reason to be afraid of splitting up things. Should the need arise to exchange data you can always import projects into one another, save animation presets, render reference sequences and clips, use proxies and so on. You need to educate yourself about these possibilities more than anything else.