First, make sure your AE is up to date. Second, you should not be using H.264 in a Quicktime container. That format is no longer supported by Apple and it was never dependable either. You should be rendering your compressed deliverables using the Adobe Media Encoder.
Same thing goes for MPEG 4... Use the AME.
I would suggest that you spend some time studying up on rendering by typing rendering in the Search Help field at the top right corner of AE and checking out the community resources. The Render Cue is best suited for creating visually lossless digital intermediates that you use for further editing or production. It has never been good at producing highly compressed deliverables in h.264 or any other MPEG format because the Output Module is incapable of things like multiple pass compression and inter frame comparisons.