H.264 files should be in an MP4 container. Apple never did a good job supporting h.264 in a QT container, it was buggy and unreliable, and they are not doing any further development. That is why it was removed. It should have been removed a long time ago.
You should be using the Adobe Media Encoder to render h.264 MP4 files and unless you have expert skills and a thorough knowledge of video formats and compression you should never see the words Custom in a composition setting or a render setting. Just use the presets that match your intended audience and then study up on video formats and compression.
BTW, this question has been asked and answered hundreds of times. Many of those answers are on this forum and they are all the same. Do not use H.264 Quicktime. It's a dead format.