how to export AE composition with transparent background

On the After Effects side, make sure that you can see the Transparency Grid in the Composition panel.  You can then add the Comp to the After Effects Render Queue and choose one of the following Render Settings Templates:

• Lossless with Alpha
  Results in an Animation compressed QuickTime file with an alpha
  Available in most versions of AE for a long time now
• High Quality with Alpha
  Results in an Apple ProRes 4444 compressed QuickTime file with an alpha
  Only available in the current version of AE

On the Media Encoder Side, presets that will include an alpha channel in the exported file have "with alpha" in the preset name.  I usually set the Format to QuickTime and then choose Apple ProRes 4444 with alpha.  Whatever format/CODEC you choose, the video CODEC must support embedding an alpha channel and ProRes 4444 is on the short list of CODECs that do.

If have to use a format/CODEC that does not support an embedded alpha, you can render an RGB pass and an Alpha Only pass.  For example, after setting the format to H264 and the preset to Match Source - High Bitrate, you can click the Add Output button and then click the preset name for the second output to open the Export Settings dialog box and check "Render Alpha Only".  Media Encoder with append "_1" to the end of the filename, but it's common to change this to "_matte".  The resulting two files must be kept together.

You may hear about including an alpha channel in an H265 file.  This is something Apple came up with via AV Foundation Frameworks.  If running macOS, you'd render Apple ProRes 4444 with alpha from AME then right-click the resulting file, choose Encode Selected Video Files, choose one of the HEVC options, enable the "Preserve Transparency" checkbox, and then click Continue.