The relevant samples are IO (for video and audio files), and FBIO (for frame-based formats, or stills).
NOTE: It is not possible to override AE's file handling for subtypes of supported file formats.
So, while you can't take over AVI handling for a specific codec or subtype, you're free to write support for .xyz file format.