Resolve internal hyperlinks across multiple ICML files

I have a project with multiple ICML files (generated via Pandoc, plus some additional scripting), with internal hyperlink references between files—so one file might define an anchor like:

<ParagraphStyleRange AppliedParagraphStyle="ParagraphStyle/Header2">
  <CharacterStyleRange AppliedCharacterStyle="$ID/NormalCharacterStyle">
    <HyperlinkTextDestination Self="HyperlinkTextDestination/#substrates" Name="#substrates" DestinationUniqueKey="2394463933"/>
    <Content>Substrates</Content>
  </CharacterStyleRange>
</ParagraphStyleRange>

and another file reference it like:

  <HyperlinkTextSource Self="htss-16" Name="#substrates" Hidden="false">
    <CharacterStyleRange AppliedCharacterStyle="CharacterStyle/Link">
      <Content>Substrates</Content>
    </CharacterStyleRange>
  </HyperlinkTextSource>

  <!-- ... -->

  <Hyperlink Self="uf-16" Name="#substrates" Source="htss-16" Visible="false" DestinationUniqueKey="2394463933">
    <Properties>
      <BorderColor type="enumeration">Black</BorderColor>
      <Destination type="object">HyperlinkTextDestination/#substrates</Destination>
    </Properties>
  </Hyperlink>

This all seems to work fine, if the <HyperlinkTextDestination> appears earlier than the <Hyperlink>.

But if a <Hyperlink> in an earlier file refers to a <HyperlinkTextDestination> defined in a later file, InDesign doesn't seem to know what to do with it — it rewrites the <Destination> of the <Hyperlink> to point to the first <HyperlinkTextDestination> it knows about, and I end up with a dozen links all pointing to the same (wrong) destination early in the document.

I've tried using <CrossReferenceSource> instead of <HyperlinkTextSource> and it seems to be broken in the same way (as well as causing additional problems like spurious out-of-date cross reference warnings that can't be cleared).

Is there a workaround for this? If I could somehow explicitly reference the destination ICML file in the link, would InDesign be smart enough to resolve that? Or is this just not possible unless all the links are internal to the same ICML file?