Mikael,
This sounds like one of the many quirks (perhaps bugs) related to how FrameMaker handles non-FM files in books. The current book model doesn't play well with XML files as chapters and this seems like yet another problem. Apparently, if an xref does not target another .fm file in the book, Frame assumes that the target file is not in the book and therefore will not be in the book PDF.
There have been discussions here about this in the past. The solution that I use is to run an API client before publishing that converts all XML files to .fm files, and redirect all xrefs appropriately. Then, book features work as expected and PDFs come out as normal. This is not feasible, however, without the automation from the API client.
There may be some who would say that the best approach is to use XML and books the way that the designers did account for... that is, the whole book becomes an XML document with entity references to the separate chapters. In my work, though, this model was not appropriate... rather, I needed a binary FM book to act like a book, except with XML files as chapters. So, I think I understand your situation.
Is API programming or FrameScript an option for you? I think it may be the only way to get what you are looking for.
Russ