As far as an explanation, WYSIWYG displays a form of the document equivalent to saving as an FM document. FM saves "dimensions" in units much smaller than any displayable units, so whether the values were originally entered as in, mm, or any of the other allowed units, they will be correct. You can display them in various dialog boxes (such as Resize Selected Columns) in the units of your choice. To change the displayed units, use View > Options and set the Display Units. (Sorry for referring to command and option names in English.)
When you save the document as XML, including by switching to XML View, FM converts the internal binary form of the document to XML, and writes dimensions to 3 decimal places in the selected units. There is a read/write rule that lets you specify the units for exporting the dimensions of graphics, but, unfortunately none for that of table cells. If you open an XML document in FM and save it immediately as XML without making any edits, you are correct that column widths originally specified in mm will be converted to in. You are also correct that if you convert the widths from in to mm, there may well be a slight difference from the original due to rounding to 3 decimal places. (Since 0.001in is 0.0254mm, which rounds to 3 places as 0.025mm, you may see a difference in the second decimal place.) When you look at the table, any difference will be too small for the human eye to see, but the XML files will be different.
As far as preventing the change, your XML application would need to use a custom API client to export the widths in mm. I also suggest submitting an enhancement request to allow the Specify Size read/write rule in rules for tables as well as for graphics and equations.