Colin, that is indeed possible.
<lecture>
The problem with "floating point numbers" is, surprising as it sounds, they are not as accurate as most people think. One key problem is that the space in which they are stored is limited: 2 bytes for the most basic version, mostly 4 but even as much as 8 bytes in most current software. This means that the range of values is limited. Simplified, that means you can store 2,000 numbers ranging from -1 to +1 with an accuracy of 0.001 (no real numbers but that is the general idea), or, by using more bytes, you can store 2 million numbers with an accuracy of 0.000001 – but you cannot store "0.000001 / 2" without using another extra byte.
Also, no number of bytes in the universe is large enough to store a value such as "0.1" exactly – it will be stored as either 0.9999999 or 1.00000001 (where the number of '9's and '0's varies with the total storage size). It does not matter how many bytes you use; that inaccuracy will always be there, at the final digit!
That is due to how a decimal fraction is stored. Just like you can write one divided by three in a concisive way as "1/3" but there is no way to write it down 'exactly' in decimals – 0.33333333333.... (fill the rest of universe with 3s and it's still not 'exact') –, decimal fractions do not play nice with binary numbers. A floating point number fraction is stored as 1 divided by a power of 2, so 1/2, 1/4, 1/8, 1/16, and so on. "Multiples" of these are exact; three-eights, for example, is 1/4 + 1/8. But '0.1' cannot be formed with these multiples, and, as when expressing 1/3 in decimals, expressing 1/10 in a binary fraction leads to a repeating series "000110011(0011)+".
So "0.1" cannot be stored exactly, but if you show it on screen with, say, 5 decimals of precision, it will still display as "0.10000" – what's the problem? Well, if you add up 0.1 to itself a few times, you suddenly see that final digit jump from '0' to something else! It's that pesky final-digit that suddenly adds up, just like when you add "0.33333 + 0.33333 + 0.33333", you do not get "1.00000".
</lecture>
However, I would regard it as "probably not the case here".
The internal calculations of InDesign are magnitudes more accurate than Andy_Baz' 13 decimal digits, where it is already off by the 6th digit. This should be at least 1,000 times smaller (off by the 9th digit); the full range is something like 18 significant decimal digits.
A possible other cause could be that "the" size of an A4, as stored in InDesign, changed by just a tiny bit over several versions. I just created an A4 sized document and requested its size in millimeters with JavaScript (which shows much more decimals than InDesign itself):
alert (app.activeDocument.documentPreferences.pageWidth + ' mm\n' + app.activeDocument.documentPreferences.pageHeight+' mm');
and instead of the expected "210 mm, 297 mm" I got this:
209.999999999936 mm
296.999999999461 mm
Yes, there is a difference (more than nine decimals in), but it's nowhere as large as Andy_Baz finds: the width is off by 99.999999999969524%, the height by 99.999999999818519%. That is nearly spot-on in the range I'd expect for 'normal' inaccuracies because of recalculating millimeters into the internally-used points. Not surprisingly, requesting the A4 size in points and recalculating them back to millimeters returns the exact same 12 decimal fraction digits.
6
Replies
AdChoices