JavaScript Subtraction Value is different

[SumitKumar]

Hi,

Can anyone explain why subtraction value is different?
Expected output : 12
Result output: 11.9999999999998

baseline
Result: 732
textColumn.endBaseline
Result: 720
baseline - textColumn.endBaseline
Result: 11.9999999999998

[Peter Kahrel · Accepted solution]

A rounding error. This is typical of all computer languages. In scripting you can make allowances so that you check not whether something has a precise value, but is within a range.

[SumitKumar]

Thank you Peter for your valueable.

[Manan Joshi]

Due to this be careful when you use the greater than and less than operator in your code, you might be in for some surprises if you are not mindful of it and it's nasty to debug

-Manan

[SumitKumar]

Thank you Manan,

I used below snippet for the time being and asked to expert like you, how to solve this problem.

var difference = Math.round((baseline - textColumn.endBaseline) * 100) / 100;

[Marc Autret]

Hi @SumitKumar 

In addition to what my colleagues wrote and in order to prevent any confusion, it's worth noting that there is no rounding error with integers as such. That is, the subtraction 732-720 would obvously lead to 12 in any programming language:

var baseline = 732;
var endBaseline = 720;
var diff = baseline - endBaseline;
alert( diff ); // 12 (of course.)

Here the issue comes from the fact that either your baseline variable does not actually contain 732, or textColumn.endBaseline does not actually contain 720, despite what the JS console claims. So, the “rounding error” is that particular case arises from the fact that floating-point numbers are mistakenly shown as integers.

[SumitKumar]

Hi @Marc Autret,

Thank you for the response. Below you can see data browser and available space in page.

Under the last line exactly 12pt space is available.

Here you can see data browser shows:
baseline: 732
endBaseline: 720
diff: 11.9999999999998

[Marc Autret]

Hi again @SumitKumar ,

Yes, I can see what is displayed in InDesign and what the “data browser shows.” My point is, the shown values may be rounded without your knowledge. In your capture above, baseline is not strictly 732 and/or endBaseline is not strictly 720.

To convince you, here is another screenshot of a real example (no fake, I swear!) with additional tests of strict equality:

 

As you can see, the JS console claims that 252 is strictly different from 252 😉

In fact, my baseline variable is not strictly equal to 252—despite what is prompted.

[SumitKumar]

Thank you @Marc Autret ,

I believe on you and on your post so please do not swear. I learn lot of things from your website and also I am your fan.

https://www.indiscripts.com/

And the question is how to resolve it? Below code is okay to resolve this issue?

var difference = Math.round((baseline - textColumn.endBaseline) * 100) / 100;

Regards,