JS: toPrecision()

[Peter Kahrel]

I used to think that to Number.toPrecision(d) set the number of decimal places (and the OMV help confirms that) so that, for instance,

n = 12.34567

n.toPrecision (2)

would return 123.45. But it doesn't: it returns 12. n.toPrecision(4) returns 12.34. In fact, it looks as if toPrecision now works like a kind of slice():

n = 12.34567

n.toPrecision (4)

returns 12.35: it removes the decimal point, takes the first four digits, rounds up.

Clearly there's something I miss. But what?

Thanks,

Peter

[Pickory]

I just gooled this.

http://www.w3schools.com/jsref/jsref_toprecision.asp

[Peter Kahrel]

Thanks. The description in your linked web page made me look at the OMV again, and the help there appears to be completely wrong. It says

Number.toPrecision (decimals: number 😞 number

with an additional comment that decimals is the number of decimals. In addition, it returns a string, not a number.

Anyway, another look showed that what I was after was Number.toFixed(d), which does clip decimals the way I thought toPrecision would.

Peter

[Marc Autret]

Hi Peter,

Be very careful when you manipulate JS rounding methods such as Number.toFixed()—especially if you treat currency values that require correct results! Since JS floating point numbers are considered "only as reliable as possible and no more," rounding methods are highly non-predictable. For example,

(1.255).toFixed(2) may return 1.25

whereas

(1.355).toFixed(2) may return 1.36

I wrote 'may' because it also depends on the OS:

(19.125).toFixed(2) returns 19.12 on Mac Lion (64bits) while the same code returns 19.13 on Win XP (32bits), all tested with ID CS5.5.

I learned the hard way that if you critically need to preserve correct rounded values, the best is to process your data as integers (which are reliable to 15 digits in JS).

@+

Marc

[Peter Kahrel]

Thanks, Marc, for pointing out those inconsistencies.

> the best is to process your data as integers (which are reliable to 15 digits in JS)

You did mean to write 'reals', did you not

Peter

[Marc Autret]

No, I really meant integers. E.g.: store 123.45 as 12345 and keep somewhere else the E-2.

@+

Marc

[Peter Kahrel]

Aha, in that way. Thanks.

P.

[John Hawkinson]

Pickory wrote:

I just gooled this.

http://www.w3schools.com/jsref/jsref_toprecision.asp

w3schools is a very problematic reference. Read http://w3fools.com/ for some info on why.

Instead, Mozilla has a much better reference: https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Number/toPrecision