Skip to main content
Peter Kahrel
Community Expert
Peter KahrelCommunity Expert
Community Expert
May 12, 2012
Question

JS: toPrecision()

  • May 12, 2012
  • 1 reply
  • 2523 views

I used to think that to Number.toPrecision(d) set the number of decimal places (and the OMV help confirms that) so that, for instance,

n = 12.34567

n.toPrecision (2)

would return 123.45. But it doesn't: it returns 12. n.toPrecision(4) returns 12.34. In fact, it looks as if toPrecision now works like a kind of slice():

n = 12.34567

n.toPrecision (4)

returns 12.35: it removes the decimal point, takes the first four digits, rounds up.

Clearly there's something I miss. But what?

Thanks,

Peter

This topic has been closed for replies.

1 reply

P
Pickory
Legend
May 12, 2012

I just gooled this.

http://www.w3schools.com/jsref/jsref_toprecision.asp

Peter Kahrel
Community Expert
Peter KahrelCommunity ExpertAuthor
Community Expert
May 12, 2012

Thanks. The description in your linked web page made me look at the OMV again, and the help there appears to be completely wrong. It says

Number.toPrecision (decimals: number ): number

with an additional comment that decimals is the number of decimals. In addition, it returns a string, not a number.

Anyway, another look showed that what I was after was Number.toFixed(d), which does clip decimals the way I thought toPrecision would.

Peter

Marc Autret
Marc Autret
Legend
May 13, 2012

Hi Peter,

Be very careful when you manipulate JS rounding methods such as Number.toFixed()—especially if you treat currency values that require correct results! Since JS floating point numbers are considered "only as reliable as possible and no more," rounding methods are highly non-predictable. For example,

(1.255).toFixed(2) may return 1.25

whereas

(1.355).toFixed(2) may return 1.36

I wrote 'may' because it also depends on the OS:

(19.125).toFixed(2) returns 19.12 on Mac Lion (64bits) while the same code returns 19.13 on Win XP (32bits), all tested with ID CS5.5.

I learned the hard way that if you critically need to preserve correct rounded values, the best is to process your data as integers (which are reliable to 15 digits in JS).

@+

Marc

Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices