Skip to main content
Known Participant
December 3, 2011
Question

Actionscript can't round numbers properly

  • December 3, 2011
  • 1 reply
  • 1697 views

Hi, I'm making an application in actionscript that deals a lot with numbers. One of the problems, which has been driving me nuts for a while now, is "almost integers". Essentially, if I do something like the square root of 3, then square it, it gives me 2.99999999999999 (this is just the easiest example to test with). I've tried forcing it to round to a certain decimal place (http://kb2.adobe.com/cps/155/tn_15542.html), using toFixed, toPrecision, and math.round. I've been spending all afternoon on this, and have found that actionscript uses 15 point/precision floating numbers. So with the squareroot/square of 3 above, if I specify it to go to 14 decimal places/significant figures, it gives me 3.0000000000001 (essentially ending with a 1), and if I specify it to 15 digits, it gives me (2.9999999999999).

I'm pretty sure that Actionscript is trolling me, because if the number was 2.9999999999999 (15 digits) and I rounded it to 15 digits... it'd make sense to keep it the same because there is no number after the last 9. However, if I round it to 14 digits, shouldn't the 14th 9 (15th digit) be chopped off, and the 13th 9 (now the last digit) rounded up, and making it carry over until it just becomes 3? Where did the 1 come from when rounding to 14 digits? If there was a 9 before that, shouldn't it have made it round to 3 when rounding to 15 digits?

Not sure if I'm making sense, but hopefuly I am, although this is driving me up the wall. Thanks in advance!

This topic has been closed for replies.

1 reply

kglad
Community Expert
Community Expert
December 3, 2011

that's not an actionscript issue.  it's an issue of all software that uses binary computing.

within 15 decimal point accuracy 2.999999999999999 = 3.0.  if you need more than 15 decimal point accuracy, you cannot use actionscript.  if you don't need more than 15 decimal point accuracy, flash has all the tools you need for accurate computing.

RaekyeAuthor
Known Participant
December 3, 2011

It doesn't matter if it's 14 or 15 decimals/digits, I just need 2.999999999999 (whatever ammount of 9s) to go to 3 or 3.0 . And at 15 decimal points (toFixed(15)), 2.99999999999999 doesn't go to 3.0, for me at least

kglad
Community Expert
Community Expert
December 4, 2011

use toPrecision().  if you want to round at 15 decimal places:

var yourroundedNumber:Number=yournumber.toPrecision(15);

p.s.  please mark helpful/correct responses, if there are any.