Actionscript can't round numbers properly

[Raekye]

Hi, I'm making an application in actionscript that deals a lot with numbers. One of the problems, which has been driving me nuts for a while now, is "almost integers". Essentially, if I do something like the square root of 3, then square it, it gives me 2.99999999999999 (this is just the easiest example to test with). I've tried forcing it to round to a certain decimal place (http://kb2.adobe.com/cps/155/tn_15542.html), using toFixed, toPrecision, and math.round. I've been spending all afternoon on this, and have found that actionscript uses 15 point/precision floating numbers. So with the squareroot/square of 3 above, if I specify it to go to 14 decimal places/significant figures, it gives me 3.0000000000001 (essentially ending with a 1), and if I specify it to 15 digits, it gives me (2.9999999999999).

I'm pretty sure that Actionscript is trolling me, because if the number was 2.9999999999999 (15 digits) and I rounded it to 15 digits... it'd make sense to keep it the same because there is no number after the last 9. However, if I round it to 14 digits, shouldn't the 14th 9 (15th digit) be chopped off, and the 13th 9 (now the last digit) rounded up, and making it carry over until it just becomes 3? Where did the 1 come from when rounding to 14 digits? If there was a 9 before that, shouldn't it have made it round to 3 when rounding to 15 digits?

Not sure if I'm making sense, but hopefuly I am, although this is driving me up the wall. Thanks in advance!

[kglad]

that's not an actionscript issue.  it's an issue of all software that uses binary computing.

within 15 decimal point accuracy 2.999999999999999 = 3.0.  if you need more than 15 decimal point accuracy, you cannot use actionscript.  if you don't need more than 15 decimal point accuracy, flash has all the tools you need for accurate computing.

[Raekye]

It doesn't matter if it's 14 or 15 decimals/digits, I just need 2.999999999999 (whatever ammount of 9s) to go to 3 or 3.0 . And at 15 decimal points (toFixed(15)), 2.99999999999999 doesn't go to 3.0, for me at least

[kglad]

use toPrecision().  if you want to round at 15 decimal places:

var yourroundedNumber:Number=yournumber.toPrecision(15);

p.s.  please mark helpful/correct responses, if there are any.

[Raekye]

I've used toPrecision(15) and it still gives me 2.9999999999999

I'll mark the helpful answers at the end, since (if I rememeber correctly) you're limited to only 2. So just the best 2

[kglad]

what do you expect?  you have 13 9's.   look:

var n:Number=2.9999999999999;  // you have 13 9's.

n.toPrecision(10)=3;

n.toPrecision(15)=2.9999999999999;  //because, to 15 decimal places 2.9999999999999 is 2.999999999999900.

// maybe the following will help you understand decimal precision:

for(var p:int=0;p<15;++){

trace(p,n.toPrecision(p));

}

[Raekye]

When I type the 2.99999's here I'm not putting them exactly (maybe i should have). When I'm testing what works, I have a text box that shows the number, a button that squares numbers and one that square roots them. So I square root 3 which gives me 2. followed by 14 nines which is placed in the text box. When I use toPrecision(15) of that number (2. followed by 14 nines) I still get 2. followed by 14 nines . toPrecision(14) of 2. followed by 14 nines gives me 3. followed by 13 zeros and 1 one.

[kglad]

you're not making sense.

here's one more try:  www.kglad.com/Files/forums/Untitled-1.fla

after this you'll need to apply a hammer to your head.

[Raekye]

Think the problem was that I applied the toPrecision after I did the calculation. So I'm assuming flash calculates to 16 digits then rounds it off?