5.52 + 13.49 - 19.01 - 0 = -3.5527136788 ??? why not 0??

[Tiono]

I have a little problem with my cf application.

I have code like this : <cfset holidayclaim = val(holidayclaim) + val(leave) - val(leavetaken) - val(truncday)>

when I output value for each variable, here is what I got :

Before calculation :
holidayclaim = 5.52
leave = 13.49
leavetaken = 19.01
truncday = 0

after calculation
holidayclaim = -3.5527136788

This is WRONG, after calculation holidayclaim result should be 0.

Then I run another test using <cfset test = 5.52 + 13.49 - 19.01 - 0>
When I output "test" value I got -3.5527136788E-015

How come this happen? This is just a simple math operation involve add and subtract only. Anyone know what function should I use to make the calculation result correct? I'm using CF 8,0,1,195765 on windows 2003 server.

Thanks

[Newsgroup_User]

Tiono wrote:
> How come this happen? This is just a simple math operation involve add and
> subtract only. Anyone know what function should I use to make the calculation
> result correct? I'm using CF 8,0,1,195765 on windows 2003 server.

welcome to the world of computer rounding errors & precision. i suggest some
basic compsci 101 reading.

in the meantime have a look at the precisionEvaluate() method:

test=precisionEvaluate(5.52 + 13.49 - 19.01 - 0);
holidayclaim=precisionEvaluate(holidayclaim+leave-leavetaken-truncday);

also the val() method seems to return "normal" integer or floating point values
(precisionEvaluate uses BigDecimal) so maybe see if you can avoid it.

[Tiono]

test=precisionEvaluate(5.52 + 13.49 - 19.01 - 0); result = 0

holidayclaim=precisionEvaluate(holidayclaim+leave-leavetaken-truncday); result = 0

holidayclaim = precisionEvaluate(val(holidayclaim) + val(leave) - val(leavetaken) - val(truncday)); result = -3.5527136788

really strange, if i'm not wrong, in cf documentation mention all value in a math operation must be put into val function.

also very strange, why i calculate at windows calculator, i get 0? think that is also computer.

Do I need to add precisionEvaluate and remove val() to all of my math operation?

[Newsgroup_User]

Tiono wrote:
>
> really strange, if i'm not wrong, in cf documentation mention all value in a
> math operation must be put into val function.

NO! The purpose of the val() function is to explicitly extract leading
numbers from a string. It is unneeded for basic math.

>
> also very strange, why i calculate at windows calculator, i get 0? think that
> is also computer.

Yes it is, and it can also be susceptible to rounding errors, this is a
binary thing. As a program created by programmers that understand the
limitations of decimal to binary conversion and back; they have built
logic to handle the rounding errors, most of the time.

[Kronin555]

Tiono, note that the result you're getting is:
-3.5527136788E-015
-NOT-
-3.5527136788

Very big difference. Namely:
-0.0000000000000035527136788
versus
-3.5527136788

You can't simply drop the exponent, as you're doing. You _could_ round holidayclaim to 2 decimal places, which will end up with 0.

This is simply a result of binary math on floating-point numbers, as already stated.

[BKBK]

Anyone know what function should I use to make the calculation result correct?

Consistent with two decimal places:
<cfset test = decimalFormat(5.52 + 13.49 - 19.01 - 0)>

[Tiono]

quote:

---

Originally posted by: BKBK
Anyone know what function should I use to make the calculation result correct?

Consistent with two decimal places:
<cfset test = decimalFormat(5.52 + 13.49 - 19.01 - 0)>

---

if I need 5 decimal places? use round(number * 100000) / 100000 ?

[BKBK]

if I need 5 decimal places? use round(number * 100000) / 100000 ?

No. Round() returns an integer, not a decimal. Use NumberFormat() or LSNumberFormat, together with a mask for 5 decimal places. For example,

<cfset test = 5.52 + 13.49 - 19.01 - 0.1234567>
<cfoutput>#NumberFormat(test,"__._____")#</cfoutput>