5.52 + 13.49 - 19.01 - 0 = -3.5527136788 ??? why not 0??

Forum|Forum|17 years ago
November 26, 2008
5 replies
728 views

I have a little problem with my cf application.

I have code like this : <cfset holidayclaim = val(holidayclaim) + val(leave) - val(leavetaken) - val(truncday)>

when I output value for each variable, here is what I got :

Before calculation :
holidayclaim = 5.52
leave = 13.49
leavetaken = 19.01
truncday = 0

after calculation
holidayclaim = -3.5527136788

This is WRONG, after calculation holidayclaim result should be 0.

Then I run another test using <cfset test = 5.52 + 13.49 - 19.01 - 0>
When I output "test" value I got -3.5527136788E-015

How come this happen? This is just a simple math operation involve add and subtract only. Anyone know what function should I use to make the calculation result correct? I'm using CF 8,0,1,195765 on windows 2003 server.

Thanks

This topic has been closed for replies.

BKBK

Community Expert

if I need 5 decimal places? use round(number * 100000) / 100000 ?

No. Round() returns an integer, not a decimal. Use NumberFormat() or LSNumberFormat, together with a mask for 5 decimal places. For example,

<cfset test = 5.52 + 13.49 - 19.01 - 0.1234567>
<cfoutput>#NumberFormat(test,"__._____")#</cfoutput>

BKBK

Community Expert

Anyone know what function should I use to make the calculation result correct?

Consistent with two decimal places:
<cfset test = decimalFormat(5.52 + 13.49 - 19.01 - 0)>

T

TionoAuthor

Inspiring

quote:

Originally posted by: BKBK
Anyone know what function should I use to make the calculation result correct?

Consistent with two decimal places:
<cfset test = decimalFormat(5.52 + 13.49 - 19.01 - 0)>

if I need 5 decimal places? use round(number * 100000) / 100000 ?

N

Newsgroup_User

Inspiring

Tiono wrote:
>
> really strange, if i'm not wrong, in cf documentation mention all value in a
> math operation must be put into val function.

NO! The purpose of the val() function is to explicitly extract leading
numbers from a string. It is unneeded for basic math.

>
> also very strange, why i calculate at windows calculator, i get 0? think that
> is also computer.

Yes it is, and it can also be susceptible to rounding errors, this is a
binary thing. As a program created by programmers that understand the
limitations of decimal to binary conversion and back; they have built
logic to handle the rounding errors, most of the time.

K

Kronin555

Participating Frequently

Tiono, note that the result you're getting is:
-3.5527136788E-015
-NOT-
-3.5527136788

Very big difference. Namely:
-0.0000000000000035527136788
versus
-3.5527136788

You can't simply drop the exponent, as you're doing. You _could_ round holidayclaim to 2 decimal places, which will end up with 0.

This is simply a result of binary math on floating-point numbers, as already stated.

T

TionoAuthor

Inspiring

test=precisionEvaluate(5.52 + 13.49 - 19.01 - 0); result = 0

holidayclaim=precisionEvaluate(holidayclaim+leave-leavetaken-truncday); result = 0

holidayclaim = precisionEvaluate(val(holidayclaim) + val(leave) - val(leavetaken) - val(truncday)); result = -3.5527136788

really strange, if i'm not wrong, in cf documentation mention all value in a math operation must be put into val function.

also very strange, why i calculate at windows calculator, i get 0? think that is also computer.

Do I need to add precisionEvaluate and remove val() to all of my math operation?

N

Newsgroup_User

Inspiring

Tiono wrote:
> How come this happen? This is just a simple math operation involve add and
> subtract only. Anyone know what function should I use to make the calculation
> result correct? I'm using CF 8,0,1,195765 on windows 2003 server.

welcome to the world of computer rounding errors & precision. i suggest some
basic compsci 101 reading.

in the meantime have a look at the precisionEvaluate() method:

test=precisionEvaluate(5.52 + 13.49 - 19.01 - 0);
holidayclaim=precisionEvaluate(holidayclaim+leave-leavetaken-truncday);

also the val() method seems to return "normal" integer or floating point values
(precisionEvaluate uses BigDecimal) so maybe see if you can avoid it.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded