rounding error

Question

Please help!  Trying to beat a work deadline.  The rounding issue can clearly be seen if I can send someone the pdf

gkaiseril · Accepted Answer

Acrobat and Excel behave in a very similar manner for this type of calculation.

Without any formatting or rounding 7 * 0.535 is displayed as 3.7450 the total of 5 iterations is 18.725.

Formatted to 2 decimal places the displayed value is 3.75 the total of 5 iterations is 18.73.

Individual result values rounded to 2 decimal places the displayed total is 3.75 the total of 5 iterations is 18.75.

When dealing with financial reports or government forms the result of each calculation should be rounded to the required number of decimal places for the report or form.

Rounding Example in PDF and Excel

Since Acrobat does not include a rounding function, a user written rounding function has been added as a document level function to the PDF. This function was adapted form a PDF rounding examples developed by D.P. Story.

Most individuals when computing this type of problem subconsciously round the individual results to 2 decimal places, but computers do not.

gkaiseril · Answer

Computers do the calculations using the binary number system, so as many are aware the decimal vlaues do not always convert to a rational value but result in an irrational value. The resulting values is stored as decimal floating point values.

The Math.round method works only on intiger values or whole numbers. One can multiply by 100 use the Math.round method and divide the result by 100 and for most values come up with the correct result but not in all cases. It is possible to set the displayed value of a field to 2 decimal places, but this does not change the actual computed value.

If one needs to round the actual value of the calculation to 2 decimal places then one needs to write a custom function to perform the rounding unless one wnats to repeat the code for rounding at every situation where the result needs to be rounded.

The following document level function will decimal round a number to the power of 10 places. For 2 decimal places or 1/100 th one would use a value of -2 for the decimal level to. A sample file of rounding a value to 2 decimal places.

Miileage	0.535000
				Error
	7.000000	3.7450	3.75	-0.0050
	7.000000	3.7450	3.75	-0.0050
	7.000000	3.7450	3.75	-0.0050
	7.000000	3.7450	3.75	-0.0050
	7.000000	3.7450	3.75	-0.0050
		18.7250	18.75	-0.0250

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded