Truncate error?

[jimi]

5500*0.0157 should be exactly 86.35 right?

but in FMP it gives me 86.349999999999999 where truncate(num, 2) will be 86.34, I have tried to round it to 3 dp b4 truncate, even tried to change the round result to text and then back to number for truncating, it still gives me 86.34

how can I fix this error?

-jimi

[LiveOak]

Why don't you just use round(number,2)? Why to you need to round to 3 places and truncate? The reason FM produces this result is that the calculation is done in binary, not decimal. Pocket calculators typically use a very different algorithm to give the kind of precision they do. -bd

[jimi]

because what I need is truncate.

for 1.536, I want it to be 1.53 but not round to 1.54

really no solution for this truncate error?

-jimi

[LiveOak]

I tried rounding to 3 and truncating to 2 and it gave me the correct answer (86.35):

Truncate(Round(num1 * num2,3),2)

I did this on a mac, I'm not sure if there in an arithmetic implementation between machines that would make a difference on the PC.

-bd

[This message has been edited by LiveOak (edited March 04, 2001).]

[BobWeaver]

It's not a logarithm problem. It's as LiveOak says. When decimal numbers are converted to binary, the fractional parts of certain numbers end up being represented as repeating fractions (just like when you convert 1/3 to decimal you get .333333333... repeating). Normally you can fix this by rounding, but if you must truncate, the safest approach is to round about 10 decimal places and then truncate.

Truncate(Round(theNumber,10),3)

[danjacoby]

Comes out to 86.35 for me.

Formatting to require 8 decimal places, it comes out to 86.35000000.

Of course, when I use my computer's calculator, I get ... 86.35. But when I subtract 86, I get ... well, I get .35. However, when I subtract .35, I get the following:

1.38236E-18

It should be noted that computers perform many arithmetic problems by using logarithms which, being [generally] irrational numbers, are inexact. Therefore, the answers are inexact -- usually this error is only noticeable if you go out to enough decimal places, but occasionally an error does crop up. So while log5500 + log0.0157 = log86.35, the computer, by truncating the logarithms, will give a slightly erroneous answer.

It's also possible that the logarithms preloaded into your computer's ROM contain an error -- I remember when such errors cropped up with the release of either the Pentium or PentiumII chip, forcing a temporary recall.

My first advice is to check the same problem on a different model computer.

HTH,

Dan