Jump to content

Truncate error?

This topic is 8475 days old. Please don't post here. Open a new topic instead.

Recommended Posts

5500*0.0157 should be exactly 86.35 right?

but in FMP it gives me 86.349999999999999 where truncate(num, 2) will be 86.34, I have tried to round it to 3 dp b4 truncate, even tried to change the round result to text and then back to number for truncating, it still gives me 86.34

how can I fix this error?


Link to comment
Share on other sites

Why don't you just use round(number,2)? Why to you need to round to 3 places and truncate? The reason FM produces this result is that the calculation is done in binary, not decimal. Pocket calculators typically use a very different algorithm to give the kind of precision they do. -bd

Link to comment
Share on other sites

I tried rounding to 3 and truncating to 2 and it gave me the correct answer (86.35):

Truncate(Round(num1 * num2,3),2)

I did this on a mac, I'm not sure if there in an arithmetic implementation between machines that would make a difference on the PC.


[This message has been edited by LiveOak (edited March 04, 2001).]

Link to comment
Share on other sites

It's not a logarithm problem. It's as LiveOak says. When decimal numbers are converted to binary, the fractional parts of certain numbers end up being represented as repeating fractions (just like when you convert 1/3 to decimal you get .333333333... repeating). Normally you can fix this by rounding, but if you must truncate, the safest approach is to round about 10 decimal places and then truncate.


Link to comment
Share on other sites

Comes out to 86.35 for me.

Formatting to require 8 decimal places, it comes out to 86.35000000.

Of course, when I use my computer's calculator, I get ... 86.35. But when I subtract 86, I get ... well, I get .35. However, when I subtract .35, I get the following:


It should be noted that computers perform many arithmetic problems by using logarithms which, being [generally] irrational numbers, are inexact. Therefore, the answers are inexact -- usually this error is only noticeable if you go out to enough decimal places, but occasionally an error does crop up. So while log5500 + log0.0157 = log86.35, the computer, by truncating the logarithms, will give a slightly erroneous answer.

It's also possible that the logarithms preloaded into your computer's ROM contain an error -- I remember when such errors cropped up with the release of either the Pentium or PentiumII chip, forcing a temporary recall.

My first advice is to check the same problem on a different model computer.



Link to comment
Share on other sites

This topic is 8475 days old. Please don't post here. Open a new topic instead.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.