# Truncate error?

This topic is 8475 days old. Please don't post here. Open a new topic instead.

## Recommended Posts

5500*0.0157 should be exactly 86.35 right?

but in FMP it gives me 86.349999999999999 where truncate(num, 2) will be 86.34, I have tried to round it to 3 dp b4 truncate, even tried to change the round result to text and then back to number for truncating, it still gives me 86.34

how can I fix this error?

-jimi

##### Share on other sites

Why don't you just use round(number,2)? Why to you need to round to 3 places and truncate? The reason FM produces this result is that the calculation is done in binary, not decimal. Pocket calculators typically use a very different algorithm to give the kind of precision they do. -bd

##### Share on other sites

because what I need is truncate.

for 1.536, I want it to be 1.53 but not round to 1.54

really no solution for this truncate error?

-jimi

##### Share on other sites

I tried rounding to 3 and truncating to 2 and it gave me the correct answer (86.35):

Truncate(Round(num1 * num2,3),2)

I did this on a mac, I'm not sure if there in an arithmetic implementation between machines that would make a difference on the PC.

-bd

[This message has been edited by LiveOak (edited March 04, 2001).]

##### Share on other sites

It's not a logarithm problem. It's as LiveOak says. When decimal numbers are converted to binary, the fractional parts of certain numbers end up being represented as repeating fractions (just like when you convert 1/3 to decimal you get .333333333... repeating). Normally you can fix this by rounding, but if you must truncate, the safest approach is to round about 10 decimal places and then truncate.

Truncate(Round(theNumber,10),3)

##### Share on other sites

Comes out to 86.35 for me.

Formatting to require 8 decimal places, it comes out to 86.35000000.

Of course, when I use my computer's calculator, I get ... 86.35. But when I subtract 86, I get ... well, I get .35. However, when I subtract .35, I get the following:

1.38236E-18

It should be noted that computers perform many arithmetic problems by using logarithms which, being [generally] irrational numbers, are inexact. Therefore, the answers are inexact -- usually this error is only noticeable if you go out to enough decimal places, but occasionally an error does crop up. So while log5500 + log0.0157 = log86.35, the computer, by truncating the logarithms, will give a slightly erroneous answer.

It's also possible that the logarithms preloaded into your computer's ROM contain an error -- I remember when such errors cropped up with the release of either the Pentium or PentiumII chip, forcing a temporary recall.

My first advice is to check the same problem on a different model computer.

HTH,

Dan

##### Share on other sites

This topic is 8475 days old. Please don't post here. Open a new topic instead.

## Create an account

Register a new account

• ### Similar Content

• #### Filemaker 6 for Mac issue.

By Tumma K,

• (and 4 more)

Tagged with:

• 8 replies
• 4,444 views
• #### FM3 upgrade to FM Pro 5.5 - Windows 8 installation possible?

By MrEddByrnes,

• (and 4 more)

Tagged with:

• 5,237 views
• #### Moving data from fp5 to fp13

By bmill,

• (and 4 more)

Tagged with:

• 2 replies
• 3,475 views
• #### Filemaker 5.5 database will not delete records on-line

By randyinla,

• (and 4 more)

Tagged with:

• 3,091 views
• #### Are there still hosting providers for filemaker 5.5?

By randyinla,

• (and 4 more)

Tagged with:

• 2 replies
• 2,726 views
×
×
• Create New...