Jump to content
Sign in to follow this  
thelebk

Rounding Error?

Recommended Posts

I have been tasked to support a database to track expenses in FM 7.0 that works kind of like an excel spreadsheet. I.e. there is a column of enterable number fields (say e1,e2,e3,e4 etc) and at the bottum is a calc field (Sub) that is defined as e1+e2+e3+e4. The fields are all set to display 2 decimal places, use currency, thousands separator.

What is baffling me is that FM cannot seem to add properly on some records. Example e1=0, e2=0, e3=$63.75, e4=$63.75, Sub=$128. Where the heck is the extra $.50 coming from? 63.75+63.75= 127.5 not 128. If it is rounding the .75 up to .8 the total should be 127.6, but it not. Perhaps it is rounding the calculated result of 127.5 to 128. If so how do I turn rounding off? In financial calculations even a single penny is significant... I erased the data and re-entered it, but get the same result.

Does anyone know what is happening and how to fix it?

Share this post


Link to post
Share on other sites

From Rounding?

.75 rounds up to the next whole number.

In stead of Round (YourField; 0)

Try Round (YourField; 2)

Share this post


Link to post
Share on other sites

it did not round the .75 to .8 or the calc result would have been 127.6. The calc result is actually 128. Thus it appears to not like to display 127.5 even though two decimal places are specified, and my guess is that it is rounding to 128 to avoid it.

Please note I am not using the round function. I do not want rounding. I want precise numbers, even one penny off will hose my calculations. :)

The calc is sub=e1+e2+e3+e4. For some reason FM cannot perform this simple addition task accurately. I am guessing, and it is only a guess, that this may be a rounding problem. Another example:

when I try to add $41.39 + $41.39, FM returns the calculated total of $82.77. This is clearly wrong as these figures add to $82.78. Any thoughts on how to fix this?

Share this post


Link to post
Share on other sites

It would be easier to see what you are doing Right or Wrong if you would attach a copy of your file, or make a demo of this.

Lee

Share this post


Link to post
Share on other sites

A sample file demonstrating these errors would be useful.

Also, are you patched up to v.7.0v3?

Share this post


Link to post
Share on other sites

Check what's actually been entered into the fields.

Number fields can be set to display numbers with fixed decimal places and FMP displays these numbers rounded, but continues to use the full precision for calculations.

Share this post


Link to post
Share on other sites

I figured out what was happening. FM Was doing the subtotal calculation in floating point keeping half pennies etc. When the figures in the column displayed in two decimal places however FM rounded them causing the descrepancy.

Thanks.

Share this post


Link to post
Share on other sites

That makes sense, except your example was 4 items that resulted in a .50 difference. That's not a half penny x 4 difference.

Share this post


Link to post
Share on other sites

That is why I suggested a copy, or a demo of what the member is trying to do.

Saves a lot of guess work.

Lee

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

By using this site, you agree to our Terms of Use.