12.95 - 12 = .949999999999999 - FREAKY!

[John Wallace]

OK - this is starting to bother me. Is 12.95 minus 12 really .949999999999999? Can anyone explain what is going on here?

Try it for yourself. Create a number field (NumField) and put 12.95 in it. Then create a calculation that subtracts 12 from the NumField. What's the result? .949999999999999? Huh?

NumFormat.zip

[The Shadow]

Floating-point math, its actually done in base-2 (binary) not base ten (decimal), so the result doesn't always make sense back in the real world.

FM7 fixes this, math is done in base-10.

An interesting fact, 1/10 is infinitely repeating in binary form, since it cannot be cleanly represented as: