July 13, 200421 yr OK - this is starting to bother me. Is 12.95 minus 12 really .949999999999999? Can anyone explain what is going on here? Try it for yourself. Create a number field (NumField) and put 12.95 in it. Then create a calculation that subtracts 12 from the NumField. What's the result? .949999999999999? Huh? NumFormat.zip
July 13, 200421 yr Floating-point math, its actually done in base-2 (binary) not base ten (decimal), so the result doesn't always make sense back in the real world. FM7 fixes this, math is done in base-10. An interesting fact, 1/10 is infinitely repeating in binary form, since it cannot be cleanly represented as:
Create an account or sign in to comment