Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 2 of 2
Thread: Strange math output in Java
12-19-2008, 03:20 PM #1
Strange math output in Java
Can someone explain why the formula (0.278 * 1.0 * 10) equates to 2.7800000000000002 in java? I have a program that is comparing the result of a dynamic formula in order to validate the result, and this little mismatch is causing quite a few false negatives in my validation. These particular numbers are one of the examples that is causing a problem. Strangely enough, (0.27 * 1.0 * 10) equates to 2.7 as it should.
I've fixed the issue by ensuring all values are floats, but if anyone could enlighten me on why precision is such an issue here I would appreciate it.
Last edited by brad211987; 12-19-2008 at 03:31 PM.
12-20-2008, 12:47 AM #2
- Join Date
- Sep 2002
- Saskatoon, Saskatchewan
- Thanked 2,659 Times in 2,628 Posts
It has to do with computational mathematics. I'll have to break out old school notes (or maybe we can wait for JDK since I'm pretty sure he has a masters in mathematics), but I believe its the base conversion that does that. I remember covering this in school.
Consider 1 / 3 in base 10. There is no calculable answer since you end up with 0.333 repeating. Binary however can represent this number (though offhand I can't recall how to calculate it >.<). On the other hand, base 2 cannot handle 1 / 10, which can be handled in base 10.
I'll look up some info after and post back (if I remember) :P
header('HTTP/1.1 420 Enhance Your Calm');