Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 2 of 2
  1. #1
    Regular Coder brad211987's Avatar
    Join Date
    Sep 2005
    Location
    Ohio
    Posts
    631
    Thanks
    10
    Thanked 50 Times in 50 Posts

    Strange math output in Java

    Can someone explain why the formula (0.278 * 1.0 * 10) equates to 2.7800000000000002 in java? I have a program that is comparing the result of a dynamic formula in order to validate the result, and this little mismatch is causing quite a few false negatives in my validation. These particular numbers are one of the examples that is causing a problem. Strangely enough, (0.27 * 1.0 * 10) equates to 2.7 as it should.


    *Edit*
    I've fixed the issue by ensuring all values are floats, but if anyone could enlighten me on why precision is such an issue here I would appreciate it.
    Last edited by brad211987; 12-19-2008 at 03:31 PM.

  • #2
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,978
    Thanks
    4
    Thanked 2,659 Times in 2,628 Posts
    It has to do with computational mathematics. I'll have to break out old school notes (or maybe we can wait for JDK since I'm pretty sure he has a masters in mathematics), but I believe its the base conversion that does that. I remember covering this in school.
    Consider 1 / 3 in base 10. There is no calculable answer since you end up with 0.333 repeating. Binary however can represent this number (though offhand I can't recall how to calculate it >.<). On the other hand, base 2 cannot handle 1 / 10, which can be handled in base 10.

    I'll look up some info after and post back (if I remember) :P
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •