PDA

View Full Version : 69/100=0.689999, not 0.69 ???


peterinwa
03-05-2003, 10:47 PM
I have some very complex coding (for me!) that rounds numbers of less than one to two decimals, e.g., 0.68720 becomes 0.69.

To do this I multiply the number by 100 then round it, then divide it by 100. It words great, except that with WebTV with some (but not all numbers) when you divide by 100 it doesn't just move the decimal point but instead gives you a fraction.

I can by-pass all my code with a simple test: alert(69/100);

With IE I get 0.69, but with WebTV I get 0.689999...

I know this sounds like a browser problem and it probably is, but I'm such a self-taught JS novice that I wanted to post here to be sure I'm not doing something wrong. Or that I'm not wrong in expecting the result to be 0.69.

chrismiceli
03-05-2003, 11:04 PM
the numbering system in javascript isn't that great, a while ago there was a thread about like 1 divided by a number number was completely wrong, but javascript will do mostly what you want with numbers. I do probably think that it is webtv's browser that is making that error though.

joh6nn
03-05-2003, 11:36 PM
almost definitely a problem with webtv

peterinwa
03-05-2003, 11:46 PM
Thanks, I didn't want to look like a fool telling them THEY had a problem if it was me. Not that I'll be able to get anybody there to listen!

jalarie
03-06-2003, 03:17 PM
It's not a problem with WebTV, it's the difference between how humans and computers work with fractions. We count base 10, so every decimal "comes out even" so to speak. Computers count base 2, and their "decimals" come out nicely for them, but most base 10 fractions do not work out nicely in base 2; they are infinitely long!

IE carries enough trailing bits to be able to round it back to what you expect when it displays, but the "funny" number is still hiding in the background waiting to catch you. WebTV is carrying fewer trailing bits, so you get to see the problem more often.

peterinwa
03-06-2003, 03:49 PM
Great info.

I guess I'll just check and if there are too many digits I'll just trunkate after two. For my purposes that will be plenty accurate enough.

Hmm. Actually it's easier than that. I'll just display two digits and won't need to check if there are more or not.