View Full Version : Unexpected result from addition

Bruce.Burnell

Sep 24th, 2005, 01:57 AM

I have what I thought was a simple while loop:

while (i <= 1 && returnValue == 1) {

alert("the value of i is " + parseFloat(i));

if (testNum == i) {

returnValue = 0; }

i = i + .01;

}

i was set to .01 before this loop. testNum is supposed to be a number between .01 and 1, but that is not my problem. The alert was added for debugging purposes, and I've tried it with and without the parseFloat. It displays .01 through .05 OK, but instead of .06 it gives me .060000000000000005.

I've tried this on IE, Firefox, and Netscape, on two different machines (on Win 2K and XP Professional). I can't believe it actually works this way, so it must be something I'm doing.

rm-f

Sep 24th, 2005, 03:58 AM

results are ok, as per http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf

The Number Type

The Number type has exactly 18437736874454810627 (that is, 264−253+3) values, representing the doubleprecision

64-bit format IEEE 754 values as specified in the IEEE Standard for Binary Floating-Point

Arithmetic, except that the 9007199254740990 (that is, 253−2) distinct “Not-a-Number” values of the IEEE

Standard are represented in ECMAScript as a single special NaN value. (Note that the NaN value is

produced by the program expression NaN, assuming that the globally defined variable NaN has not been

altered by program execution.) In some implementations, external code might be able to detect a difference

between various Non-a-Number values, but such behaviour is implementation-dependent; to ECMAScript

code, all NaN values are indistinguishable from each other.

There are two other special values, called positive Infinity and negative Infinity. For brevity, these values

are also referred to for expository purposes by the symbols +∞ and −∞, respectively. (Note that these two

infinite number values are produced by the program expressions +Infinity (or simply Infinity) and

-Infinity, assuming that the globally defined variable Infinity has not been altered by program

execution.)

The other 18437736874454810624 (that is, 264−253) values are called the finite numbers. Half of these are

positive numbers and half are negative numbers; for every finite positive number there is a corresponding

negative number having the same magnitude.

Note that there is both a positive zero and a negative zero. For brevity, these values are also referred to for

expository purposes by the symbols +0 and −0, respectively. (Note that these two zero number values are

produced by the program expressions +0 (or simply 0) and -0.)

The 18437736874454810622 (that is, 264−253−2) finite nonzero values are of two kinds:

18428729675200069632 (that is, 264−254) of them are normalised, having the form

s × m × 2e

where s is +1 or −1, m is a positive integer less than 253 but not less than 252, and e is an integer ranging

from −1074 to 971, inclusive.

The remaining 9007199254740990 (that is, 253−2) values are denormalised, having the form

s × m × 2e

where s is +1 or −1, m is a positive integer less than 252, and e is −1074.

Note that all the positive and negative integers whose magnitude is no greater than 253 are representable in

the Number type (indeed, the integer 0 has two representations, +0 and -0).

A finite number has an odd significand if it is nonzero and the integer m used to express it (in one of the

two forms shown above) is odd. Otherwise, it has an even significand.

In this specification, the phrase “the number value for x” where x represents an exact nonzero real

mathematical quantity (which might even be an irrational number such as π ) means a number value chosen

in the following manner. Consider the set of all finite values of the Number type, with −0 removed and

with two additional values added to it that are not representable in the Number type, namely 21024 (which is

+1 × 253 × 2971) and −21024 (which is −1 × 253 × 2971). Choose the member of this set that is closest in value

to x. If two values of the set are equally close, then the one with an even significand is chosen; for this

purpose, the two extra values 21024 and −21024 are considered to have even significands. Finally, if 21024 was

chosen, replace it with +∞; if −21024 was chosen, replace it with −∞; if +0 was chosen, replace it with −0 if

and only if x is less than zero; any other chosen value is used unchanged. The result is the number value for

x. (This procedure corresponds exactly to the behaviour of the IEEE 754 “round to nearest” mode.)

Some ECMAScript operators deal only with integers in the range −231 through 231−1, inclusive, or in the

range 0 through 232−1, inclusive. These operators accept any value of the Number type but first convert

each such value to one of 232 integer values. See the descriptions of the ToInt32 and ToUint32 operators in

sections 0 and 0, respectively.

diablo2_v

Sep 24th, 2005, 04:07 AM

I have come across this problem before also. I was using integers however, so the same solution won't work here.

I'll play around with it a bit, and see what I can come up with.

diablo2_v

Sep 24th, 2005, 04:15 AM

This worked for me:

function round(num, digits)

{

num1 = Math.pow(10,digits);

return Math.round(num*num1)/num1;

}

i=0.1;

testNum = 0.17;

returnValue = 1;

while (i <= 1 && returnValue == 1)

{

alert("the value of i is " + i, 3);

if (testNum <= i)

{

returnValue = 0;

}

i = round(i + 0.01, 3);

}

rm-f

Sep 24th, 2005, 04:31 AM

explanation:

http://www2.hursley.ibm.com/decimal/

liorean

Sep 24th, 2005, 04:42 AM

I can't believe it actually works this way, so it must be something I'm doing.Nope. It's just computers being computers. To a computer everything is binary, meaning that if you can't represent the number exactly in base 2, you have to approximate. And sometimes with approximations, you get holdovers from when a number doesn't really fit. Take 0.1 for instance. 0.1 can't be described by the computer as 1/2, nor as 1/4, nor as 1/8. Those are all larger than 0.1. But 1/16 is smaller than 0.1, so the number has to be 1/16 + something. And something is smaller than 1/16, so it'll try 1/32. Okay, 1/16 + 1/32 is not quite enough. It has to add further....

As you can see, if a number can't be represented exactly in base 2, it will have to be a series of added together exactly represented numbers. But just like 1/3 can't be represented in base 10 exactly, 1/10 can't be exactly represented in base 2. And since the computer can't store an infinite number of progressively smaller numbers, it'll have to cut it off somewhere. And the problem is when one number with a certain cutoff point is added to another number with a different cutoff point, some fraction of the number can end up above the rounding point.

Powered by vBulletin® Version 4.2.2 Copyright © 2019 vBulletin Solutions, Inc. All rights reserved.