View Full Version : Unexpected result from addition
Bruce.Burnell 09242005, 01:57 AM I have what I thought was a simple while loop:
while (i <= 1 && returnValue == 1) {
alert("the value of i is " + parseFloat(i));
if (testNum == i) {
returnValue = 0; }
i = i + .01;
}
i was set to .01 before this loop. testNum is supposed to be a number between .01 and 1, but that is not my problem. The alert was added for debugging purposes, and I've tried it with and without the parseFloat. It displays .01 through .05 OK, but instead of .06 it gives me .060000000000000005.
I've tried this on IE, Firefox, and Netscape, on two different machines (on Win 2K and XP Professional). I can't believe it actually works this way, so it must be something I'm doing.
results are ok, as per http://www.ecmainternational.org/publications/files/ECMAST/Ecma262.pdf
The Number Type
The Number type has exactly 18437736874454810627 (that is, 264−253+3) values, representing the doubleprecision
64bit format IEEE 754 values as specified in the IEEE Standard for Binary FloatingPoint
Arithmetic, except that the 9007199254740990 (that is, 253−2) distinct “NotaNumber” values of the IEEE
Standard are represented in ECMAScript as a single special NaN value. (Note that the NaN value is
produced by the program expression NaN, assuming that the globally defined variable NaN has not been
altered by program execution.) In some implementations, external code might be able to detect a difference
between various NonaNumber values, but such behaviour is implementationdependent; to ECMAScript
code, all NaN values are indistinguishable from each other.
There are two other special values, called positive Infinity and negative Infinity. For brevity, these values
are also referred to for expository purposes by the symbols +∞ and −∞, respectively. (Note that these two
infinite number values are produced by the program expressions +Infinity (or simply Infinity) and
Infinity, assuming that the globally defined variable Infinity has not been altered by program
execution.)
The other 18437736874454810624 (that is, 264−253) values are called the finite numbers. Half of these are
positive numbers and half are negative numbers; for every finite positive number there is a corresponding
negative number having the same magnitude.
Note that there is both a positive zero and a negative zero. For brevity, these values are also referred to for
expository purposes by the symbols +0 and −0, respectively. (Note that these two zero number values are
produced by the program expressions +0 (or simply 0) and 0.)
The 18437736874454810622 (that is, 264−253−2) finite nonzero values are of two kinds:
18428729675200069632 (that is, 264−254) of them are normalised, having the form
s × m × 2e
where s is +1 or −1, m is a positive integer less than 253 but not less than 252, and e is an integer ranging
from −1074 to 971, inclusive.
The remaining 9007199254740990 (that is, 253−2) values are denormalised, having the form
s × m × 2e
where s is +1 or −1, m is a positive integer less than 252, and e is −1074.
Note that all the positive and negative integers whose magnitude is no greater than 253 are representable in
the Number type (indeed, the integer 0 has two representations, +0 and 0).
A finite number has an odd significand if it is nonzero and the integer m used to express it (in one of the
two forms shown above) is odd. Otherwise, it has an even significand.
In this specification, the phrase “the number value for x” where x represents an exact nonzero real
mathematical quantity (which might even be an irrational number such as π ) means a number value chosen
in the following manner. Consider the set of all finite values of the Number type, with −0 removed and
with two additional values added to it that are not representable in the Number type, namely 21024 (which is
+1 × 253 × 2971) and −21024 (which is −1 × 253 × 2971). Choose the member of this set that is closest in value
to x. If two values of the set are equally close, then the one with an even significand is chosen; for this
purpose, the two extra values 21024 and −21024 are considered to have even significands. Finally, if 21024 was
chosen, replace it with +∞; if −21024 was chosen, replace it with −∞; if +0 was chosen, replace it with −0 if
and only if x is less than zero; any other chosen value is used unchanged. The result is the number value for
x. (This procedure corresponds exactly to the behaviour of the IEEE 754 “round to nearest” mode.)
Some ECMAScript operators deal only with integers in the range −231 through 231−1, inclusive, or in the
range 0 through 232−1, inclusive. These operators accept any value of the Number type but first convert
each such value to one of 232 integer values. See the descriptions of the ToInt32 and ToUint32 operators in
sections 0 and 0, respectively.
diablo2_v 09242005, 04:07 AM I have come across this problem before also. I was using integers however, so the same solution won't work here.
I'll play around with it a bit, and see what I can come up with.
diablo2_v 09242005, 04:15 AM This worked for me:
function round(num, digits)
{
num1 = Math.pow(10,digits);
return Math.round(num*num1)/num1;
}
i=0.1;
testNum = 0.17;
returnValue = 1;
while (i <= 1 && returnValue == 1)
{
alert("the value of i is " + i, 3);
if (testNum <= i)
{
returnValue = 0;
}
i = round(i + 0.01, 3);
}
explanation:
http://www2.hursley.ibm.com/decimal/
liorean 09242005, 04:42 AM I can't believe it actually works this way, so it must be something I'm doing.Nope. It's just computers being computers. To a computer everything is binary, meaning that if you can't represent the number exactly in base 2, you have to approximate. And sometimes with approximations, you get holdovers from when a number doesn't really fit. Take 0.1 for instance. 0.1 can't be described by the computer as 1/2, nor as 1/4, nor as 1/8. Those are all larger than 0.1. But 1/16 is smaller than 0.1, so the number has to be 1/16 + something. And something is smaller than 1/16, so it'll try 1/32. Okay, 1/16 + 1/32 is not quite enough. It has to add further....
As you can see, if a number can't be represented exactly in base 2, it will have to be a series of added together exactly represented numbers. But just like 1/3 can't be represented in base 10 exactly, 1/10 can't be exactly represented in base 2. And since the computer can't store an infinite number of progressively smaller numbers, it'll have to cut it off somewhere. And the problem is when one number with a certain cutoff point is added to another number with a different cutoff point, some fraction of the number can end up above the rounding point.

