Enjoy an ad free experience by logging in. Not a member yet? Register.


Results 1 to 6 of 6
Thread: Unexpected result from addition

09242005, 01:57 AM #1
 Join Date
 Sep 2005
 Posts
 4
 Thanks
 0
 Thanked 0 Times in 0 Posts
Unexpected result from addition
I have what I thought was a simple while loop:
while (i <= 1 && returnValue == 1) {
alert("the value of i is " + parseFloat(i));
if (testNum == i) {
returnValue = 0; }
i = i + .01;
}
i was set to .01 before this loop. testNum is supposed to be a number between .01 and 1, but that is not my problem. The alert was added for debugging purposes, and I've tried it with and without the parseFloat. It displays .01 through .05 OK, but instead of .06 it gives me .060000000000000005.
I've tried this on IE, Firefox, and Netscape, on two different machines (on Win 2K and XP Professional). I can't believe it actually works this way, so it must be something I'm doing.
09242005, 03:58 AM
#2
 Join Date
 Aug 2005
 Location
 Toronto, ON, Canada
 Posts
 231
 Thanks
 0
 Thanked 0 Times in 0 Posts
results are ok, as per http://www.ecmainternational.org/pu...T/Ecma262.pdf
The Number Type
The Number type has exactly 18437736874454810627 (that is, 264−253+3) values, representing the doubleprecision
64bit format IEEE 754 values as specified in the IEEE Standard for Binary FloatingPoint
Arithmetic, except that the 9007199254740990 (that is, 253−2) distinct “NotaNumber” values of the IEEE
Standard are represented in ECMAScript as a single special NaN value. (Note that the NaN value is
produced by the program expression NaN, assuming that the globally defined variable NaN has not been
altered by program execution.) In some implementations, external code might be able to detect a difference
between various NonaNumber values, but such behaviour is implementationdependent; to ECMAScript
code, all NaN values are indistinguishable from each other.
There are two other special values, called positive Infinity and negative Infinity. For brevity, these values
are also referred to for expository purposes by the symbols +∞ and −∞, respectively. (Note that these two
infinite number values are produced by the program expressions +Infinity (or simply Infinity) and
Infinity, assuming that the globally defined variable Infinity has not been altered by program
execution.)
The other 18437736874454810624 (that is, 264−253) values are called the finite numbers. Half of these are
positive numbers and half are negative numbers; for every finite positive number there is a corresponding
negative number having the same magnitude.
Note that there is both a positive zero and a negative zero. For brevity, these values are also referred to for
expository purposes by the symbols +0 and −0, respectively. (Note that these two zero number values are
produced by the program expressions +0 (or simply 0) and 0.)
The 18437736874454810622 (that is, 264−253−2) finite nonzero values are of two kinds:
18428729675200069632 (that is, 264−254) of them are normalised, having the form
s × m × 2e
where s is +1 or −1, m is a positive integer less than 253 but not less than 252, and e is an integer ranging
from −1074 to 971, inclusive.
The remaining 9007199254740990 (that is, 253−2) values are denormalised, having the form
s × m × 2e
where s is +1 or −1, m is a positive integer less than 252, and e is −1074.
Note that all the positive and negative integers whose magnitude is no greater than 253 are representable in
the Number type (indeed, the integer 0 has two representations, +0 and 0).
A finite number has an odd significand if it is nonzero and the integer m used to express it (in one of the
two forms shown above) is odd. Otherwise, it has an even significand.
In this specification, the phrase “the number value for x” where x represents an exact nonzero real
mathematical quantity (which might even be an irrational number such as π ) means a number value chosen
in the following manner. Consider the set of all finite values of the Number type, with −0 removed and
with two additional values added to it that are not representable in the Number type, namely 21024 (which is
+1 × 253 × 2971) and −21024 (which is −1 × 253 × 2971). Choose the member of this set that is closest in value
to x. If two values of the set are equally close, then the one with an even significand is chosen; for this
purpose, the two extra values 21024 and −21024 are considered to have even significands. Finally, if 21024 was
chosen, replace it with +∞; if −21024 was chosen, replace it with −∞; if +0 was chosen, replace it with −0 if
and only if x is less than zero; any other chosen value is used unchanged. The result is the number value for
x. (This procedure corresponds exactly to the behaviour of the IEEE 754 “round to nearest” mode.)
Some ECMAScript operators deal only with integers in the range −231 through 231−1, inclusive, or in the
range 0 through 232−1, inclusive. These operators accept any value of the Number type but first convert
each such value to one of 232 integer values. See the descriptions of the ToInt32 and ToUint32 operators in
sections 0 and 0, respectively.
Last edited by rmf; 09242005 at 04:01 AM.
rm f /
09242005, 04:07 AM
#3
 Join Date
 Apr 2005
 Location
 Burnaby, BC, Canada
 Posts
 68
 Thanks
 0
 Thanked 0 Times in 0 Posts
I have come across this problem before also. I was using integers however, so the same solution won't work here.
I'll play around with it a bit, and see what I can come up with.
'Programming is the only artform that fights back!'
09242005, 04:15 AM
#4
 Join Date
 Apr 2005
 Location
 Burnaby, BC, Canada
 Posts
 68
 Thanks
 0
 Thanked 0 Times in 0 Posts
This worked for me:
Code:function round(num, digits) { num1 = Math.pow(10,digits); return Math.round(num*num1)/num1; } i=0.1; testNum = 0.17; returnValue = 1; while (i <= 1 && returnValue == 1) { alert("the value of i is " + i, 3); if (testNum <= i) { returnValue = 0; } i = round(i + 0.01, 3); }
'Programming is the only artform that fights back!'
09242005, 04:31 AM
#5
 Join Date
 Aug 2005
 Location
 Toronto, ON, Canada
 Posts
 231
 Thanks
 0
 Thanked 0 Times in 0 Posts
explanation:
http://www2.hursley.ibm.com/decimal/
rm f /
09242005, 04:42 AM
#6
 Join Date
 Feb 2003
 Location
 Umeå, Sweden
 Posts
 5,575
 Thanks
 0
 Thanked 83 Times in 74 Posts
Nope. It's just computers being computers. To a computer everything is binary, meaning that if you can't represent the number exactly in base 2, you have to approximate. And sometimes with approximations, you get holdovers from when a number doesn't really fit. Take 0.1 for instance. 0.1 can't be described by the computer as 1/2, nor as 1/4, nor as 1/8. Those are all larger than 0.1. But 1/16 is smaller than 0.1, so the number has to be 1/16 + something. And something is smaller than 1/16, so it'll try 1/32. Okay, 1/16 + 1/32 is not quite enough. It has to add further....Originally Posted by Bruce.Burnell
As you can see, if a number can't be represented exactly in base 2, it will have to be a series of added together exactly represented numbers. But just like 1/3 can't be represented in base 10 exactly, 1/10 can't be exactly represented in base 2. And since the computer can't store an infinite number of progressively smaller numbers, it'll have to cut it off somewhere. And the problem is when one number with a certain cutoff point is added to another number with a different cutoff point, some fraction of the number can end up above the rounding point.
liorean <[lio@wg]>
Articles: RegEx evolt wsabstract , Named Arguments
Useful Threads: JavaScript Docs & Refs, FAQ  HTML & CSS Docs, FAQ  XML Doc & Refs
Moz: JavaScript DOM Interfaces MSDN: JScript DHTML KDE: KJS KHTML Opera: Standards