Many many AJAX calls, how to make it work

[jfreak53]

I have this system that I am making and the final part of it requires that I transfer content from one system to another once a day. We are talking about somewhere around 50K sql queries (Don't ask, I didn't make it I am just moving it ). Problem is I cannot do that many in PHP without a timeout issue, not to mention the client will get tired of just sitting there waiting and the apache timeout limit will be met also.

So I figured I'd do it with a progress bar and ajax. I got the system setup right to accept the calls and run through the script just fine, that's not a problem. The problem is the DOM freezes because my ajax calls are not Async. Here is my script:

function ajax ( func, params, coms ) {
	return j.ajax({
   type:"GET",
   url: "libraries/caller.php?function=tras_"+func+"&params="+params+coms,
   async: false,
   dataType: "json"
  }).responseText;
}

function ajaxP ( func, js, params, coms ) {
	return j.ajax({
   type:"POST",
   url: "libraries/caller.php?function=tras_"+func+"&params="+params+coms,
   async: false,
   data: js,
   dataType: "json",
   beforeSend: function(x) {
     if (x && x.overrideMimeType) {
       x.overrideMimeType("application/j-son;charset=UTF-8");
     }
   }
  }).responseText;
}

function startTras () {
	var startDate = new Date(j('#del').val());
	var endDate = new Date(j('#al').val());
	var del = getEl('del');
	var al = getEl('al');

	if (startDate <= endDate){
		j('#main').html('<p><div id="fech"></div><div id="mes"></div><div id="error"></div></p><div id="progressbar" style="width: 200px; height: 22px;"></div>');
		a = ajax('checkInfo', '', '&del='+del+'&al='+al);
		s = j.parseJSON(a);
		if(s.err > '') {
			j('#err').html('Error, no se puede continuar...<br>'+s.err);
			j('#progressbar').progressbar({ value: 100 });
		} else {
			var curdat = s.params.curdate;
			var num1 = s.params.num1;
			var num2 = s.params.num2;
			var num3 = s.params.num3;
			var num4 = s.params.num4;
			
			while(num1 <= num4) {
				j('#progressbar').progressbar({ value: 5 });
				j('#fech').html(curdat);
				j('#mes').html('[1/3] Generando Poliza de Cargos...');
				z = j.parseJSON(ajax('countPolCar', s.params.curdate, ''));
				cur = z.cur;
				//alert(z[1][1]);
				valor = 5;
				for(i=0;i<z.length;i++) {
					expres = j.parseJSON(ajaxP('polCargosIns', {params: JSON.stringify(s.params), emp: z[i][0] }, '', ''));
					perc = 100 / z[i][1];
					for(e=0;e<z[i][1];e++) {
						valor = valor + perc;
						j('#progressbar').progressbar({ value: valor });
						b = ajaxP('polCargos', {params: JSON.stringify(s.params), cur: e, emp: z[i][0], expression: expres.ex }, '', '');
					}
				}
				j('#mes').html('Contabilizando...');
				curdat = ajax('addDay', s.params.curdate, '');
				s.params.curdate = curdat;
				num1 = num1+1;
				num2 = num2+1;
			}
		}
	} else {
		alert('La fecha final debe ser mayor a la fecha de inicio.');
	}
}

Now you would normally say, well call async instead of sync, I know, but the problem is that if it's async I cannot assign it to a variable for one. I have to callback a function. If I do that how do I stay inside the same loop?

As you can see I have to run a date loop to run through all days selected. Plus I also have to run through a mysql query loop, hence the returned Z data, there are 625 fields that are returned, for each field returned there are two companies and for all of them there are approx. 5 queries to run. So I have a loop that has to be run through.

Any thoughts on how to do this? It works right now, it just doesn't update the DOM since it's frozen on a sync loop. Thanks for the help.

[rnd me]

collect then report:

function square(n){return n*n;}
r=[1,2,3,4,5];
res=[];

//sync solution:
alert(r.map(square));


//async solution:
function done(){
  alert(res);
}

function addSolution(){
 var base=r[res.length]; 
 res.push(square(base));
 if(res.length>=r.length){ done(); }else{ setTimeout( addSolution, 500 );}
}

addSolution();

[jfreak53]

Yeah, but you can only have X amount of max async calls in a browser at one time, I would max way out with the 625 calls that it makes to the script. Plus there's no way to know if their are 625 records or less or more.

[rnd me]

Originally Posted by jfreak53

Yeah, but you can only have X amount of max async calls in a browser at one time, I would max way out with the 625 calls that it makes to the script. Plus there's no way to know if their are 625 records or less or more.

you could only do one call at a time, you never need more than a dozen hanging out there at once...

you trigger the next call or batch of calls upon the arrival of a call.

i'm not sure what your server spits out, but i faced a related challenge doing a CMS content dump, from a mysql db. i used LIMIT to ask for 100 rows at a time, range specified in the url. if the result set came back with 100 entries, i incremented the range by 100. if it had fewer than 100 rows, that set was complete and i invoked the collector callback to save the collated results to a file.


to get started, i would break your code into more re-usable functions. get the logic out of the loop. once you get everything broken up, it will be much easier to start making it async.
in particular,

while(num1 <= num4) {

could be

function next(num1, num4) {
 if(num1>=num4){ return;}
 ...

also, i would look at the sql, you should be able to combine a lot of those queries before they are served. use joins to create a wider table of data to avoid repeated calls.


also, this line:

z = j.parseJSON(ajax('countPolCar', s.params.curdate, ''));

can be moved to before the while loop (or function), which saves an IO every loop.

[jfreak53]

Hmm, ok now that makes more sense. Thanks, let me check into that tomorrow then. Thanks for the help.

[jfreak53]

WOW, worked like a charm! Thanks for the advise, now I just gotta finish all the other queries ha ha ha