...

View Full Version : Script not completing



timgolding
01-08-2010, 12:51 PM
Hi,

I have a php cron script that is converting a CSV to a mysql database table. Though i am echoing some data from the script when i run it nothing is being displayed. The script runs and completed most of the inserts but it doesnt finish all the way. I get a empty page returned when it is done. I have set time limit to take as long as it needs with



set_time_limit(0);


The script doesn't display any errors it just returns a empty. What could cause the script to halt half way and display no errors even when no time limit is set?

angst
01-08-2010, 03:28 PM
could you post your script?

timgolding
01-08-2010, 05:32 PM
<?php
// convert all data feeds to MySQL databases for improved efficiency.
set_time_limit(0);

if($_SERVER['REMOTE_ADDR']!="192.168.84.21")
exit;

require("../../config/config2.php");
require_once("/home/shared_php_lib_v2/database_objects/mysql_connect_class.php");
require_once("/home/shared_php_lib_v2/database_objects/query_class.php");
require_once("/home/shared_php_lib_v2/form_objects/form_class.php");
require_once("/home/shared_php_lib_v2/admin_objects/admin_class.php");

$admin = new admin($webpage);

// DEALS SALT
$query = "DROP TABLE IF EXISTS `DEALS_SALT`";
if(!$result = $admin->do_task($query, "convert_2db.php: Drop table DEALS_SALT"))
{
echo "drop table error<br />";
}
else
echo "DEALS_SALT table dropped<br />";

$query = "CREATE TABLE `DEALS_SALT` (
`KEY` INT( 10 ) UNSIGNED ZEROFILL NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`SKU` varchar(255) NOT NULL default '',
`HSNUM` int(10) NOT NULL default '0',
`TARIFFNUM` int(10) NOT NULL default '0',
`PRICE` float NOT NULL default '0',
`FLR` tinyint(2) NOT NULL default '0',
`HPLR` tinyint(2) NOT NULL default '0',
`CASH_BACK` float NOT NULL default '0',
`GIFT` varchar(255) NOT NULL default '',
`RET` varchar(255) NOT NULL default '',
`CASH_TYPE` tinyint(1) NOT NULL default '0',
`CLEARANCE` tinyint(1) NOT NULL default '0',
`STOCK` tinyint(1) NOT NULL default '0',
`TOTAL_COST` float NOT NULL default '0',
`EMLR` float NOT NULL default '0',
`SKUPOPMONTH` int(10) NOT NULL default '0',
`DEALSRES1` varchar(255) NOT NULL default '',
`DEALSRES2` varchar(255) NOT NULL default '',
KEY `HSNUM` (`HSNUM`)
) ENGINE=MyISAM;
";
if(!$result = $admin->do_task($query, "convert_2db.php: Create table DEALS_SALT"))
{
echo "create table error<br />";
}
else
echo "DEALS_SALT table created<br />";

$flag = true;
$i=1;
$fname="part.".$i.".txt";

while(file_exists("../../data/dealsalt/".$fname))
{
$data = get_feed("dealsalt/".$fname);
//
foreach($data as $dat)
{
//INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9);
$sql = "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`)
VALUES('".mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')";
if(!$result = $admin->do_task($sql, "convert_2db.php: Insert into table DEALS_SALT"))
{
echo "error ". $sql . "<br />";
$flag = false;
}

}
$i++;
$fname="part.".$i.".txt";
}
if($flag)
echo "DEALS_SALT data inserted<br />";


$admin->db->db_close();

function get_feed($file)
{
$data = array();
$data_lines = file("../../data/".$file);
foreach($data_lines as $dat)
{
array_push($data, explode("\t", $dat));
}
return $data;
}
?>

JAY6390
01-08-2010, 06:04 PM
add error_reporting(E_ALL); to the top of the script in case it's got error messages turned off

ninnypants
01-08-2010, 07:02 PM
Your problem is this line

set_time_limit(0;)
http://us.php.net/manual/en/function.set-time-limit.php

you are giving your script 0 seconds to run so it is probably thowing a fatal error as soon as it reaches 1 second of execution time

Fou-Lu
01-08-2010, 08:04 PM
Your problem is this line

set_time_limit(0;)
http://us.php.net/manual/en/function.set-time-limit.php

you are giving your script 0 seconds to run so it is probably thowing a fatal error as soon as it reaches 1 second of execution time

No, 0 indicates no time limit imposed.
Try putting more weight against you're sql instead of you're php code. Develop you're insertions to take multiple inserts on a single execution:


$insert = array();
foreach($data as $dat)
{
//INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9);
$insert[] = "('".mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')";
}
$sql = "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`)
VALUES " . implode(', ', $insert);
if(!$result = $admin->do_task($sql, "convert_2db.php: Insert into table DEALS_SALT"))
{
echo "error ". $sql . "<br />";
$flag = false;
}


As long as you don't run out of memory, it will create 1 insertion call for each file, instead of 1 insertion call for each line in each file.

ninnypants
01-08-2010, 08:09 PM
No, 0 indicates no time limit imposed.

Should have looked closer when I passed the parameters section thank you for correcting me

timgolding
01-11-2010, 10:54 AM
No, 0 indicates no time limit imposed.
Try putting more weight against you're sql instead of you're php code. Develop you're insertions to take multiple inserts on a single execution:


$insert = array();
foreach($data as $dat)
{
//INSERT INTO tbl_name (a,b,c) VALUES(1,2,3,4,5,6,7,8,9);
$insert[] = "('".mysql_real_escape_string($dat[0])."', '".mysql_real_escape_string($dat[1])."','".mysql_real_escape_string($dat[2])."','".mysql_real_escape_string($dat[3])."','".mysql_real_escape_string($dat[4])."','".mysql_real_escape_string($dat[5])."', '".mysql_real_escape_string($dat[6])."','".mysql_real_escape_string($dat[7])."','".mysql_real_escape_string($dat[8])."','".mysql_real_escape_string($dat[9])."','".mysql_real_escape_string($dat[10])."', '".mysql_real_escape_string($dat[11])."','".mysql_real_escape_string($dat[12])."','".mysql_real_escape_string($dat[13])."','".mysql_real_escape_string($dat[14])."','".mysql_real_escape_string($dat[15])."','".mysql_real_escape_string($dat[16])."')";
}
$sql = "INSERT INTO `DEALS_SALT` (`SKU`,`HSNUM`,`TARIFFNUM`,`PRICE`,`FLR`,`HPLR`,`CASH_BACK`,`GIFT`,`RET`,`CASH_TYPE`,`CLEARANCE`, `STOCK`, `TOTAL_COST`, `EMLR`, `SKUPOPMONTH`, `DEALSRES1`, `DEALSRES2`)
VALUES " . implode(', ', $insert);
if(!$result = $admin->do_task($sql, "convert_2db.php: Insert into table DEALS_SALT"))
{
echo "error ". $sql . "<br />";
$flag = false;
}


As long as you don't run out of memory, it will create 1 insertion call for each file, instead of 1 insertion call for each line in each file.

Thanks i tried your suggested code and echoed the mysql_error but i got this errors



DEALS_SALT table dropped
DEALS_SALT table created
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes
error Got a packet bigger than 'max_allowed_packet' bytes


Each file is 3.7MB unfortunately there is a lot of data to insert. Does anyone know the max allowed size of the packets?

timgolding
01-11-2010, 11:45 AM
Ok i changed the max_allowed_packet to 8M and the script now runs untill it gets to the very bottom of part.8.txt . I looked at the data folder and there are part.9.txt , part.10.txt and part.11.txt still to process but the script seems to stop before processing these files. Any ideas why this might be?

timgolding
01-11-2010, 11:56 AM
I checked the file permissions on those files and found that part.8.txt permissions where set to 000 somehow. For some reason this was affecting the transition from part.8.txt to part.9.txt very strange behaviour. Anyway i finally got the script to finish running and display the results expected i will monitor it next few days and make sure it is completing.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum