CodingForums.com - Keeping string contents in memory across requests?

CodingForums.com (http://www.codingforums.com/index.php)

- PHP (http://www.codingforums.com/forumdisplay.php?f=6)

- - Keeping string contents in memory across requests? (http://www.codingforums.com/showthread.php?t=288016)

Keeping string contents in memory across requests?

Hi everyone,

let me first tell you a bit of the background to my recent problem.

Some months ago, I wrote a series of PHP scripts, that are based around an SQLite database. One part of the scripts is used to generate new or fetch existing external data, which is then saved in the db-file. The other half of the scripts later extracts this data and removes it from the database after it has been used.

The problem stems from the new level of use the scripts have seen and the way these scripts are designed: if data is fetched from an external source, it always happens in smaller pieces (of 5-100 kilobyte) per single request. Data-input tasks are split up in most cases, which means, that a single task can generate a few dozen HTTP-requests and disk-writes or more. The read/delete part of the equation have proven to not be an issue.

For technical reasons - I'm not root on the server this runs on and have no physical access to the hardware - I can not use a different SQL database with a server process and load the whole database into the RAM, put in an SSD for better performance or maybe even create a RAM-disk and move the db-file there. Since the process of fetching external data will always be fragmented, this creates a problem. When too many scripts are both writing and reading/deleting from the SQLite database file, the CPU load spikes due to the high number of disk-writes and the hard-disk fails to keep up with the demand created by the many disk-seeks and -writes.

Now my question is: can I somehow keep a few strings (to the total amount of a few megabytes at a time) in the memory for a short while and later access them from a different script instance? At first, I thought about using sessions, but as far as I know, they will also be saved to disk immediately.

If an in-memory solution exists, I would be able to cache everything until the last fragmented database entry and write them in one go at the end. This would reduce the number of disk-writes by a few orders of magnitude and solve my problem, because the current level of activity will not rise again.

If it's not possible, a new server is needed and I will have free choice of the different solutions I already outlined earlier anyway.

Thanks for your help,
Daniel

Quote:

Originally Posted by Daniel Evans (Post 1314836)

Now my question is: can I somehow keep a few strings (to the total amount of a few megabytes at a time) in the memory for a short while and later access them from a different script instance? At first, I thought about using sessions, but as far as I know, they will also be saved to disk immediately.

Not that I know of. PHP isn't designed to work like that as a web language. No idea if you're running it as a scripting system from the command line but I would assume its the same.

The only way to keep stuff somewhere to remember it is in a DB or in a file such as the session file (which CAN be access from other scripts but you need the session id).

As you're not wanting to use the disk, neither of those are particularly great ideas... so there is one more option..

You'd need another script that cyces in a loop with a listening TCP socket. It then keeps its own variables in memory like a normal script but can accept socket connections and read/write to the connected client. It's probably not the most efficient way of doing what you want but given that your disk is already frying with activity its probably the only realistic way of achieving the desired effect. Be warned though, you can't multithread PHP scripts on windows so as far as I know, you can only handle one connection at a time. On linux you can fork so you can effectively handle more clients (as I understand it). I did once find a script that claimed to do this on windows but I didn't entirely understand how it worked after several days of disecting it.

..there is the pecl extension memcache but you wont be able to install that if you are not root.

@ firepages : I might be able to swing the addition of a php-extension with the server admin. But this won't be the route to go, since memcache/d is only the connecting piece to a separate memchached server. No go, unfortunately.

@ tangoforce : your idea sounds very intriguing. I have never done anything like that, but I will use the might of the Internet to look for some examples and convert it to my use.

Thank you :)

Quote:

Originally Posted by Daniel Evans (Post 1314961)

@ tangoforce : your idea sounds very intriguing. I have never done anything like that, but I will use the might of the Internet to look for some examples and convert it to my use.

Thank you :)

No worries, as I mentioned though, its not ideal and you may only be able to handle one connection at a time so don't get your hopes up too much..

This is the multi-client code I was talking of.. it's not multithreaded but supposedly deals with multiple connections via a loop. I have no idea quite what its doing (I could have studied it a but more but had other more pressing things) or whether its safe to use without muddling TCP connections but it is supposed to do what the author reckons (not that I can even remember whereI got this - probably PHP classes).

PHP Code:


		
			
// Configuration variables
$host = "127.0.0.1";
$port = 4041;
$max = 20;
$client = array();

// No timeouts, ignore browser being closed..
set_time_limit(0);
ignore_user_abort();

// Server functions
function rLog($msg){
             $msg = "[".date('Y-m-d H:i:s')."] ".$msg;
             print($msg."\n");

}
// Create socket
$sock = socket_create(AF_INET,SOCK_STREAM,0) or die("[".date('Y-m-d H:i:s')."] Could not create socket\n");
// Bind to socket
socket_bind($sock,$host,$port) or die("[".date('Y-m-d H:i:s')."] Could not bind to socket\n");
// Start listening
socket_listen($sock) or die("[".date('Y-m-d H:i:s')."] Could not set up socket listener\n");

rLog("Server started at ".$host.":".$port);
// Server loop
while(true)
   {
   socket_set_block($sock);
   // Setup clients listen socket for reading
   $read[0] = $sock;
   
   for($i = 0;$i<$max;$i++)
      {
      if($client[$i]['sock'] != null)
      $read[$i+1] = $client[$i]['sock'];
      }
   // Set up a blocking call to socket_select()
   $ready = socket_select($read,$write = NULL, $except = NULL, $tv_sec = NULL);
   // If a new connection is being made add it to the clients array
   if(in_array($sock,$read))
      {
      for($i = 0;$i<$max;$i++)
         {
         if($client[$i]['sock']==null)
            {
            if(($client[$i]['sock'] = socket_accept($sock))<0)
               {
               rLog("socket_accept() failed: ".socket_strerror($client[$i]['sock']));
               }
            else
               {
               rLog("Client #".$i." connected");
               }
            break;
            }
         elseif ($i == $max - 1)
            {
            rLog("Too many clients");
            }
         }
         
      if(--$ready <= 0)
      continue;
      }
      
   for($i=0;$i<$max;$i++)
      {
      if(in_array($client[$i]['sock'],$read))
         {
         $input = socket_read($client[$i]['sock'],1024);
         
         if($input==null)
            {
            unset($client[$i]);
            }
            
         $n = trim($input);
         $com = split(" ",$n);
         
         if($n=="EXIT")
            {
            if($client[$i]['sock']!=null)
               {
               // Disconnect requested
               socket_close($client[$i]['sock']);
               unset($client[$i]['sock']);
               rLog("Disconnected(2) client #".$i);
               
               for($p=0;$p<count($client);$p++)
                  {
                  socket_write($client[$p]['sock'],"DISC ".$i.chr(0));
                  }
               
               if($i == $adm)
                  {
                  $adm = -1;
                  }
               }
            }
         elseif($n=="TERM")
            {
            // Server termination requested
            socket_close($sock);
            rLog("Terminated server (requested by client #".$i.")");
            exit();
            }
         elseif($input)
            {
            // Strip whitespaces and write back to user
            // Respond to commands
            /*$output = ereg_replace("[ \t\n\r]","",$input).chr(0);
            socket_write($client[$i]['sock'],$output);*/
            if($n=="PING")
               {
               socket_write($client[$i]['sock'],"PONG".chr(0));
               }
            }
         }
      else
         {
         //if($client[$i]['sock']!=null){
         // Close the socket
         //socket_close($client[$i]['sock']);
         //unset($client[$i]);
         //rLog("Disconnected(1) client #".$i);
         //}
         }
      }
   }
// Close the master sockets
socket_close($sock);

Again, use of this is at your own risk and it may be CPU intensive (no idea how well it performs I'm afraid). I couldn't find a way to shut it down when in blocking mode since by default blocking sockets don't return from their read call until data is available meaning that it won't do anything else. In other words, you could put a stop command in your database but until there is a new connection / data sent, it will sit waiting indefinitely until it gets traffic. Not a big problem if you don't mind shutting it down via a command over a TCP connection to it but still not ideal. It may get you out of this fix though.

Additionally, you'd probably want to trigger it via an ajax call instead of via url otherwise your browser will wait for data indefinitely whereas an ajax call can disconnect after a timeout.