Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    New to the CF scene
    Join Date
    Aug 2010
    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Keeping string contents in memory across requests?

    Hi everyone,

    let me first tell you a bit of the background to my recent problem.

    Some months ago, I wrote a series of PHP scripts, that are based around an SQLite database. One part of the scripts is used to generate new or fetch existing external data, which is then saved in the db-file. The other half of the scripts later extracts this data and removes it from the database after it has been used.

    The problem stems from the new level of use the scripts have seen and the way these scripts are designed: if data is fetched from an external source, it always happens in smaller pieces (of 5-100 kilobyte) per single request. Data-input tasks are split up in most cases, which means, that a single task can generate a few dozen HTTP-requests and disk-writes or more. The read/delete part of the equation have proven to not be an issue.

    For technical reasons - I'm not root on the server this runs on and have no physical access to the hardware - I can not use a different SQL database with a server process and load the whole database into the RAM, put in an SSD for better performance or maybe even create a RAM-disk and move the db-file there. Since the process of fetching external data will always be fragmented, this creates a problem. When too many scripts are both writing and reading/deleting from the SQLite database file, the CPU load spikes due to the high number of disk-writes and the hard-disk fails to keep up with the demand created by the many disk-seeks and -writes.

    Now my question is: can I somehow keep a few strings (to the total amount of a few megabytes at a time) in the memory for a short while and later access them from a different script instance? At first, I thought about using sessions, but as far as I know, they will also be saved to disk immediately.

    If an in-memory solution exists, I would be able to cache everything until the last fragmented database entry and write them in one go at the end. This would reduce the number of disk-writes by a few orders of magnitude and solve my problem, because the current level of activity will not rise again.

    If it's not possible, a new server is needed and I will have free choice of the different solutions I already outlined earlier anyway.

    Thanks for your help,
    Daniel

  • #2
    Senior Coder
    Join Date
    Feb 2011
    Location
    Your Monitor
    Posts
    4,301
    Thanks
    57
    Thanked 525 Times in 512 Posts
    Blog Entries
    5
    Quote Originally Posted by Daniel Evans View Post
    Now my question is: can I somehow keep a few strings (to the total amount of a few megabytes at a time) in the memory for a short while and later access them from a different script instance? At first, I thought about using sessions, but as far as I know, they will also be saved to disk immediately.
    Not that I know of. PHP isn't designed to work like that as a web language. No idea if you're running it as a scripting system from the command line but I would assume its the same.

    The only way to keep stuff somewhere to remember it is in a DB or in a file such as the session file (which CAN be access from other scripts but you need the session id).

    As you're not wanting to use the disk, neither of those are particularly great ideas... so there is one more option..

    You'd need another script that cyces in a loop with a listening TCP socket. It then keeps its own variables in memory like a normal script but can accept socket connections and read/write to the connected client. It's probably not the most efficient way of doing what you want but given that your disk is already frying with activity its probably the only realistic way of achieving the desired effect. Be warned though, you can't multithread PHP scripts on windows so as far as I know, you can only handle one connection at a time. On linux you can fork so you can effectively handle more clients (as I understand it). I did once find a script that claimed to do this on windows but I didn't entirely understand how it worked after several days of disecting it.
    See my new CodingForums Blog: http://www.codingforums.com/blogs/tangoforce/

    Many useful explanations and tips including: Cannot modify headers - already sent, The IE if (isset($_POST['submit'])) bug explained, unexpected T_CONSTANT_ENCAPSED_STRING, debugging tips and much more!

  • #3
    Super Moderator
    Join Date
    May 2002
    Location
    Perth Australia
    Posts
    4,040
    Thanks
    10
    Thanked 92 Times in 90 Posts
    ..there is the pecl extension memcache but you wont be able to install that if you are not root.
    resistance is...

    MVC is the current buzz in web application architectures. It comes from event-driven desktop application design and doesn't fit into web application design very well. But luckily nobody really knows what MVC means, so we can call our presentation layer separation mechanism MVC and move on. (Rasmus Lerdorf)

  • #4
    New to the CF scene
    Join Date
    Aug 2010
    Posts
    6
    Thanks
    1
    Thanked 0 Times in 0 Posts
    @ firepages : I might be able to swing the addition of a php-extension with the server admin. But this won't be the route to go, since memcache/d is only the connecting piece to a separate memchached server. No go, unfortunately.

    @ tangoforce : your idea sounds very intriguing. I have never done anything like that, but I will use the might of the Internet to look for some examples and convert it to my use.

    Thank you

  • #5
    Senior Coder
    Join Date
    Feb 2011
    Location
    Your Monitor
    Posts
    4,301
    Thanks
    57
    Thanked 525 Times in 512 Posts
    Blog Entries
    5
    Quote Originally Posted by Daniel Evans View Post
    @ tangoforce : your idea sounds very intriguing. I have never done anything like that, but I will use the might of the Internet to look for some examples and convert it to my use.

    Thank you
    No worries, as I mentioned though, its not ideal and you may only be able to handle one connection at a time so don't get your hopes up too much..

    This is the multi-client code I was talking of.. it's not multithreaded but supposedly deals with multiple connections via a loop. I have no idea quite what its doing (I could have studied it a but more but had other more pressing things) or whether its safe to use without muddling TCP connections but it is supposed to do what the author reckons (not that I can even remember whereI got this - probably PHP classes).

    PHP Code:
    // Configuration variables
    $host "127.0.0.1";
    $port 4041;
    $max 20;
    $client = array();

    // No timeouts, ignore browser being closed..
    set_time_limit(0);
    ignore_user_abort();

    // Server functions
    function rLog($msg){
                 
    $msg "[".date('Y-m-d H:i:s')."] ".$msg;
                 print(
    $msg."\n");

    }
    // Create socket
    $sock socket_create(AF_INET,SOCK_STREAM,0) or die("[".date('Y-m-d H:i:s')."] Could not create socket\n");
    // Bind to socket
    socket_bind($sock,$host,$port) or die("[".date('Y-m-d H:i:s')."] Could not bind to socket\n");
    // Start listening
    socket_listen($sock) or die("[".date('Y-m-d H:i:s')."] Could not set up socket listener\n");

    rLog("Server started at ".$host.":".$port);
    // Server loop
    while(true)
       {
       
    socket_set_block($sock);
       
    // Setup clients listen socket for reading
       
    $read[0] = $sock;
       
       for(
    $i 0;$i<$max;$i++)
          {
          if(
    $client[$i]['sock'] != null)
          
    $read[$i+1] = $client[$i]['sock'];
          }
       
    // Set up a blocking call to socket_select()
       
    $ready socket_select($read,$write NULL$except NULL$tv_sec NULL);
       
    // If a new connection is being made add it to the clients array
       
    if(in_array($sock,$read))
          {
          for(
    $i 0;$i<$max;$i++)
             {
             if(
    $client[$i]['sock']==null)
                {
                if((
    $client[$i]['sock'] = socket_accept($sock))<0)
                   {
                   
    rLog("socket_accept() failed: ".socket_strerror($client[$i]['sock']));
                   }
                else
                   {
                   
    rLog("Client #".$i." connected");
                   }
                break;
                }
             elseif (
    $i == $max 1)
                {
                
    rLog("Too many clients");
                }
             }
             
          if(--
    $ready <= 0)
          continue;
          }
          
       for(
    $i=0;$i<$max;$i++)
          {
          if(
    in_array($client[$i]['sock'],$read))
             {
             
    $input socket_read($client[$i]['sock'],1024);
             
             if(
    $input==null)
                {
                unset(
    $client[$i]);
                }
                
             
    $n trim($input);
             
    $com split(" ",$n);
             
             if(
    $n=="EXIT")
                {
                if(
    $client[$i]['sock']!=null)
                   {
                   
    // Disconnect requested
                   
    socket_close($client[$i]['sock']);
                   unset(
    $client[$i]['sock']);
                   
    rLog("Disconnected(2) client #".$i);
                   
                   for(
    $p=0;$p<count($client);$p++)
                      {
                      
    socket_write($client[$p]['sock'],"DISC ".$i.chr(0));
                      }
                   
                   if(
    $i == $adm)
                      {
                      
    $adm = -1;
                      }
                   }
                }
             elseif(
    $n=="TERM")
                {
                
    // Server termination requested
                
    socket_close($sock);
                
    rLog("Terminated server (requested by client #".$i.")");
                exit();
                }
             elseif(
    $input)
                {
                
    // Strip whitespaces and write back to user
                // Respond to commands
                /*$output = ereg_replace("[ \t\n\r]","",$input).chr(0);
                socket_write($client[$i]['sock'],$output);*/
                
    if($n=="PING")
                   {
                   
    socket_write($client[$i]['sock'],"PONG".chr(0));
                   }
                }
             }
          else
             {
             
    //if($client[$i]['sock']!=null){
             // Close the socket
             //socket_close($client[$i]['sock']);
             //unset($client[$i]);
             //rLog("Disconnected(1) client #".$i);
             //}
             
    }
          }
       }
    // Close the master sockets
    socket_close($sock); 
    Again, use of this is at your own risk and it may be CPU intensive (no idea how well it performs I'm afraid). I couldn't find a way to shut it down when in blocking mode since by default blocking sockets don't return from their read call until data is available meaning that it won't do anything else. In other words, you could put a stop command in your database but until there is a new connection / data sent, it will sit waiting indefinitely until it gets traffic. Not a big problem if you don't mind shutting it down via a command over a TCP connection to it but still not ideal. It may get you out of this fix though.

    Additionally, you'd probably want to trigger it via an ajax call instead of via url otherwise your browser will wait for data indefinitely whereas an ajax call can disconnect after a timeout.
    Last edited by tangoforce; 02-21-2013 at 09:53 PM.
    See my new CodingForums Blog: http://www.codingforums.com/blogs/tangoforce/

    Many useful explanations and tips including: Cannot modify headers - already sent, The IE if (isset($_POST['submit'])) bug explained, unexpected T_CONSTANT_ENCAPSED_STRING, debugging tips and much more!


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •