Benchmarking: race two functions

[ralph l mayo]

I just wrote this recently and have been using it quite a lot. Whenever I have creative block I can go back and try to refactor things with it. It's not as robust as the Apache bench tool or anything, but I still find it handy:

PHP Code:

function bench($func1, $func2, $laps = 1000, $trials = 11)
{
    $index[0] = preg_replace('/(.+)\(.*\).*/', '$1()', $func1);
    $index[1] = preg_replace('/(.+)\(.*\).*/', '$1()', $func2);
    
    if ($index[0] == $index[1])
    {
        $index[1] .= '-2';
    }
    for ($trial = 0; $trial < $trials; ++$trial)
    {
        foreach (array($func1, $func2) as $key=>$func)
        {
            $results[$trial][$index[$key]]['starttime'] = microtime(true);
            for ($lap = 0; $lap < $laps; ++$lap)
            {
                eval($func);
            }
            $results[$trial][$index[$key]]['endtime'] = microtime(true);
        }
    }
    
    // calculations are separated from the main loop to prevent interference
    
    // also, ugly code warning... most people probably don't share my sick love of multidimensional arrays
    for ($trial = 0; $trial < $trials; ++$trial)
    {
        $results[$trial][$index[0]]['elapsedtime'] = $results[$trial][$index[0]]['endtime'] - $results[$trial][$index[0]]['starttime'];
        $results[$trial][$index[1]]['elapsedtime'] = $results[$trial][$index[1]]['endtime'] - $results[$trial][$index[1]]['starttime'];
        
        $tempindex = ($results[$trial][$index[0]]['elapsedtime'] < $results[$trial][$index[1]]['elapsedtime']) ? array('winner'=>$index[0], 'loser'=>$index[1]) : array('winner'=>$index[1], 'loser'=>$index[0]);
        
        $results[$trial]['winner'] = $tempindex['winner'];
        $results[$trial]['ratio'] = $results[$trial][$tempindex['loser']]['elapsedtime'] / $results[$trial][$tempindex['winner']]['elapsedtime'];
    }
    return($results);
} 


A usage example:

PHP Code:

// don't use this stupid function, it's just here to test
function bbcodestr($text)
{
    return(str_ireplace(array('[b]','[u]','[i]', '[/i][/u][/b][u][i]', '[/i][/u][i]', '[/i]'), array('<b>', '<u>','<i>', '</b>', '</u>', '</i>'), $text));
}

// this one either
function bbcodepreg($text)
{
    return(preg_replace(array('/\[([biu])\]/i', '/\[\/([biu])\]/i'), array('<$1>', '</$1>'), $text));
}

$teststring = '[b]what [u]is[/u] [i]up[/i][/b]?';

$speedtest = bench('bbcodestr(\'' . $teststring . '\');', 'bbcodepreg(\'' . $teststring . '\');', 10000); // run 10,000 times per trial, for the default 11 trials.  Note the quotation marks and terminating semicolons are preserved in the function calls

//print_r($speedtest); // more detailed info
foreach ($speedtest as $key=>$trial)
{
    echo 'Trial ' . ($key + 1) . ' winner: ' . $trial['winner'] . ' by a factor of ' . $trial['ratio'] . '<br />';
}

/* Sample results, YMMV:

Trial 1 winner: bbcodepreg() by a factor of 1.0701883025352
Trial 2 winner: bbcodestr() by a factor of 1.1007553220986
Trial 3 winner: bbcodestr() by a factor of 1.1745991875412
Trial 4 winner: bbcodestr() by a factor of 1.1061013972411
Trial 5 winner: bbcodestr() by a factor of 1.1113493309722
Trial 6 winner: bbcodestr() by a factor of 1.0427249521624
Trial 7 winner: bbcodestr() by a factor of 1.0399104266645
Trial 8 winner: bbcodestr() by a factor of 1.0071057292584
Trial 9 winner: bbcodestr() by a factor of 1.1914867817959
Trial 10 winner: bbcodestr() by a factor of 1.0953776958654
Trial 11 winner: bbcodestr() by a factor of 1.0666147280889

The first result tends to be fishy and is usually discarded, hence the default of 11 trials.
*/ 


Try not to run this on shared hosting

[firepages]

cool, interesting use of eval.. which I thought may skew the results a little , and this can be seen in functions with marginal differences.

But playing around it still cool , I tried (among other things) a static call to a simple class method vs a call to a class instance.....

PHP Code:

<?php
class my_str_replace_class{
    function replace($s,$r,$str){
        return str_replace($s,$r,$str);
    }
}
$speedtest = bench(
    'my_str_replace_class::replace(\'a\',\'b\',\'aaaaccbabcc\');',
    '$yaks=new my_str_replace_class;$yaks->replace(\'a\',\'b\',\'aaaaccbabcc\');'
    , 1000); 
?>

OK, I knew what the answer was supposed to be

[Velox Letum]

Instead of microtime(), you might consider using getrusage(), which calculates the time actually used, rather than sitting around waiting for its turn in the CPU. It outputs the time both in system time and user time

PHP Code:

<?php
$dat = getrusage();
$utime_before = $dat["ru_utime.tv_sec"]*1e6 + $dat["ru_utime.tv_usec"];
$stime_before = $dat["ru_stime.tv_sec"]*1e6 + $dat["ru_stime.tv_usec"];

// Code here

$dat = getrusage();
$utime_after = $dat["ru_utime.tv_sec"]*1e6 + $dat["ru_utime.tv_usec"];
$stime_after = $dat["ru_stime.tv_sec"]*1e6 + $dat["ru_stime.tv_usec"];

$utime_elapsed = ($utime_after - $utime_before);
$stime_elapsed = ($stime_after - $stime_before);

// Outputs by default microseconds...convert and output
echo 'Elapsed user time: ' . ($utime_elapsed / 1e+6) . ' seconds';
echo 'Elapsed system time: '  . ($stime_elapsed / 1e+6) .  ' seconds';
?>

Edit: Forgot to equalize the times.

[ralph l mayo]

Originally Posted by firepages

cool, interesting use of eval.. which I thought may skew the results a little , and this can be seen in functions with marginal differences.

Yeah I was going to mention that eval() is slow as hell inside loops, and with most trivial functions nearly all the execution time will be eval related overhead, so this function is no good for testing how fast a function is, only for seeing how it compares to another.

I'm not sure if it'll really skew the results one way or another as both functions are subject to its overhead, but I don't know exactly how it's implemented and it's very possible that certain operations have a bigger footprint in an eval() context than in regular code.

[ralph l mayo]

Originally Posted by Velox Letum

Instead of microtime(), you might consider using getrusage(), which calculates the time actually used, rather than sitting around waiting for its turn in the CPU. It outputs the time both in system time and user time

Thanks for this, never saw that function before.