you shoud start with unloading all modules you don't use in apache's configuration and php.ini.
don't use .htaccess files - make everything in server's configuration instead.
if you want to go even further, then you might want to switch to some lightweight server daemon like lighttpd.
and if you want REALLY high performance, write your application in C and trow it in cgi-bin
obviusly running desktop env on your server computer don't help overall performance