Displaying Apache Server Status HTML

Display Your Apache Server Status As HTML Webpage

Diagnosing and fixing Apache issues can be a fucking nightmare (excuse the french). However one thing you might want to do is allow yourself access to view key Apache stats on a webpage rather than have to SSH in and use the console.

If you read some articles on the web to set this up it's as simple as adding the following lines to your main apache2.conf or httpd.conf file.

Depending on your server you will need to go into /etc/apache2/apache2.conf or etc/httpd/conf/httpd.conf and then adding the following lines.

# Allow server status reports, with the URL of http://dv-example.com/server-status
# Change the ".dv-example.com" to match your domain to enable.
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from mywebsite.com

Or if you wanted to only allow your own IP address to access the page you could do something like this.

# Allow server status reports, with the IP addresss you want to allow access
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from 86.42.219.12

However when I tried either of these approaches I was just met with an error message:

Forbidden

You don't have permission to access /server-status on this server.

The Fix

At the bottom of my main apache2.conf file I noticed the lines

# Include generic snippets of statements
Include /etc/apache2/conf.d/

Therefore I went into that folder and was met with 3 other files:

apache2-doc
charset
security

The top of the security file had these lines.

# Disable access to the entire file system except for the directories that
# are explicitly allowed later.
#
# This currently breaks the configurations that come with some web application
# Debian packages. It will be made the default for the release after lenny.
#
#
# AllowOverride None
# Order Deny,Allow
# Deny from all
#

So it seems that the main file was loading in these other files and the security file was blocking access to all other directories. By adding my rule to the top conf file the security file just over-ruled it again which caused the 403 error. Therefore I added the rules to the bottom of the security file and hey presto it worked.

Now I can access one of my domains to find my apache info. However I found that due to most of my sites on this virtual server using WordPress their ISAPI rules prevent a simple mysite.com/server-status from working.

WordPress is obviously trying to resolve the URL to a page, post or category list.

Therefore if you have got this problem you might need to use a domain not full of rules or create a rule that bypasses the WordPress guff.

Once you get it working you will get Apache info such as the following.

Current Time: Saturday, 31-Aug-2013 08:28:06 BST
Restart Time: Saturday, 31-Aug-2013 08:15:13 BST
Parent Server Generation: 0
Server uptime: 12 minutes 53 seconds
Total accesses: 622 - Total Traffic: 4.2 MB
CPU Usage: u26.65 s3.24 cu0 cs0 - 3.87% CPU load
.805 requests/sec - 5.5 kB/second - 6.9 kB/request
3 requests currently being processed, 4 idle workers

Plus lots of info about current HTTP requests, CPU usage and session info.

It is well worth doing if you need info about your server and you are away from your SSH console.

Handle jQuery requests when you want to reference code that hasn't loaded yet

Handle jQuery requests when you want to reference code that hasn't loaded yet

As you should be aware it is best practise to load your JavaScripts at the bottom of your HTML for performance and to stop blocking or slow load times.

However sometimes you may want to reference an object that is not yet loaded higher up in the page.

If you are using a CMS or code that you cannot change then you may not be able to add your event handlers below any scripts that maybe needed to use them. This can cause errors such as:

Uncaught ReferenceError: $ is not defined 
Uncaught ReferenceError: jQuery is not defined

If you cannot move your code below where the script is loaded then you can make use of a little PageLoader function that you can pass any functions to and which will hold them until jQuery (or any other object) is loaded before running the functions.

A simple implementation of this would involve a setTimeout call that constantly polls the check function until your script has loaded.

For example:

PageLoader = { 
 
 // holds the callback function to run once jQuery has loaded if you are loading jQuery in the footer and your code is above
 jQueryOnLoad : function(){}, 

 // call this function with your onload function as the parameter
 CheckJQuery : function(func){
  var f = false;

  // has jQuery loaded yet?
  if(window.jQuery){
   f=true;
  }
  // if not we store the function if first time in loop otherwise and set a timeout
  if(!f){
   // if we have a function store it until jQuery has loaded
   if(typeof(func)=="function"){    
    PageLoader.jQueryOnLoad = func;
   }
   // keep looping until jQuery is in the DOM
   setTimeout(PageLoader.CheckJQuery,200);
  }else{
   // jQuery has loaded so call the function
   PageLoader.jQueryOnLoad.call();    
  }
 }
}

As you can see the object just holds the function passed to it in memory until jQuery has been loaded in the DOM. This will be apparent because window.jQuery will be true.

If the object isn't in the DOM yet then it just uses a setTimeout call to poll the function until it has loaded.

You could increase the length of time between the polls or even have a maximum limit so that it doesn't poll for ever and instead after 10 loops returns an error to the console. However this is just a simple example.

You would call the function by passing your jQuery referencing function to the CheckJQuery function like so.

<script>
PageLoader.CheckJQuery(function(){
 $("#myelement").bind('click', function(e) { 
  // some code
  alert("hello");
 });
});
</script>

It's just a simple way to overcome a common problem where you cannot move your code about due to system limitations but require access to an object that will be loaded later on.

MySQL Server Has Gone Away Fix For Wordpress 3.6

MySQL Server Has Gone Away Fix For Wordpress 3.6

I have been constantly having issues with the bag of XXXX that is Wordpress and I really, really hate relying on someone else's code especially when there is so much I would do to improve it.

Adding new indexes and writing my own plugins for heavy task can only do so much to tweak the performance of a 3rd party CMS system.

However one of the problems I have been having lately is the dreaded "MySQL Server Has Gone Away" error littering my error log.

The symptoms I have been getting include:

• Trying to load a page but it just spinning away.
• Checking the console with a TOP command to see very low server loads (0.00 to 0.03)
• No Apache / MySQL processes running.

I first thought I had solved the problem some months back when I de-activated the Apache Caching plugins my server was using.

I did this because of the high number of Apache related errors in the error log files like this:

[Thu Feb 02 16:30:57 2012] [error] (103)Software caused connection abort: cache: error returned while trying to return mem cached data

These errors related to the times I was getting the slow page loads, high disk swapping and low server loads.

Apache was obviously causing me issues with it's own caching and a restart always fixed it.

As I was using WP Super Cache there was no need for duplicate caching and there are far too many levels on a LAMP set-up where caching can be enabled.

For me removing the Apache caching seemed to fix the issue, for a while at least.

However I keep getting intermittent issues where the same symptoms are present except instead of Apache errors in the error log I am getting lots of MySQL Server Has Gone Away errors like this:

[Wed Aug 14 23:37:29 2013] [error] [client 173.203.107.206] WordPress database error MySQL server has gone away for query UPDATE wp_posts SET robotsmeta = '' WHERE ID = 42693 made by WPOMatic->runCron, WPOMatic->processAll, WPOMatic->processCampaign, WPOMatic->processFeed, WPOMatic->processItem, WPOMatic->insertPost, wp_insert_post, do_action('wp_insert_post'), call_user_func_array, RobotsMeta_Admin->robotsmeta_insert_post

Therefore I looked into the MySQL Server Has Gone Away error and came across the Robs Note Book workaround for WordPress.

The fix he uses is pretty simple and involves using a custom wp-db.php file for your WordPress installation.

As this is a core file that handles all database queries it is a bit annoying in that it may need constant updates when new versions come out.

As the highest version of the workaround on his site is for WordPress 2.8.1 and I was on WordPress version 3.6 I had to create my own workaround for the file.

However the fix is pretty simple to implement and just involves using a new function with a number of retries for the failed query with a server re-connect between each loop iteration.

Basically all the fix does is something I do in my own projects many times when I encounter, lock timeouts or deadlocks e.g retry the query X number of times before quitting with an error.

You can see one such example I use to handle LOCK TIMEOUTS in MS SQL here.

Therefore if you need to apply the same fix in a future version of WordPress you can just follow these steps yourself. Download the existing copy of wp-db.php from the wp-includes folder, back it up and then make a copy before applying these changes.

1. In the db_connect() function that attempts to connect to your database ensure that the initialquery flags are set on and then off are placed around the first query run on the class initialise. This happens to currently be the $this->set_charset function which sets the connections correct character set.

The code should be wrapped around this function call and also before the $this->select function which selects the database to use.

$this->initialquery=1;

$this->set_charset( $this->dbh );

$this->ready = true;

$this->initialquery=0;

$this->select( $this->dbname, $this->dbh );

2. You need to create the following query and put in the file somewhere. This query attempts to run the query and then retries a number of times with a re-connect to the database in-between.

This is what handles the "MySQL Server Has Gone Away" error as it re-connects if the connection is no longer present.

You can change your retries to any number you want. I use 3 re-attempts as I don't see the point of any further retries as if you can't re-connect to your database after 3 goes you certainly have an issue.

function queryWithReconnect($query)
{
 // set this to the number of re-attempts you want to try
 $maxcount = 3;
 $cnt = 1;

 // loop until we reach our $maxcount value OR we get a good query result
 while($cnt < $maxcount)
 {
  // close our connection and then re-connect to our database - this is what fixes the MySQL Has Gone Away error
  // as if it has gone away we are re-attempting the connection.
  @mysql_close($this->dbh);

  // re-connect to our database
  $this->db_connect();

  // re-run our query and store the result in a global variable
  $this->result = @mysql_query($query, $this->dbh);
  
  // if we dont have an error from the server quit now, otherwise carry on!
  if (!mysql_error($this->dbh))
  {
   // return 0 so we know the query ran ok
   return 0;  
  }

  // if we are here we had an error so increment our $cnt loop counter
  $cnt+=1;
 }
 
 // we have looped up to our $maxcount number and all attempts at running the query failed so quit with a failure code!
 return 1;  
}

3. In the main query() function which runs ALL queries in WordPress you need to ensure that the call to our new function that re-attempts the query 3 times is placed just after the initial attempt at running the query.

Find the code that checks for a MySQL error and place our call to the new function inside it and above the code that clears any insert_id that may have been stored.

// If there is an error then take note of it..
if ( $this->last_error = mysql_error( $this->dbh ) ) {
 
 // If its the first initial query e.g in the db_connect() function OR we have a MySQL error then call our queryWithReconnect
 // function until it either passes OR fails after X attempts.

 if (($this->initialquery)||($this->queryWithReconnect($query)!=0)) {
  
  // Clear insert_id on a subsequent failed insert.
  if ( $this->insert_id && preg_match( '/^\s*(insert|replace)\s/i', $query ) )
   $this->insert_id = 0;

  $this->print_error();
  return false;
 }
}

Since putting this re-try code in my wp-db.php file I have had no MySQL Gone Away Errors but it is too early to tell if my main issue of low server loads, no processes running and no website activity is solved yet.

However even if it hasn't this is a good trick to use for your own WordPress code.

If you want to download the latest version of the wp-db.php workaround file which works with WordPress 3.6 then you can get it from the link below.

Just re-name it from wp-db.txt to wp-db.php and then copy it into the wp-includes folder of your site. Be sure to make a backup first in-case it all goes tits up!

Download WordPress 3.6 Fix For MySQL Server Has Gone Away Error - wp-db.php

http://www.strictly-software.com/scripts/downloads/wp-db.txt

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

To celebrate it's 10th birthday the worlds most infamous censored site, The Pirate Bay, has introduced its own Internet browser to enable people to access its website even if it's being blocked by your ISP.

Most big ISP's have blocked The Pirate Bay for its users claiming it breaches copyright by allowing people to download Torrents of films and music.

If you don't know what a torrent it it's a movie split into thousands of small pieces. Each piece is stored on various computers so that each user is not in theory holding a full version of a film that may be breaching copyright. When you download  the torrent the files are all put together and downloaded from their various locations.

However most ISP's still see this as copyright violation and even though The Pirate Bay is just like Google in the fact they are just a search engine and don't actually host the films or music they have been attacked from all quarters.

Is Your ISP Blocking You?

You can quickly test whether you are being blocked by your ISP by clicking these links which all point to the Pirate Bay Website.

http://thepiratebay.org

http://thepiratebay.sx

https://piratereverse.info

Check More Pirate Bay Proxies

If you want a PHP script that will scan a number of known Pirate Bay Proxies then you can download this one I quickly knocked up from here: Pirate Bay Proxy Checker Script.

Just change the file extension to .php and either run it from your local computer or server. If you are running Windows I recommend downloading WAMP so that you can run an Apache server on your PC.

I also recommend changing the port number the server runs on to 8080 or 8888 so it can run side by side with IIS. This article will explain how to set WAMP up.


Use Pirate Bay Tor Browser

If you are being blocked from the Pirate Bay by your ISP then you have a number of choices.

1. Find a working mirror to the site - e.g search for Pirate Bay Proxies on DuckDuckgo.com or try this page: http://torrentproxies.com/ to find some not blocked by your ISP. Many mirrors will first open up in something called adf.ly with a JavaScript countdown. After a few seconds a link will appear saying "click here to continue". On clicking the page will change (usually to an advert with a blue bar at the top) and in the top right corner will be another countdown saying "Please Wait". Usually after 5 seconds it will say "SKIP AD" and on clicking it you will end up at the Pirate Bay mirror site.

2. Use a proxy to access the Pirate Bay. There are many free proxies out there and you can find a good list at http://nntime.com/proxy-list-01.htm. Use a tool like FoxyProxy to manage your proxy lists and toggle between them. Or you can go into your browsers network / proxy settings and manually enter the IP address and port number you want to use.

3. Download the new Pirate Bay Browser or the Mozilla TOR Browser. Both of which access the TOR network to help disguise your internet footprint by bouncing your HTTP requests through a series of servers.  As the TOR website explains.

"Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location. Tor works with many of your existing applications, including web browsers, instant messaging clients, remote login, and other applications based on the TCP protocol."

With all the outrage surrounding the NSA spying on ALL American citizens and the recent revelations that they are using this information not only to catch terrorists but to catch petty criminals, drug dealers, tax avoiders and to spy on and blackmail politicians and judges it is in everyone's best interests that they tighten up on their Internet Security.

As this article I wrote explains there are a number of measures you can take to reduce your Internet footprint and whilst you might be totally invisible you can blend into the background.

So do yourself a favour and use the Pirate Bay browser or the TOR Mozilla Browser. Not only does making use of the TOR network help protect your privacy but the Pirate Bay Browser will let you access a number of Pirate Bay Mirror sites and bypass any censorship that your ISP or country may have introduced.

Download the TOR Browser

Once installed when you open the .EXE you will connect to the TOR network.



Once connected you can use the Firefox browser to surf the web. All your Pirate Bay Mirror sites and other Torrent sites are linked at the top in the icon bar. Pick one and then carry out your search for movies, applications or other content.

As you can see it's just a search engine like Google which makes it very unfair that the Pirate Bay is being blocked by ISP's as Google also indexes illegal content such as porn and copyrighted material.

This is a search for the US TV show Dexter.



Like most search engines once you run a search you get your results.



As you can see from the results on the right there are two columns, seeders and leachers.

You want to choose an item with as many seeders (uploaders) and as few leachers (downloaders) as possible to get a quick download.

You can also tell from the names of the file what sort they are e.g if the file has the word CAM in it then it's a poor quality camera in a cinema job. If it's HDTV quality it will be a bigger better quality file and BRRip is a BluRay copy.

Clicking on the file you want will open up the result page. Hopefully you will get comments telling you the quality of the file. Click on "Get This Torrent" to download your torrent file.



When you want to download a torrent from the Pirate Bay you will first need to have a torrent client. There are many out there including:

www.utorrent.com/downloads/win

bitlord.soft32.com/

www.bittorrent.com/

Once you have one installed your application of choice you can just click on the torrent in your search results and it will open up in the client and start downloading.

As you can see whilst you download the file you will also be uploading at the same time. This way you are not just taking without giving back the files you have already leached. You can control the bandwidth ratio between upload and download as well as set limits on the rate you upload.



However bewarned, ISP's are also on the lookout for people downloading (leaching) and uploading (seeding) torrents so you need to protect yourself.

Some of the ways include:

-Using someone else's WIFI or network.

-Using an anonymous proxy that doesn't leak identifying information or even better a VPN. A paid for tool like BTGuard which is both a proxy and encryption tool which helps prevent your ISP throttle your traffic.

-Using a block list which will mean that all traffic is bounced around ISP routers so that they cannot scan the traffic.

-Setting your download port to a common port for other traffic such as 80 or 8080  (HTTP) so that it doesn't look obvious you are downloading P2P data. Most people use a "random port" but it is better to look like you are downloading HTTP content especially if you are encrypting your traffic as well.

-Limiting your upload rate to a minimum. Although this violates the spirit of P2P (sharing) the people going after you for stealing copyrighted material are more interested in those spreading (uploading) the content rather than downloading it.

-Ensuring you force outgoing traffic to be encrypted. This will help prevent your ISP see that you are using BitTorrent traffic and may prevent them throttling your bandwidth.

-Set a download cap on your traffic. Even if you are encrypting your traffic some ISP's may see the amount you are downloading and throttle it if they think you are up to no good.

-Using an application like PeerBlock which will block traffic from known bad IP addresses such as P2P blocklists, known bad IP addresses, spyware, FBI and copyright monitoring sites and so on. It is worth downloading and just sits in the background running as you do you're downloading.

As with most security measures a wide range of different measures is much better than just relying on one system.

Hopefully this article will help you make use of The Pirate Bay if you have been blocked so far from accessing it.

Handling Blocking in SQL 2005 with LOCK_TIMEOUT and TRY CATCH statements

Handing Blocking and Deadlocks in SQL Stored Procedures

We recently had an issue at night during the period in which daily banner hit/view data is transferred from the daily table to the historical table.

During this time our large website was being hammered by BOTs and users and we were getting lots of timeout errors reported due the the tables we wanted to insert our hit records into being DELETED and UPDATED causing locks.

The default lock time is -1 (unlimited) but we had set it to our default command timeout of 30 seconds.

However if the DELETE or UPDATE in the data transfer job took over 30 seconds then the competing INSERT (to insert a banner hit or view) would time out and error with a database timeout due to the Blocking process not allowing our INSERT to do its job.

We tried a number of things including:

• Ensuring all tables were covered by indexes to speed up any record retrieval

• Reducing the DELETE into small batches of 1000 or 100 at a time in a WHILE loop to reduce the length of time the LOCK was held each time.

• Ensuring any unimportant SELECT statements from these tables were using WITH (NOLOCK) to get round any locking issues.

However none of these actually helped solve the problem so in the end we rewrote our stored procedure (SQL 2005 - 2008) so that it handled the LOCK TIMEOUT error and didn't return an error.

In SQL 2005 you can make use of TRY CATCH statements which meant that we could try a certain number of times to insert our data and if it failed we could just return quickly without an error as we also used a TRANSACTION to enable us to ROLLBACK or COMMIT the transaction.

We also set the LOCK_TIMEOUT to 500 milliseconds (so x 3 = 1.5 seconds) as if the insert couldn't be done in that time frame then there was no point logging it. We could have inserted it into another table to be added to our statistics later on but that is another point.

The code is below and shows you how to trap BLOCKING errors including DEADLOCKS and handle them.

Obviously this doesn't fix anything it just "masks" the problem from the end user and reduces the number of errors due to database timeouts due to long waiting blocked processes.

CREATE PROCEDURE [dbo].[usp_net_update_banner_hit]

@BannerIds varchar(200), -- CSV of banner IDs e.g 100,101,102

@HitType char(1) = 'V', -- V = banner viewed, H = banner hit

AS

SET NOCOUNT ON

SET LOCK_TIMEOUT 500 -- set to half a second


DECLARE @Tries tinyint

-- start at 1
SELECT @Tries = 1


-- loop for 3 attempts

WHILE @Tries <= 3

  BEGIN

 BEGIN TRANSACTION

 BEGIN TRY

  -- insert our banner hits we are only going to wait half a second

  INSERT INTO tbl_BANNER_DATA
  (BannerFK, HitType, Stamp)
  SELECT  [Value], @HitType, getdate()
  FROM dbo.udf_SPLIT(@BannerIds,',') -- UDF that splits a CSV into a table variable
  WHERE [Value] > 0

  --if we are here its been successful ie no deadlock or blocking going on
  COMMIT    

  -- therefore we can leave our loop
  BREAK

 END TRY

 -- otherwise we have caught an error!
 BEGIN CATCH

  --always rollback   
  ROLLBACK

  -- Now check for Blocking errors 1222 or Deadlocks 1205 and if its a deadlock wait for a while to see if that helps

  IF ERROR_NUMBER() = 1205 OR ERROR_NUMBER() = 1222

    BEGIN

   -- if its a deadlock wait 2 seconds then try again
   IF ERROR_NUMBER() = 1205
     BEGIN

    -- wait 2 seconds to see if that helps the deadlock
    WAITFOR DELAY '00:00:02'

     END   

       -- no need to wait for anything for BLOCKING ERRORS as our LOCK_TIMEOUT is going to wait for half a second anyway
       -- and if it hasn't finished by then (500ms x 3 attempts = 1.5 seconds) there is no point waiting any longer

    END      

  -- increment and try again for 3 goes
  SELECT @Tries = @Tries + 1

  -- we carry on until we reach our limit i.e 3 attempts
  CONTINUE    

   END CATCH

  END

MySQL Server won't restart

MySQL Server won't restart

Today I went to restart MySQL from my SSH console with the following command:

/etc/init.d/mysql restart

However even though the database server stopped it wouldn't restart.

I tried opening another console and running the status command.

/etc/init.d/mysql status

But this just told me it was stopped and a start command kept failing.

Even when I went into my VirtualMin website that manages my virtual server the service wouldn't restart.

I dug into the services and databases and tried accessing a database from VMIN and saw a message saying the system couldn't retrieve a list of databases. Further digging gave me this error:

can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'

Now I knew I had changed some settings in the my.cnf configuration file the other day for performance tuning. So I searched the web and found that I had added a command that wasn't supported by my older version i.e MySQL 5.0.51.

The command in question was:

skip-external-locking

Because Java had crashed and I couldn't access the file to edit it easily.

I quickly ran a:

CHMOD 777 /etc/mysql/my.cnf 

command to allow me to edit the file from FTP and then I swapped the new command with the older one which my version of MySQL supported:

skip-locking

I copied the file back and then hey presto a start command got the server back and running:

/etc/init.d/mysql status

I then made sure to CHMOD the file back so it couldn't be written by the website.

However on checking my website I only got to see the theme but NO articles.

It was a Wordpress site and the problem was probably WP-Super-Cache caching a page without data. I needed to run a REPAIR command on my wp_posts table to ensure all posts were visible again.

This is something I have seen many times before with hard reboots. The system comes back up but no articles appear. I always REPAIR and OPTIMIZE my wp_posts and wp_posts_meta table to rectify this.

This obviously locked the database up, as well as consuming 99% CPU whilst it ran - something which really annoys me, but afterwards the site was working.

So if you have been performance tuning your own MySQL database make sure you are not adding commands that your server doesn't support. A failed restart is a sure sign of unsupported configuration commands.

Apache Performance Tuning BASH Script

BASH Script to tune Apache Configuration Settings

As you might know a lot of the time I think the LAMP / Wordpress combo is a big bag of shite.

There are so many configuration options, at so many different levels that need tuning to get optimal performance, it is a nightmare to find the right information. There is also too many people offering various solutions for Wordpress / Linux / Apache / MySQL configuration.

Different people recommend different sizes for your config values and just trying to link up server load with page/URL/script requests to find out the cause of any performance issue is a nightmare in itself.

I would have thought there would have been a basic tool out there that could log server load, memory, disk swapping over time and then link that up with the MySQL slow query log, Apache error AND access logs so that you could easily tell when you had issues what processes were running, which URL's were being hit and how much activity was going on to identify culprits for tuning. I have even thought of learning PERL just to write one - not that I want to!

Even with all the MySQL tuning possible, caching plugins installed and memory limits on potentially intensive tasks it can be a nightmare to get the best out of a 1GB RAM, 40GB Virtual Server that is constantly hammered by BOTS, Crawlers and humans. I ban over 50% of my traffic and I still get performance issues at various times of the day - why? I have no FXXING idea!

Without throwing RAM at the problem you can try and set your APACHE values in the config file to appropriate values for your server and MPM fork type.

For older versions of Apache the Multi-Processing Module, non-threaded, pre-forking webserver is well suited as long as the configuration is correct. However it can consume lots of memory if not configured correctly.

For newer versions (2+) the Worker MPM is better as each thread handles a connection at a time and this is considered better for high traffic servers due to the smaller memory footprint. However to get PHP working on this setting apparently needs a lot of configuration and you should read up about this before considering a change.

Read about Apache performance tuning here Apache Performance Tuning.

To find out your current apache version from the console run

apache2 -v OR httpd -v (depending on your server type, if you run top and see apache2 threads then use apache2 otherwise use httpd)

You will get something like this.

Server version: Apache/2.2.9 (Debian) Server built: Feb 5 2012 21:40:20

To find out your current module configuration from the console run

apache2 -V OR httdp -V

Server version: Apache/2.2.9 (Debian)
Server built: Feb 5 2012 21:40:20
Server's Module Magic Number: 20051115:15
Server loaded: APR 1.2.12, APR-Util 1.2.12
Compiled using: APR 1.2.12, APR-Util 1.2.12
Architecture: 64-bit Server
MPM: Prefork threaded: no forked: yes (variable process count)
etc etc etc...

There are lots of people giving "suitable" configuration settings for the various apache settings but one thing you need to do if you run TOP and notice high memory usage and especially high virtual memory usage is try and reduce disk swapping.

I have noticed that when Apache is consuming a lot of memory that your virtual memory (disk based) will be high and you will often experience either high server loads and long wait times for pages to load OR very small server loads e.g 0.01-0.05, an unresponsive website and lots of MySQL Server Gone Away messages in your error log file.

You need to optimise your settings so that disk swapping is minimal which means trying to optimise your MySQL settings using the various MySQL tuning tools I have wrote about as well as working out the right size for your Apache configuration values.

One problem is that if you use up your memory by allowing MySQL to have enough room to cache everything it needs then you can find yourself with little left for Apache. Depending on how much memory each process consumes you can easily find that a sudden spike in concurrent hits uses up all available memory and starts disk swapping.

Therefore apart from MySQL using the disk to carry out OR caching large queries you need to find the right number of clients to allow at any one time. If you allow too many and don't have enough memory to contain them all then the server load will go up, people will wait and the amount of disk swapping will increase and increase until you enter a spiral of doom that only a restart fixes.

It is far better to allow fewer connections and serve them up quickly with a small queue and less waiting than open too many for your server to handle and create a massive queue with no hope of ending.

One of the things you should watch out for is Twitter Rushes caused by automatically tweeting your posts to twitter accounts as this can cause 30-50 BOTS to hit your site at once. If they all consume your memory up then it can cause a problem that I have wrote about before.

Working out your MaxClients value

To work out the correct number of clients to allow you need to do some maths and to help you I have created a little bash script to do this.

What it does is find out the average size of an Apache thread then restarts Apache so that the correct "free size" value can be obtained.

It then divides the remainder by the Apache process size. The value you get should be roughly the right value for your MaxClients.

It will also show you how much disk swapped or virtual memory you are using as well as the size of your MySQL process.

I noticed on my own server that when it was under-performing I was using twice as much disk space as RAM. However when I re-configured my options and gave the system enough RAM to accommodate all the SQL / APACHE processes then it worked fine with low swapping.

Therefore if your virtual memory is greater than the size of your total RAM e.g if you are using 1.5GB of hard disk space as virtual memory and only have 1GB of RAM then it will show an error message.

Also as a number of Apache tuners claim that your MinSpareServers should be 10-25% of your MaxClients value and your MaxSpareServers value 25-50% of your MaxClientsValue I have also included the calculations for these settings as well.

#!/bin/bash
echo "Calculate MaxClients by dividing biggest Apache thread by free memory"
if [ -e /etc/debian_version ]; then
 APACHE="apache2"
elif [ -e /etc/redhat-release ]; then
 APACHE="httpd"
fi
APACHEMEM=$(ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1)
APACHEMEM=$(expr $APACHEMEM / 1024)
SQLMEM=$(ps -aylC mysqld |grep "mysqld" |awk '{print $8'} |sort -n |tail -n 1)
SQLMEM=$(expr $SQLMEM / 1024)
echo "Stopping $APACHE to calculate the amount of free memory"
/etc/init.d/$APACHE stop &> /dev/null
TOTALFREEMEM=$(free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}')
TOTALMEM=$(free -m |head -n 2 |tail -n 1 |awk '{total=($2); print total}')
SWAP=$(free -m |head -n 4 |tail -n 1 |awk '{swap=($3); print swap}')
MAXCLIENTS=$(expr $TOTALFREEMEM / $APACHEMEM)
MINSPARESERVERS=$(expr $MAXCLIENTS / 4)
MAXSPARESERVERS=$(expr $MAXCLIENTS / 2)
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "Total memory $TOTALMEM"
echo "Free memory $TOTALFREEMEM"
echo "Amount of virtual memory being used $SWAP"
echo "Largest Apache Thread size $APACHEMEM"
echo "Amount of memory taking up by MySQL $SQLMEM"
if [[ SWAP > TOTALMEM ]]; then
      ERR="Virtual memory is too high"
else
      ERR="Virtual memory is ok"
fi
echo "$ERR"
echo "Total Free Memory $TOTALFREEMEM"
echo "MaxClients should be around $MAXCLIENTS"
echo "MinSpareServers should be around $MINSPARESERVERS"
echo "MaxSpareServers should be around $MAXSPARESERVERS"

If you get 0 for either of the last two values then consider increasing your memory or working out what is causing your memory issues. Either that or set your MinSpareServers to 2 and MaxSpareServers to 4.

There are many other settings which you can find appropriate values for but adding indexes to your database tables and ensuring your database table/query caches can fit in memory rather than swapped to disk is a good way to improve performance without having to resort to more caching at all the various levels Wordpress/Apache/Linux users love doing.

If you do use a caching plugin for Wordpress then I would recommend tuning it so that it doesn't cause you problems.

At first I thought WP SuperCache was a solution and pre-caching all my files would speed things up due to static HTML being served quicker than PHP.

However I found that the pre-cache stalled often, caused lots of background queries to rebuild the files which consumed memory and also took up lots of disk space.

If you are going to pre-cache everything then hold the files as long as possible as if they don't change there seems little point in deleting and rebuilding them every hour or so and using up SQL/IO etc.

I have also turned off gzip compression in the plugin and enabled it at Apache level. It seems pointless doing it twice and PHP will use more resources than the server.

The only settings I have enabled in WP-Super-Cache at the moment are:

• Don’t cache pages with GET parameters. (?x=y at the end of a url) 
• Cache rebuild.
• Serve a supercache file to anonymous users while a new file is being generated. 
• Extra homepage checks. (Very occasionally stops homepage caching)
• Only refresh current page when comments made. 
• Cache Timeout is set to 100000 seconds (why rebuild constantly?)
• Pre-Load - disabled.

Also in the Rejected User Agents box I have left it blank as I see no reason NOT to let BOTS like googlebot create cached pages for other people to use. As bots will most likely be your biggest visitor it seems odd to not let these BOTS create cached files.

So far this has given me some extra performance.

Hopefully the tuning I have done tonight will help the issue I am getting of very low server loads, MySQL gone away errors and high disk swapping. I will have to wait and see!