Stop BOTS and Scrapers from bringing your site down

Blocking Traffic using WebMin on LINUX at the Firewall

If you have read my survival guides on Wordpress you will see that you have to do a lot of work just to get a stable and fast site due to all the code that is included.

The Wordpress Survival Guide

1. Wordpress Basics - Tools of the trade, useful commands, handling emergencies, banning bad traffic.
2. Wordpress Performance - Caching, plugins, bottlenecks, Indexing, turning off features
3. Wordpress Security - plugins, htaccess rules, denyhosts.

For instance not only do you have to handle badly written plugins that could contain security holes and slow the performance of your site but the general WordPress codebase is in my opinion a very badly written piece of code.

However they are slowly learning and I remember once (and only a few versions back) that on the home page there were over 200+ queries being run most of them were returning single rows.

For example if you used a plugin like Debug Queries you would see lots of SELECT statements on your homepage that returned a single row for each article shown for every post as well as the META data, categories and tags associated with the post.

So instead of one query that returned the whole data set for the page in one query (post data, category, tag and meta data) it would be filled with lots of single queries like this.

SELECT wp_posts.* FROM wp_posts WHERE ID IN (36800)

However they have improved their code and a recent check of one of my sites showed that although they are still using seperate queries for post, category/tag and meta data they are at least getting all of the records in one go e.g

SELECT wp_posts.* FROM wp_posts WHERE ID IN (36800,36799,36798,36797,36796)

So the total number of queries has dropped which aids performance. However in my opinion they could write one query for the whole page that returned all the data they needed and hopefully in a future edition they will.

However one of the things that will kill a site like Wordpress is the amount of BOTS that hit you all day long. These could be good BOTS like GoogleBOT and BingBOT which crawl your site to find out where it should appear in their own search engine or they could be social media BOTS that look for any link Twitter shows or scrapers trying to steal your data.

Some things you can try to stop legitimate BOTS like Google and BING from hammering your site is to set up a Webmaster Tools account in Google and then change the Crawl Rate to a much slower one.

You can also do the same with BING and their webmaster tools account. However with BING they apparently respect the ROBOTS.txt command DELAY e.g

Crawl-delay: 3

Which supposedly tells BOTS that respect the Robots.TXT commands that they should wait 3 seconds before each crawl. However as far as I know only BING support this at the moment and it would be nice if more SERP BOTS did in future.

If you want a basic C# Robots.txt parser that will tell you whether your agent can crawl a page on a site, extract any sitemap command then check out > http://www.strictly-software.com/robotstxt however if you wanted to extend it to add in the Crawl-Delay command it wouldn't be hard ( line 175 in Robot.cs ) to add in so that you could extract and respect it when crawling yourself.

Obviously you want all the SERP BOTS like GoogleBot and Bingbot to search you but there are so many Social Media BOTS and Spammers out there nowadays that they can literally hammer your site into the ground no matter how many caching plugins and .htacess rules you put in to return 403 codes.

The best way to deal with traffic you don't want to hit your site is as high up the chain as possible. 

Just leaving Wordpress to deal with it means the overhead of PHP code running, include files being loaded, regular expression to test for harmful parameters being run and so on.

Moving it up to the .htaccess level is better but it still means your webserver is having to process all the .htacess rules in your file to decide whether or not to let the traffic through or not.

Therefore if you can move the worst offenders up to your Firewall then it will save any code below that level from running and the TCP traffic is stopped before any regular expressions have to be run elsewhere.

Therefore what I tend to do is follow this process:

• Use the Wordpress plugin "Limit Login Attempts" to log people trying to login (without permission) into my WordPress website. This will log all the IP addresses that have attempted and failed as well as those tht have been blocked. This is a good starting list for your DENY HOSTS IP ban table
• Check the same IP's as well as using the command: tail -n 10000 access_log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more  to see which IP addresses are visiting my site the most each day.
• I then check the log files either in WebMin or in an SSH tool like PUTTY to see how many times they have been trying to visit my site. If I see lots of HEAD or POST/GET requests within a few seconds from the same IP I will then investigate them further. I will do an nslookup and a whois and see how many times the IP address has been visiting the site.
• If they look suspicious e.g the same IP with multiple user-agents or lots of requests within a short time period I will comsider banning them. Anyone who is using IE 6 as a user-agent is a good suspect (who uses IE 6 anymore apart from scrapers and hackers!)
• I will then add them to my .htaccess file and return a [F] (403 status code) to all their requests.
• If they keep hammering my site I wll then move them from my DENY list in my .htaccess fle and add them to my firewall and Deny Hosts table.
• The aim is to move the most troublesome IP's and BOTS up the chain so they cause the least damage to your site. 
• Using PHP to block access is not good as it consumes memory and CPU, the .htaccess file is better but still requires APACHE to run the regular expressions on every DENY or [F] command. Therefore the most troublesome users should be moved up to the Firewall level to cause the less server usage to your system.
• Reguarly shut down your APACHE server and use the REPAIR and OPTIMIZE options to de-frag your table indexes and ensure the tables are performing as well as possible. I have many articles on this site on other tools which can help you increase your WordPress sites perforance with free tools.

In More Details

You should regularly check the access log files for the most IP's hitting your site, check them out with a reverse DNS tool to see where they come from and if they are of no benefit to you (e.g not a SERP or Social Media agent you want hitting your site) then add them to your .htaccess file under the DENY commands e.g

order allow,deny
deny from 208.115.224.0/24
deny from 37.9.53.71

Then if I find they are still hammering my site after a week or month of getting 403 commands and ignoring them I add them to the firewall in WebMin.


Blocking Traffic at the Firewall level

If you use LINUX and have WebMin installed it is pretty easy to do.

Just go to the WebMin panel and under the "Networking" menu is an item called "Linux Firewall". Select that and a panel will open up with all the current IP addresses, Ports and packets that allowed or denied access to your server.

Choose the "Add Rule" command or if you have an existing Deny command you have setup then it's quicker to just clone it and change the IP address. However if you don't have any setup yet then you just need to do the following.

In the window that opens up just follow these steps to block an IP address from accessing your server.

In the Chain and Action Details Panel at the top:


Add a Rule Comment such as "Block 71.32.122.222 Some Horrible BOT"
In the Action to take option select "Drop"
In the Reject with ICMP Type select "Default"

In Condition Details Panel:

In source address of network select "Equals" and then add the IP address you want to ban e.g 71.32.122.222
In network protocol select "Equals" and then "TCP"

Hit "Save"

The rule should now be saved and your firewall should now ban all TCP traffic from that IP address by dropping any packets it receives as soon as it gets them.

Watch as your performance improves and the number of 403 status codes in your access files drop - until the next horrible social media BOT comes on the scene and tries scrapping all your data.

IMPORTANT NOTE

WebMin isn't very clear on this and I found out the hard way by noticing that IP addresses I had supposedly blocked were still appearing in my access log.

You need to make sure all your DENY RULES are above the default ALLOW rules in the table WebMin will show you.

Therefore your rules to block bad bots, and IP addresses that are hammering away at your server - which you can check in PUTTY with a command like this:
tail -n 10000 access_log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more 

Should be put above all your other commands e.g:


Drop If protocol is TCP and source is 91.207.8.110
Drop If protocol is TCP and source is 95.122.101.52
Accept If input interface is not eth0
Accept If protocol is TCP and TCP flags ACK (of ACK) are set
Accept If state of connection is ESTABLISHED
Accept If state of connection is RELATED
Accept If protocol is ICMP and ICMP type is echo-reply
Accept If protocol is ICMP and ICMP type is destination-unreachable
Accept If protocol is ICMP and ICMP type is source-quench
Accept If protocol is ICMP and ICMP type is parameter-problem
Accept If protocol is ICMP and ICMP type is time-exceeded
Accept If protocol is TCP and destination port is auth
Accept If protocol is TCP and destination port is 443
Accept If protocol is TCP and destination ports are 25,587
Accept If protocol is ICMP and ICMP type is echo-request
Accept If protocol is TCP and destination port is 80
Accept If protocol is TCP and destination port is 22
Accept If protocol is TCP and destination ports are 143,220,993,21,20
Accept If protocol is TCP and destination port is 10000


If you have added loads at the bottom then you might need to copy out the IPTables list to a text editor, change the order by putting all the DENY rules at the top then re-saving the whole IPTable list to your server before a re-start of APACHE.

Or you can use the arrows by the side of each rule to move the rule up or down in the table - which is a very laborious task if you have lots of rules.

So if you find yourself still being hammered by IP addresses you thought you had blocked then check the order of your commands in your firewall and make sure they are are at the top NOT the bottom of your list of IP addresses.

Problem with SFTP after new router installation

Problem with SFTP / SSH after new router installation

This may seem like an obvious one but it can catch you out if you are not aware of the implications of having a new IP address assigned to your house's broadband. You may have moved your laptop to a new house or using a new wifi system to connect to the Internet.

More commonly you may have been given a new or upgraded router by your ISP provider.

Even if you are told the IP address has not changed by BT, Virgin, Sky, Verizon or whoever is giving you the new router you should do a check on any of the many IP checking pages out there on the web.

E.G this script shows you your current IP and ISP details.


Why is this important?

Well if you have your own server being hosted by a company e.g a cloud server somewhere and you have installed DENY HOSTS to block hacking attacks then you might find that you cannot SFTP (Secure FTP) into your server anymore or that using Putty and SSH to access your remote server suddenly stops working for no apparent reason. Obviously you want to access your server so the problem needs fixing.

Symptoms of an IP change causing problems include:

• Your server reporting error messages such as "server unexpectedly closed the connection."
• When you change the file transfer settings to simple FTP from SFTP you can access the server but then experience timeout errors or when the list directory command is run nothing happens.
• Not being able to use Putty or another SSH tool to connect to your server.
• Not having changed any settings on your computer but not being able to connect to your server anymore.

Solution to IP change:

• Check and write down your new IP address.
• Log into your server through WebMin or a web based system or from another computer that hasn't had an IP change.
• Check your DENY HOSTS list to see if your IP address is listed and if so delete the record.
• Add your new IP address in the ALLOW HOSTS list.
• Re-start your server. 

If you don't know how to do this read the 3rd part of my Wordpress Survival Guide about security.

How to install WebMin on LINUX

How to install WebMin on LINUX

If you are from a Windows background and used to graphical interfaces then moving to a LINUX system can be quite a challenge due to the fact that most techies like to use commands from the console to carry out their work.

Whilst it is a good opportunity to learn some of the commands when you first rent a virtual server from Rackspace or buy your own box then you want to get your sites up and running quickly without having to read a lot of information first.

Therefore if you don't have the time to learn all the LINUX commands installing WebMin on your system is a quick way of providing you with a graphical interface to allow you to edit files and stop and start services like the Apache webserver or MySQL database server, as well as managing these services and configuring settings through a visual interface. You will need to have Java installed to use parts of WebMin.

If you want to learn how to use the console for carrying out your work then a good list of Linux applications and commands can be found here: Linux Commands and you should read some of the following articles I have written specifically about performance turning and managing a system such as Wordpress on a LINUX based system.

The Wordpress Survival Guide Part 1
The Wordpress Survival Guide Part 2
Performance Tuning Tools for MySQL
Using Host Headers to setup a test site on LINUX
Problems with LINUX Apache and PHP
Debugging Memory Issues on Wordpress

A quick cheat sheet of the most popular commands I find myself constantly using from the command line through a tool like PUTTY are below. Make a copy and save them to your desktop for quick and easy access.

Command	Details
date	Show the current date and time on the server
cd	change drive e.g cd /var (go to the var directory)
cd ../	go back up one directory
cd ../../	go back up two directories
ls	list out the contents of a directory
whoami	see who you are logged in as
su - [username]	Assume the permissions of the specified user
sudo [command]	Run a command as root but stay as the user you are logged in as
top	Show the current running processes and server load
top -d .2	Show the current running processes with .2 second refresh
tail -f access_log	View the most current entries in the sites access log
grep "61.252.14.247" access_log | tail	View the most current entries in the sites access log for a certain IP address
netstat -ta	Show all current connections to the server
grep "27/Feb/2012:" access_log | sed 's/ - -.*//' | sort | uniq -c | sort -nr | less	View the IP's that appear in your access log the most for a certain date ordered by the most frequent first.
/etc/init.d/apache2 restart	Restart Apache
apache2ctl configtest	Test the Apache configuration for configuration errors
/etc/init.d/mysql restart	Restart MySQL
wget [URL]	Remotely access, load and save a file to the current directory
chmod 777 [filepath]	Grant full read/write/delete permission to everyone to a file or folder
chmod +x [filepath]	Grant execute permission to a script
reboot	Reboot the server

How to install WebMin on a LINUX

Back to WebMin, to install WebMin on a LINUX based server you should do the following.

1. Install PUTTY if you haven't already.
2. Connect to your server through SSH using PUTTY.
3. Download the WebMin file with the following command:
wget http://prdownloads.sourceforge.net/webadmin/webmin_1.580_all.deb

4. Run the following command: dpkg --install webmin_1.580_all.deb

You should have now installed WebMin to /usr/share/webmin.

The administration username will be set to root and the password will be set to your current root password.

You should now be able to login to Webmin by accessing your hostname/IP on the port 10000.

For example e.g if your hostname is myhost.some-site.com then use http://myhost.some-site.com:10000/.

If you server is running in SSL mode (which it should) then it will initially give you a "Bad Request" error and ask you to try a secure URL instead e.g https://myhost.some-site.com:10000/

The WebMin options will appear as a tab to the right of the VirtualMin tab in the top left of the screen.

Using Host Headers on a Virtual Server and creating a new site before buying a domain name

How to create and test a website before buying the domain name


Sometimes you may want to create a new website but don't have a domain name to point it at but still want to test everything is working.

Today I wanted to do this and there were a number of issues I had to work through so I thought I would list them for all the Windows developers who are new to LINUX and APACHE.

The site was a basic Wordpress site which was going to run on a Rackspace Virtual Server which already hosted a number of sites.

If you have installed VirtualMin and WebMin it's very easy to manage your server with a graphical interface otherwise you will need to do everything through a SSH client like PUTTY terminal serviced into your server.

1. Create a new virtual site

• Login into your webserver which will be your hostname on port 1000 e.g: https://myhostname.blah.com:10000/
• If you login as root you won't need to create Webmin logins for each virtual host.
• Choose the "Create Virtual Server" option on the left menu.
• Complete the form entering the name of the domain you want to use e.g  robstestdomain  I tend to use the domain name as the description field as well.
• Enter a password - this will be used for your FTP and Database password unless you change it later - make it secure by using a mixture of letters and numbers both lower and uppercase. If you cannot remember a complex string use a combination of words or a sentence and add numbers or symbols between each word and at the beginning and end e.g 77Strictly2012Software1984
• Under the Enabled features I ticked the following options:

Setup website for domain?
Accept mail for domain?
Setup Webalizer for web logs?
Create MySQL database?
Enable AWstats reporting?

• Click the "Create Server" button and the system will generate everything it needs with an output like so

Setting Up Virtual Server

In domain robstestdomain.com

Creating administration group robstestdomain ..
.. done

Creating administration user robstestdomain ..
.. done

Creating aliases for administration user ..
.. done

Adding administration user to groups ..
.. done

Creating home directory ..
.. done

Creating mailbox for administration user ..
.. done

Adding to email domains list ..
.. done

Adding default mail aliases ..
.. done

Adding new virtual website ..
.. done

Adding webserver user www-data to server's group ..
.. done

Performing other Apache configuration ..
.. done

Setting up scheduled Webalizer reporting ..
.. done

Setting up log file rotation ..
.. done

Creating MySQL login ..
.. done

Creating MySQL database robstestdomain ..
.. done

Setting up AWstats reporting ..
.. done

Setting up password protection for AWstats ..
.. done

Applying web server configuration ..
.. done

Saving server details ..
.. done

Once you have done this open up your favourite FTP client like FileZilla and add the new domains details to the Site Manager. For example if you were creating a secure FTP site you would use these settings.

1. Click "New Site" give it the name of your domain > "robstestdomain"
2. On the right fill out the form with
3. Host: myhostname.blah.com
4. Protocol: SFTP - SSH File Transfer Protocol (secure FTP) you can choose FTP if your hosts firewall is setup to allow traffic through ports 20 and 21.
5. Logon type: Normal
6. User: robstestdomain
7. Password: [the same password you entered when creating the Virtual Server] e,g 77Strictly2012Software1984
8. Click "Connect" your FTP client should now connect to the new folder on your server which will be empty apart from some default folders like cgi-bin, homes, logs, public_html etc.

Download the latest version of Wordpress from their site: http://wordpress.org/download/

Extract the files and copy them up to the public_html folder on your server e.g: /home/robstestdomain/public_html

You will now have extra folders like wp-content, wp-includes and wp-admin under /public_html as well as a number of files including wp-config-sample.php.

You need to edit this file and replace the main database connection settings at the top of the file. These will be the same settings you used when creating the site unless you have manually gone in and changed them.

// ** MySQL settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define('DB_NAME', 'robstestdomain');

/** MySQL database username */
define('DB_USER', 'robstestdomain');

/** MySQL database password */
define('DB_PASSWORD', '77Strictly2012Software1984');

/** MySQL hostname */
define('DB_HOST', 'localhost');

One thing I found earlier is that if you created a domain with a hyphen in it like "My-Domain" then when you create the username you should replace the hyphen with an underscore otherwise it won't be able to connect to the database.

You should also add in sentences for all the security keys to make your site more secure. The longer the words the better.

define('AUTH_KEY',         'some phrase or two');
define('SECURE_AUTH_KEY',  'another unique phrase 100003433');

....

define('NONCE_SALT',       'something else 23423423');

Once complete save the file as wp-config.php and upload to the root folder e.g /public-html/wp-config.php

Now that you have a virtual server setup you need to make sure that you can access it from your own computer as it currently isn't visible online as there is no DNS record setup pointing a domain to your site.

To do this you can edit your hosts file which is located om your computer (windows 7) under
c:\windows\System32\drivers\etc\hosts.

You need to add the following line to the host file that points the IP address of your webserver to the domain you have just created.

If you don't know the IP address of your host just open a command prompt and type "PING [hostname]" e.g PING myhostname.blah.com

Reply from 179.202.123.19: bytes=32 time=105ms TTL=49

In your host file at the bottom of any exiting list put the IP address of your host and the domain you want it to point to e.g

179.202.123.19 www.robstestdomain.com

If you have problems editing the file such as it saying that the file is "Read Only" even when it's not set as Read Only, or you don't have sufficient administrator privileges to edit the file then do the following:

1. Right click on Notepad and choose "Run as administrator"
2. Use the File > Open option to open the hosts file
3. Add the line in and then choose File > Save.

Make sure you shut down your browser after editing the host file before re-opening it and then typing in the domain name in the address bar e.g http://www.robstestdomain.com.

If everything has gone okay you should be taken to the Wordpress configuration page e.g http://www.robstestdomain.com /wp-admin/install.php where you can complete the setup by entering details of your sitename, email and password.

Once the form is completed click on the "Install Wordpress" button it will build all the tables in your database and activate your site. You can check the database has been build from within VirtualMin or by using a MySQL database management system like Navicat.

Remember if you have other sites on the same server that have DNS records setup they should still work by entering their domain names into the address bar but because you have told your computer to always take the domain and point it at your server in the host file you should be able to see your new site even if no-one else can.

You could enter your IP address alone instead of the domain name but either should bring you to the new site.

If you don't want people to access your new site during development by hitting your IP address (not that many real users would do) you should setup some htaccess or deny hosts rules so that only traffic from your own PC can directly use the IP address or your new test domain name and return a 403 status code to everybody else.

Once you have set your site up, installed your themes and plugins and read my Wordpress Survival Guides, one and two on the best plugins to use for performance and the best Apache config for your site you will be ready to rumble. Go and buy a domain from http://www.123-reg.co.uk/ and point it to your new virtual server.

If you have any problems you should also check your new sites Apache config file e.g /etc/apache2/sites-available/robstestdomain.com.conf and make sure its setup like you want. Open an existing config file for one of your other working sites and copy and paste the data changing the names to your new site.


Always restart your server after changing your conflig files but only after running a test for any syntax errors in any of your config files. You can do this in your console with the following lines.

myhost:~:# apache2ctl configtest
Syntax OK
myhost:~:# /etc/init.d/apache2 restart

You can also restart Apache through WebMin which is located under the System > Bootup and Shutdown menu option.

And that my friends is how you do it.

Update: 27-FEB-2012

If you change your mind about the domain name you want to use for your site you will need to delete the existing virtual server and set-up a new one. This might cause problems if you are behind a proxy at work that doesn't allow your hosts file to work. Therefore you should read this article on deleting a site in VirtualMin and accessing a Wordpress site by IP address alone.

The Wordpress Survival Guide

Surviving Wordpress - An Introduction

As well as my technical blog here on blogger I have a number of Wordpress sites which I host myself on a virtual server.

I have now been using Wordpress and PHP for about 4 years and in that time I have learnt a hell of a lot regarding the pros and cons and do's and don'ts that are involved in running your own Wordpress blog.

As a developer who has spent most of his working career working with Microsoft products moving from a Windows environment to Linux has been a great learning curve and as well as all the tips I have gathered regarding Wordpress I thought I would write a series for other developers in a similar situation or for Webmasters who may have the sites but not the technical know how.

Covering the bases

In the following articles I will look in detail at how to get the most out of your system in terms of performance. If you are like me you are not made of money and able to afford lots of dedicated servers to host your sites on therefore you need to make the most of what you have. Performance tuning your Wordpress site is the most important thing you can do and luckily due to the nature of WordPresses plugins a lot of performance tuning can be done with a couple of clicks.

I will also be looking at performance tuning MySQL which is the database that Wordpress runs on. Moving from MS SQL with all its features and DMV's to MySQL was quite a culture shock for me so there are a few tips I have learnt which might be useful.

First things first - Tools of the trade

First off you will need to know how to get things done. My Wordpress sites are running on a Linux box and one of the first things I did was install VirutalMin which is a graphical UI you access in your browser which lets you manage your server. You could do everything from the command line but coming from a windows environment I found it very useful to be able to see a familiar environment.

After installing VirtualMin you should also install WebMin which is another graphical interface that gives you ultimate flexibility over your server without you ever needing to go near a command line prompt.

As well as setting up FTP to access your files through SFTP (secure FTP) I also installed PUTTY which enables me to connect to my server and get used to the command line way of doing things. I would definitely recommend doing this even if you were like me a Windows person as you should never be afraid to try something new and it's always good to have as many technical skills under your belt as possible. I always try to use the command line first but I know I can fall back on VMin if I need to.

Useful Commands

A good list of Linux applications and commands can be found here: Linux Commands but here are some of the key commands I find myself using over and again.

Command	Details
date	Show the current date and time on the server
cd	change drive e.g cd /var (go to the var directory)
cd ../	go back up one directory
cd ../../	go back up two directories
ls	list out the contents of a directory
whoami	see who you are logged in as
su - [username]	Assume the permissions of the specified user
sudo [command]	Run a command as root but stay as the user you are logged in as
top	Show the current running processes and server load
top -d .2	Show the current running processes with .2 second refresh
tail -f access_log	View the most current entries in the sites access log
grep "61.252.14.247" access_log | tail	View the most current entries in the sites access log for a certain IP address
netstat -ta	Show all current connections to the server
grep "27/Feb/2012:" access_log | sed 's/ - -.*//' | sort | uniq -c | sort -nr | less	View the IP's that appear in your access log the most for a certain date ordered by the most frequent first.
/etc/init.d/apache2 restart	Restart Apache
apache2ctl configtest	Test the Apache configuration for configuration errors
/etc/init.d/mysql restart	Restart MySQL
wget [URL]	Remotely access, load and save a file to the current directory
chmod 777 [filepath]	Grant full read/write/delete permission to everyone to a file or folder
chmod +x [filepath]	Grant execute permission to a script
reboot	Reboot the server

Handling Emergencies

You need to be prepared to handle emergencies and that involves a quick diagnosis and quick action. What sorts of emergencies can you expect to run into? Well the most common form will be very poor server performance that results in your site being unavailable to visitors. This can happen for a number of reasons including:

1. High visitor traffic due to a popular article appearing on a major site or another form of traffic driver.

2. High bot traffic from undesirable crawlers such as content scrapers, spammers, hackbots or even a denial of service attack. I recently experienced a DOS attack which came from an out of control bot that was making 10+ requests to my homepage a second.

3. A poorly written plugin that is eating up resources.

4. A corrupt database table causing database errors or poorly performing SQL causing long wait times.

5. Moderately high visitor traffic mixed with an unoptimised system set-up that exacerbates the problem.

Identifying the cause of your problem

If you are having a major slow down, site freeze or just don't know what is going on then the first thing is to open up a command prompt and run top to see the current processes.

The first thing to look at is the load average as this tells you the amount of resources and pressure your server is currently under. A value of 1.00 means your server is maxed out

and anything over that means you are in trouble. I have had a value of 124 before which wasn't good at all. My site was inaccessible and only a cold reboot could get me back to a controlable state.

If your load average is high then take a look at the types of processes that are consuming the most resources. You should be looking at the CPU% and Memory used by each process (the RES) column which shows the amount of physical memory in KB consumed by the process.

Each request to your site will use its own process so if your report is full of Apache rows then you are having a traffic spike. Each page request on your site should be a very quick affair so the processes will come and go very speedily and having a short delay interval is important to being able to spot problems.

Another process to look for is the MySQL process which again should come and go unless it's currently running a long performance intensive query in which case the database could be your problem. Another tool I like to use is mytop which gives you a top like display but of your MySQL processes only. You could open up your MySQL console and run SHOW PROCESSLIST constantly but using MyTop is alot easier and it will help identify problematic queries as well as queries that are being run by your site a lot.

If you don't have monitoring tools available to keep you up to date with your sites status then you may find that your system experiences severe problems periodically without your knowledge. Being informed by a customer that your site is down is never the best way of finding out therefore you might be interested in a plugin I developed called Strictly System Check.

This Wordpress plugin is a reporting tool that you run at scheduled intervals from a CRON job. The plugin will check that your site is available and reporting a 200 status code as well as scanning for known text. It will also connect to your database, check the database and server load and report on a number of important status variables such as the number of connections, aborted connections, slow queries and much more.

The great thing about this plugin is that if it finds any issues with the database it will CHECK and then REPAIR any corrupt tables as well as running the OPTIMIZE command to keep the tables defragged and fast. If any problems are found an email can be send to let you know. I wrote this plugin because there wasn't anything like it available and I have found it 100% useful in not only keeping me informed of site issues but also in maintaining my system automatically.

Scanning Access logs for heavy hitters

Something else you should take a look straight away is your access and error logs.

If you open up your access log and watch it for a while you should soon see if you are experiencing either high traffic in general or from a particular IP/Useragent such as a malicious bot. Using a command like tail or less to view the logfile with the -f flag ensures that as new data is added to the file it will be outputted to the screen which is what you want when examining current site usage.

mydomain:~# cd /home/mywebsite/logs

mydomain:~# tail -f access_log

Banning Bad Users

If the problem is down to one particular IP or Useragent who is hammering your site then one solution is to ban the robot by returning it a 403 Forbidden status code which you can do with your .htacess file by adding in the following lines:

order allow,deny

deny from 79.125.58.227

deny from 67.207.201.

deny from 89.146.55.222

allow from all

This will return 403 forbidden codes to all requests from the two IP addresses and the one IP subnet: 67.207.201.

If you don't want to ban by IP but by user-agent then you can use the Mod Rewrite rules to identify bad agents in the following manner:

RewriteCond %{HTTP_USER_AGENT} (?:ColdFusion|curl|HTTPClient|Java|libwww|LWP|Nutch|PECL|POE|Python|Snoopy|urllib|WinHttp) [NC,OR] # HTTP libraries
RewriteCond %{HTTP_USER_AGENT} (?:ati2qs|cz32ts|indy|linkcheck|Morfeus|NV32ts|Pangolin|Paros|ripper|scanner) [NC,OR] # hackbots or sql injection detector tools being misused!
RewriteCond %{HTTP_USER_AGENT} (?:AcoiRobot|alligator|auto|bandit|capture|collector|copier|disco|devil|downloader|fetch\s|flickbot|hapax|hook|igetter|jetcar|kmbot|leach|mole|miner|mirror|mxbot|race|reaper|sauger|sucker|snake|stripper|vampire|weasel|whacker|xenu|zeus|zip) [NC] # offline downloaders and image grabbers
RewriteRule .* - [F,L]

Here I am banning a multitude of known bad user-agents as well as a number of the popular HTTP libraries that script kiddies and hackers use off the shelf without knowing how to configure to hide the default values.

You should read up on banning bad robots using the htaccess file and Mod Rewrite as a considerable proportion of your traffic will be from non human bots and not the good kind e.g Googlebot or Yahoo. By banning bad bots, content scrapers, spammers, hackers and bandwidth leeches you will not only reduce the load on your server but save yourself money on bandwidth charges.

The other log file you should check ASAP in a potential emergency situation is the Apache error log as this will tell you if the problem is related to a PHP bug, a Wordpress plugin or MySQL error.

Unless you have disabled all your warnings and info messages the error log is likely to be full of non fatal errors however anything else should be checked out. If your error log is full of database errors such as "table X is corrupt" or "database has gone away" then you know where to look for a solution.

Tables get corrupted for many reasons but a common reason I have found is when I have had to carry out a cold reboot to regain control of my server. Sometimes after a reboot everything will be working okay but on accessing your website all the content will have disappeared. Do not worry yet as this could be due to the corrupt tables and carrying out a REPAIR command should remedy this.

Another potential flash point are new or recently upgraded plugins. Plugins can be written by anybody and there is no guarantee whatsoever that the code contained within the plugin is of any quality whatsoever even if the features it offers seem to be great. I have personally found some of the most popular plugins to be performance hogs due to either poor code or complex queries with missing indexes but more of that in a later article.

Unless you are prepared to tweak other peoples code you don't have many options apart from optimising the queries the plugin runs by adding missing indexes or disabling the plugin and finding an alternative. One good tip I have found is to create an empty plugin folder in the same directory as the current plugin folder and then in emergency situations you can rename your existing plugin folder to something like plugins_old and then your site will be running without plugins. Once you have remedied any problems you can add your plugins back one by one to ensure they don't cause any problems.

Regular Maintenance

You should reguarly check your access and error logs even when the site is running smoothly to ensure that problems don't build up without you realising. You should also check your slow query log for poor queries especially after installing new plugins as it's very easy to gain extra performance from adding missing indexes especially when your site has tens of thousands of articles.

You should also carry out regular backups of your database and Wordpress site and ensure that you run the OPTIMIZE command to defrag fragmented table indexes especially if you have deleted data such as posts, tags or comments. A fragmented table is slower to scan and it's very easy to optimize at the click of a button. Take a look at the Strictly System Check Wordpress Plugin which can be setup to report on and analyse your system at scheduled intervals as one of the features is the ability to run the OPTIMIZE command.

So this is the end of the first part of the Wordpress Survival Guide series and next time I will be looking at performance tuning and site optimisation techniques.

Read Part 2 - Performance
Read Part 3 - Security