Stop BOTS and Scrapers from bringing your site down

Blocking Traffic using WebMin on LINUX at the Firewall

If you have read my survival guides on Wordpress you will see that you have to do a lot of work just to get a stable and fast site due to all the code that is included.

The Wordpress Survival Guide

1. Wordpress Basics - Tools of the trade, useful commands, handling emergencies, banning bad traffic.
2. Wordpress Performance - Caching, plugins, bottlenecks, Indexing, turning off features
3. Wordpress Security - plugins, htaccess rules, denyhosts.

For instance not only do you have to handle badly written plugins that could contain security holes and slow the performance of your site but the general WordPress codebase is in my opinion a very badly written piece of code.

However they are slowly learning and I remember once (and only a few versions back) that on the home page there were over 200+ queries being run most of them were returning single rows.

For example if you used a plugin like Debug Queries you would see lots of SELECT statements on your homepage that returned a single row for each article shown for every post as well as the META data, categories and tags associated with the post.

So instead of one query that returned the whole data set for the page in one query (post data, category, tag and meta data) it would be filled with lots of single queries like this.

SELECT wp_posts.* FROM wp_posts WHERE ID IN (36800)

However they have improved their code and a recent check of one of my sites showed that although they are still using seperate queries for post, category/tag and meta data they are at least getting all of the records in one go e.g

SELECT wp_posts.* FROM wp_posts WHERE ID IN (36800,36799,36798,36797,36796)

So the total number of queries has dropped which aids performance. However in my opinion they could write one query for the whole page that returned all the data they needed and hopefully in a future edition they will.

However one of the things that will kill a site like Wordpress is the amount of BOTS that hit you all day long. These could be good BOTS like GoogleBOT and BingBOT which crawl your site to find out where it should appear in their own search engine or they could be social media BOTS that look for any link Twitter shows or scrapers trying to steal your data.

Some things you can try to stop legitimate BOTS like Google and BING from hammering your site is to set up a Webmaster Tools account in Google and then change the Crawl Rate to a much slower one.

You can also do the same with BING and their webmaster tools account. However with BING they apparently respect the ROBOTS.txt command DELAY e.g

Crawl-delay: 3

Which supposedly tells BOTS that respect the Robots.TXT commands that they should wait 3 seconds before each crawl. However as far as I know only BING support this at the moment and it would be nice if more SERP BOTS did in future.

If you want a basic C# Robots.txt parser that will tell you whether your agent can crawl a page on a site, extract any sitemap command then check out > http://www.strictly-software.com/robotstxt however if you wanted to extend it to add in the Crawl-Delay command it wouldn't be hard ( line 175 in Robot.cs ) to add in so that you could extract and respect it when crawling yourself.

Obviously you want all the SERP BOTS like GoogleBot and Bingbot to search you but there are so many Social Media BOTS and Spammers out there nowadays that they can literally hammer your site into the ground no matter how many caching plugins and .htacess rules you put in to return 403 codes.

The best way to deal with traffic you don't want to hit your site is as high up the chain as possible. 

Just leaving Wordpress to deal with it means the overhead of PHP code running, include files being loaded, regular expression to test for harmful parameters being run and so on.

Moving it up to the .htaccess level is better but it still means your webserver is having to process all the .htacess rules in your file to decide whether or not to let the traffic through or not.

Therefore if you can move the worst offenders up to your Firewall then it will save any code below that level from running and the TCP traffic is stopped before any regular expressions have to be run elsewhere.

Therefore what I tend to do is follow this process:

• Use the Wordpress plugin "Limit Login Attempts" to log people trying to login (without permission) into my WordPress website. This will log all the IP addresses that have attempted and failed as well as those tht have been blocked. This is a good starting list for your DENY HOSTS IP ban table
• Check the same IP's as well as using the command: tail -n 10000 access_log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more  to see which IP addresses are visiting my site the most each day.
• I then check the log files either in WebMin or in an SSH tool like PUTTY to see how many times they have been trying to visit my site. If I see lots of HEAD or POST/GET requests within a few seconds from the same IP I will then investigate them further. I will do an nslookup and a whois and see how many times the IP address has been visiting the site.
• If they look suspicious e.g the same IP with multiple user-agents or lots of requests within a short time period I will comsider banning them. Anyone who is using IE 6 as a user-agent is a good suspect (who uses IE 6 anymore apart from scrapers and hackers!)
• I will then add them to my .htaccess file and return a [F] (403 status code) to all their requests.
• If they keep hammering my site I wll then move them from my DENY list in my .htaccess fle and add them to my firewall and Deny Hosts table.
• The aim is to move the most troublesome IP's and BOTS up the chain so they cause the least damage to your site. 
• Using PHP to block access is not good as it consumes memory and CPU, the .htaccess file is better but still requires APACHE to run the regular expressions on every DENY or [F] command. Therefore the most troublesome users should be moved up to the Firewall level to cause the less server usage to your system.
• Reguarly shut down your APACHE server and use the REPAIR and OPTIMIZE options to de-frag your table indexes and ensure the tables are performing as well as possible. I have many articles on this site on other tools which can help you increase your WordPress sites perforance with free tools.

In More Details

You should regularly check the access log files for the most IP's hitting your site, check them out with a reverse DNS tool to see where they come from and if they are of no benefit to you (e.g not a SERP or Social Media agent you want hitting your site) then add them to your .htaccess file under the DENY commands e.g

order allow,deny
deny from 208.115.224.0/24
deny from 37.9.53.71

Then if I find they are still hammering my site after a week or month of getting 403 commands and ignoring them I add them to the firewall in WebMin.


Blocking Traffic at the Firewall level

If you use LINUX and have WebMin installed it is pretty easy to do.

Just go to the WebMin panel and under the "Networking" menu is an item called "Linux Firewall". Select that and a panel will open up with all the current IP addresses, Ports and packets that allowed or denied access to your server.

Choose the "Add Rule" command or if you have an existing Deny command you have setup then it's quicker to just clone it and change the IP address. However if you don't have any setup yet then you just need to do the following.

In the window that opens up just follow these steps to block an IP address from accessing your server.

In the Chain and Action Details Panel at the top:


Add a Rule Comment such as "Block 71.32.122.222 Some Horrible BOT"
In the Action to take option select "Drop"
In the Reject with ICMP Type select "Default"

In Condition Details Panel:

In source address of network select "Equals" and then add the IP address you want to ban e.g 71.32.122.222
In network protocol select "Equals" and then "TCP"

Hit "Save"

The rule should now be saved and your firewall should now ban all TCP traffic from that IP address by dropping any packets it receives as soon as it gets them.

Watch as your performance improves and the number of 403 status codes in your access files drop - until the next horrible social media BOT comes on the scene and tries scrapping all your data.

IMPORTANT NOTE

WebMin isn't very clear on this and I found out the hard way by noticing that IP addresses I had supposedly blocked were still appearing in my access log.

You need to make sure all your DENY RULES are above the default ALLOW rules in the table WebMin will show you.

Therefore your rules to block bad bots, and IP addresses that are hammering away at your server - which you can check in PUTTY with a command like this:
tail -n 10000 access_log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more 

Should be put above all your other commands e.g:


Drop If protocol is TCP and source is 91.207.8.110
Drop If protocol is TCP and source is 95.122.101.52
Accept If input interface is not eth0
Accept If protocol is TCP and TCP flags ACK (of ACK) are set
Accept If state of connection is ESTABLISHED
Accept If state of connection is RELATED
Accept If protocol is ICMP and ICMP type is echo-reply
Accept If protocol is ICMP and ICMP type is destination-unreachable
Accept If protocol is ICMP and ICMP type is source-quench
Accept If protocol is ICMP and ICMP type is parameter-problem
Accept If protocol is ICMP and ICMP type is time-exceeded
Accept If protocol is TCP and destination port is auth
Accept If protocol is TCP and destination port is 443
Accept If protocol is TCP and destination ports are 25,587
Accept If protocol is ICMP and ICMP type is echo-request
Accept If protocol is TCP and destination port is 80
Accept If protocol is TCP and destination port is 22
Accept If protocol is TCP and destination ports are 143,220,993,21,20
Accept If protocol is TCP and destination port is 10000


If you have added loads at the bottom then you might need to copy out the IPTables list to a text editor, change the order by putting all the DENY rules at the top then re-saving the whole IPTable list to your server before a re-start of APACHE.

Or you can use the arrows by the side of each rule to move the rule up or down in the table - which is a very laborious task if you have lots of rules.

So if you find yourself still being hammered by IP addresses you thought you had blocked then check the order of your commands in your firewall and make sure they are are at the top NOT the bottom of your list of IP addresses.

The Wordpress Survival Guide Part 3 - Security

This is the 3rd part of the Wordpress Survival Guide which looks at security measures.

The other two guides which cover basics for people new to Linux, Apache and Wordpress and Performance can be found here:

The Wordpress Survival Guide Part 1 - Linux, Apache and Wordpress
The Wordpress Survival Guide Part 2 - Performance

If you have an under powered or busy server then security and performance go hand in hand as reducing the amount of traffic from bad bots, hackbots, spammers, login hackers, heavy hitters and so on will also help reduce the load on your server.

There are many plugins out there which claim to help the security on Wordpress but you should be careful as from my own investigation of the code many of these plugins whilst protecting you from potential threats can reduce your sites performance as they carry out too many checks on submitted fields.

If a plugin is checking every form element submitted to the server for hundreds of known SQL injection or XSS hacks with regular expressions or string checks then this can slow down a page load incredibly.

Therefore the further up the chain you can push your security checks from PHP code running in Wordpress to the actual web server the better.

The aim is to move as much blocking code away from your site to your server so we want to make use of our firewall and our .htaccess file by adding a number of rules designed to identify and block potential hackers and spammers before they get to your site and any plugin code.

Blocking with our LINUX Firewall

Once you have found persistent offenders from the methods listed below the aim is to remove any CPU  and Memory from being wasted on them by WordPress and your .htaccess file and put them into your LINUX Firewall.

You can install a plugin to your server called Fail2Ban which will actually analyse your log files for you looking for spammers, hackers and bandwidth wasters and add them automatically to your IPTables (Network Firewall).

However you should read up on it carefully and configure it correctly so that you don't end up blocking yourself sending emails into WordPress or other actions.

The higher up you can block the bad traffic the better. Therefore read this article on how you can block bad BOTS and users by the WebMin interface.

Blocking with the .htaccess file

The .htaccess file sits in your websites root folder and contains rules local to the site which can allow or deny users to your site by blocking certain requests either by IP address, user-agent, or the type of request the user is making. 

I used to return a 403 forbidden status code to the people I wanted to block but I am now trying out a different format which seems to have increased performance. I suspect this might be down to the users of malicious bots seeing a 403 Forbidden code as a "challenge" to crack rather than a sign they should go away therefore I have replaced returning 403 with a 404 code.

As there doesn't seem to be a quick flag like [F] for 403 to use you should create your own 404 page which should contain very basic HTML and no references to any Wordpress include files or other code that could be loaded in.

At the top of you page you put some PHP to return the 404 status code. An example is below.

<?php
header("Status: 404 Not Found");
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-US">
<head profile="http://gmpg.org/xfn/11">
<title>My Website</title>
<link rel="canonical" href="http://www.mywebsite.com/" />
</head>
<body id="home">
 <div id="page-wrap">
  <div id="header">  
   <p id="slogan">This is my website</p>
  </div>
 </div>
 <div id="body">
  <h1>My Website</h1>

  <p>Sorry that page doesn't exist</p>
 </div>
</body>
</html>

The idea is to have a quick loading page that returns a basic response rather than a blank one so that crawlers think they have just made a mistake and that the URL they are targeting doesn't exist. A blank response or a forbidden status could signify to them that you have caught them out. The PHP at the top ensures a 404 status code is returned.

Once you have created the custom 404.php page and put some basic text in it upload it to the root of your website.

Now you can edit your .htaccess file and change some of the main checks we are going to do so that they redirect the bot to the custom 404.php page and not the Wordpress 404 page.

We don't want to get ourselves in a big loop of circular redirects which is why we check for the 404.php page on our 2nd block of rules.

The first set of rules block common SQL injection attacks, common XSS hacks which include passing JavaScript in the querystring, known file lookups as well as calls to certain applications which should never be accessible from the webserver but sometimes are.

The second block is aimed at known bad bot user-agents, common HTTP libraries such as CURL, WGet, Snoopy and other libraries which are usually downloaded by Script Kiddies and used without any modification.

A proper hacker or spammer will mask themselves a lot better than this but these rules will stop the wannabes and baby nobs that have no clue what they are doing but still overload your server.

I also then block blank and very short user-agents or jbberish user-agents as I believe if the user cannot tell me who they are then I don't want them on my site. It is up to you whether you decide you want people masking themselves in this way to access your servers. You will notice that on this section I still use the [F] forbidden flag and return a 403 code.

The last block are known email harvesters and spammers which I redirect off to a honeypot to be logged and blocked by a proper tool designed to catch out email harvesting bots.

I have found that a good set of rules can reduce traffic to a server by over 50% which is obviously a major performance benefit and since I have changed my first two sets of rules from returning 403 to 404 codes the response time of my server and sites upon it has increased.

<IfModule mod_rewrite.c>
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{QUERY_STRING} (%3C|<)/?script(%3E|>)    [NC,OR]
RewriteCond %{QUERY_STRING} (eval\(|document\.|\.cookie|createElement)    [NC,OR]
RewriteCond %{QUERY_STRING} DECLARE[^a-z]+\@\w+[^a-z]+N?VARCHAR\((?:\d{1,4}|max)\)    [NC,OR]
RewriteCond %{QUERY_STRING} ^/+\?.*sys.?(?:objects|columns|tables|[xs]p_|exec)    [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(owssvr|strmver|Auth_data|redirect\.adp|MSOffice|DCShop|msadc|winnt|system32|script|autoexec|formmail\.pl|_mem_bin|NULL\.) [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(php\-?my\-?admin\-?\d?|P\/?M\/?A(\d+)?|(db|web)?(admin|db|sql)|(my)?sql\-?(admin|manager|web)?)/? [NC]
RewriteRule ^.*$ /404.php [R=301,L]


RewriteCond %{REQUEST_FILENAME} !/404\.php # ensure we are not already on our 404.php page
RewriteCond %{HTTP_USER_AGENT} (?:ColdFusion|Jakarta|HTTPClient|Java|libwww\-perl|Nutch|PycURL|Python|Snoopy|urllib) [NC,OR] # common HTTP libraries
RewriteCond %{HTTP_USER_AGENT} (?:LWP|PECL|POE|PycURL|WinHttp|curl|Wget) [OR] # case sensitive HTTP libraries
RewriteCond %{HTTP_USER_AGENT} (?:ati2qs|cz32ts|EventMachine|indy|linkcheck|Morfeus|NV32ts|Pangolin|Paros|ripper|scanner|offline) [NC,OR] # known rippers
RewriteCond %{HTTP_USER_AGENT} (?:AcoiRobot|alligator|auto|bandit|boardreader|BCD2000|blackwidow|capture|ChinaClaw|collector|copier|disco|devil|downloader|fetch|flickbot|grabber|gosospider|Gentoo|HTMLParser|hapax|hook|igetter|jetcar|JS-Kit|kame-rt|kmbot|KKman|leach|majestic|MetaURI|mole|miner|mirror|mxbot|rogerbot|race|reaper|sauger|speedy|Sogou|sucker|snake|spinn3r|Sosospider|stripper|UnwindFetchor|vampire|whacker|xenu|zeus|zip) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (?:AhrefsBot|fairshare|proxy|PageGetter|magpie|Zemanta|baidu|MiniRedir|SurveyBot|PMAFind|SolomonoBot|whitehat|blackhat|MSIE\s6\.0|ZmEu) [NC]
RewriteRule ^.*$ /404.php [R=301,L]

# Block blank or very short user-agents. If they cannot be bothered to tell me who they are or provide jibberish then they are not welcome!                                          
RewriteCond %{HTTP_USER_AGENT} ^(?:-?|[a-z1-9\-\_]{1,10})$ [NC]
RewriteRule .* - [F,L]

# fake referrers and known email harvesters which I send off to a honeytrap full of fake emails
RewriteCond %{HTTP_USER_AGENT} (?:atomic|collect|e?mail|magnet|reaper|siphon|sweeper|harvest|(?:microsoft\surl\scontrol)|wolf) [NC,OR] # spambots, email harvesters
RewriteCond %{HTTP_REFERER} ^[^?]*(?:iaea|\.ideography|addresses)(?:(\.co\.uk)|\.org\|.com) [NC]
RewriteRule ^.*$ http://english-61925045732.spampoison.com [R,L] 

</IfModule>

You should check your websites access and error logs regularly to see who has been getting banned or 404'd a lot by these rules and then you can decide whether to block their IP address by adding it your .htaccess file like so.

order allow,deny
deny from 81.24.210.2 # example IP - not known to be bad

These rules deny all requests from a particular IP address.

OR adding it to your Firewall if it's making a large amount of calls to your site.

One trick I do like, which I have to thank a commenter, RobinInTexas for is this rule which sends the BOT back to the IP address they came from in the first place. However there are two changes to the rule he sent which used

http://%{REMOTE_ADDR} [L,R=301]

And that is to return them to their localhost address 127.0.0.1 NOT the IP they came from as many people will be going through gateways such as people at work, phone or tablet users and people on WIFI systems in shops etc.

You don't want the ISP that this gateway belongs to thinking you are sending it lots of hackers and bad bots as you might get your site blocked for sending so much traffic to it.

The other change is to make it a 302 temporary redirect instead of a 301 as it is the correct status code to use. So instead of the rule above use this.

http://127.0.0.1 [L,R=302]

You could decide to send them to a honeypot website that logs them as a bad BOT so other users know about them or even some sites that are designed to keep them crawling for days wasting time going through links that lead nowhere but to other links that lead nowhere etc.

The rule in action would look like this:

RewriteCond %{HTTP_USER_AGENT} (?:Spider|MJ12bot|seomax|atomic|collect|e?mail|magnet|reaper|tools\.ua\.random|siphon|sweeper|harvest|(?:microsoft\surl\scontrol)|wolf) [NC]
RewriteRule .* http://127.0.0.1 [L,R=302]

2. Blocking SSH Attacks with DenyHosts


Install DenyHosts if you havent already which will block attacks by SSH to your server. It is amazing how many people I have blocked since installing this application and people are always on the look out for new webservers on known cloud hosting IP ranges like Rackspace to attack and hopefully compromise.

To install this you open an SSH connection (with Putty) and run the following commands.

apt-get install denyhosts

to view by console go to the directory the application is installed to.

cd /var/log/denyhosts
tail -f /var/logs/denyhosts

This will show you the tail end of the DenyHosts log file and any newly added IP addresses.

Make sure to add any IP addresses that you access your server console by SSH to the Allow Hosts which you can do by the terminal in VI or from WebMin by going to:

Webmin > Networking > TCP Wrappers > AllowedHosts > Add New Rule

Fill out the form like so:

Services: ALL
Remote hosts: Tick the second radio button and add the IP address to the text input
Shell Commands: none
Save the form.


3. Stop Being Open To SSH / BASH Hack Attacks

Also to stop your server being vulnerable to hacks like the Shellshock hack which appeared recently and exposed nearly every LINUX machine due to their use of SSH and BASH you should do the following.

Test if you are vulnerable by running this command.

env x='() { :;}; echo vulnerable' dash -c "echo this is a test"

If you are then these are somethings you can do.

Turn off BASH and install DASH an older version.

If you are using DASH and want to run BASH just type in BASH to get there.

Also replace the default shell for root and any other users to another folder with symbolic links. Look up on the web how to do this.

Disable any cgi-bin commands in all Apache config files as this is what the hack relies on e.g

#ScriptAlias /cgi-bin/ /home/searchmysite/cgi-bin/
 
#
#allow from all
#

Remove AW stats and Webalizer for all virtual min sites. These rely on CGI-BIN as well.

Regularly change all your user passwords and especially your root password.


A good technique for a strong password is to thing of a common sentence or phrase you will remember and mix the characters up and add a number on the end only you would remember (not your Birthday!) e.g a football teams last trophy win or the year of your last holiday.

Add some dashed or underscores in as well to make it even harder for password crackers to crack it with dictionary attacks. An example would be.

hOWnOWbROWNcOW__1995**

Regularly check your users table for any that look out of place e.g inserted by a hacker.

Also regularly check your home and temp folders for any files that shouldn't be there. One hack I saw replaced the default SSH config file with a temp file in /tmp/sh that loaded up (using WGET) a file hidden in a website that then ran more WGET commands to load in a library of hacks for DDOS and SSH etc and then ran the commands he wanted.

With a compromised server and a SSH config file that had been overwritten he could then use your server to run hack attacks on other machines.

If this is happening, quickly get the IP of the site he is loading the files from and block incoming and outgoing TCP requests in your firewall. Then get a default SSH config file and replace the hacked version before changing all your passwords and ensuring BASH isn't available to be used in a hack.

You can check if anyone who shouldn't be logged into your machine is with the ps ax command.



4. Using Wordpress Plugins to block dangerous traffic.

Two plugins I have found quite useful so far for reducing hack attacks are these:


The Limit Login Attempts plugin which blocks brute force attacks on the wp-login page. If you don't want people signing up to your site anyway you should use a plugin to obfusicate this page anyway otherwise just limit the number of failed attempts so that dictionary attacks are prevented.

http://wordpress.org/extend/plugins/limit-login-attempts/

Use the IP addresses this plugin collects and take the worst offenders and put them in your DENY HOSTS table as well as considering banning them with your LINUX firewall. Read this article for more information on banning bad BOTS and blocking hackers and scrapers.

Install the Wordpress Firwewall plugin to block certain hack attempts and be notified by email when attacks occur. Make sure to add any IP address you access your website to the whitelist so you don't get blocked out.

This plugin will look for some of the same tricks our .htaccess file rules are aimed at blocking as well as some different types of attack that are used when form parameters are filled with dangerous values and submitted to the server.

http://wordpress.org/extend/plugins/wordpress-firewall-2/

There are other things you can do as well but these 3 tips are a good starting point. I will update this page as and when new features are proven at increasing security without effecting site performance.


4. Using other tools on your server to add rules to DenyHosts and your Firewall

There is a tool you can use on LINUX machines called Fail2Ban which RackSpace and other cloud hosters actually recommend using. It will constantly analyse your access and error logs and add IP addresses which it things are suspicious into your DenyHosts list and your Firewall IP Table.

However be-warned I used it myself and tried some of the email rules. I then found myself having my IP being blocked and emails sent from my own computer to my server blocked.

I then tried removing these from the configuration and still ran into problems (not immediately - so it may not have been Fail2Ban's problem) of emails sent from my PC to my WordPress site where a plugin called Postie put them into the system as articles stopped working.

In the end I had to remove the Fail2Ban program from my server. However if you are not doing anything like I am or can configure it properly (I may have made a mistake) then it could be the tool for you as it will save time adding rules into DenyHosts and your IP TABLE for your firewall to use.


Read Part 1 - An Overview
Read Part 2 - Performance