2 Quick Ways To Reduce Traffic On Your System

2 Quick Ways To Reduce Traffic On Your System

By Strictly-Software

Slowing The 3 Major SERP BOTs Down To Reduce Traffic

If you run a site with a lot of pages, good rankings, or a site that tweets out a lot e.g whenever a post comes online then you will probably get most of your traffic from getting crawled by the big 3 crawlers:

I know that whenever I check my access_log on my server to find out the top visiting IP addresses with a command like

grep "Jan/2015" access_log | sed 's/ - -.*//' | sort | uniq -c | sort -nr | less

I always find the top IP's are the main 3 Search Engines own BOTS (SERP = Search Engine Results Page), so I call their BOTS SERP BOTS.

GoogleBot: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Bing: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Yahoo: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Without asking these BOTS to come NOW by doing things like refreshing your sitemap and pinging these SERPS. 

Or tweeting out links then they will crawl your site at their own time and choosing and with nothing to tell them to slow down they will crawl at their own speed. This could be once every second and if so it could cause your site performance issues.

However thing about it logically, if you post a news article or a job advert then in reality it only needs to be crawled once by each SERP BOT for it to be indexed. 

You don't really want it to be crawled every day and on every visit by these BOTS as the content HASN'T changed so there is really no need for the visit.

Now I don't know a way of telling a BOT to only crawl a page only if it's new content or it's changed in some way even if you had a sitemap system that only put in pages that were new or edited as the BOTS will still just vist your site and crawl it.

If you cannot add rel="nofollow" on internal links that point to duplicate content which doesn't actually 100% mean the BOT won't crawl it anyway then there are some things you can try if  you find that your site is having performance problems or is under pressure from heavy loads.

Crawl-Delay

Now this only used to be supported by BingBOT and then some smaller new search engines like Blekko. 

However in recent months after some testing I noticed that all most major SERP BOTS apart from GoogleBOT now obey the command. To get Google to reduce their crawl rate you can use Webmaster Tools to set their crawl rate from the control panel.

For instance on one of my big news sites I have a Crawl-Delay: 25 setting and when I check my access log for those user-agents there is a 25 second (roughly) delay between each request.

Therefore extending this value will reduce your traffic load by the major visitors to your site and is easily done by adding it to your Robot.txt file e.g.

Crawl-delay: 25

Banning IE 6

Now there is no logical reason in the world for any REAL person to be using this user-agent.

This Browser was probably the worst ever Browser in history due to the quirks within it that made web developers jobs so hard. Even just between IE 5.5 and IE 7 there are so many differences with IE 6 and is the reason IE 8 and 9 had all the settings for compatibility modes and browser modes.

It is also the reason IE is going to scrap support for IE 7-9 because of all this hokerery pokery they introduced just to handle the massive differences between IE 6 and their new standard compliant browsers.

Anyone with a Windows computer nowadays should be on at least IE 10. Only if your still on XP and haven't done any Windows Updates since about 5 years ago would you be a real IE 6 user.

Yesterday at work I ran a report on the most used Browsers that day. 

IE 6.0 came 4th!

It was below the 3 SERP BOTS I mentioned earlier and above the latest Chrome version.

On more detailed inspection of my custom logger/defence system that analyses the behaviour of visitors rather than just assuming that because your agent is IE 6 you are actually human could I see these visitors were all BOTS. 

I check for things like whether they could run JavaScript by using JavaScript to log that they can in the same way as I do Flash. These users had no JavaScript or Flash support and the rate they went through pages was way too fast for a human controller.

The only reason I can think people are using this user-agent is because they are script kiddies who have downloaded an old crawling script and the default user-agent is IE 6 and they haven't changed it.

Either they don't have the skill or they are just lazy. However by banning all IE 6 visitors with a simple .htaccess rule like this you can reduce your traffic hugely.

RewriteRule %{HTTP_USER_AGENT} (MSIE\s6\.0|MSIE\s5\.0|MSIE\s5\.5) [NC]
RewriteRule .* http://127.0.0.1 [L,R=302]

This Rewrite rule bans IE 5, 5.5 and IE 6.0 and sends the crawler back to the localhost on the users machine with a 302  rewrite rule.

No normal person would be using these agents. There maybe some Intranets using VBScript as a client side scripting language from the 90's but no modern site is designed with IE 6 in the designers mind. Therefore most sites you find will not hanlde IE 6 very well therefore like Netscape Navigator they are an old browser so don't worry about site support for it. Therefore by banning it you will find your traffic going down a lot by banning just IE 6 and below.

So two simple ideas to reduce your traffic load. Try them and see how much your site improves.

Latest Version of Strictly AutoTags 2.8.9 - Donation Only

Latest Version of Strictly AutoTags 2.8.9 - Paid Version

By Strictly-Software

I have just released the latest version of my highly popular plugin Strictly AutoTags plugin.

Strictly AutoTags 2.8.9 is a paid for version which means it has features in it that are only available to those people who pay me £40 first.

The new paid for version includes the following features:at

Strictly AutoTags - Version 2.8.9

• Updated storage to handle new data-description and data-pin-desc attributes so they don't get tagged inside images by mistake. The same goes for shortcodes now handled by the Jetpack plugin so that [youtube http://www.youtube.com/watch?v=USbkB6rVbpc] doesn't get accidentally tagged either if one of the words within the [shortcode] contained a word e.g [customField sales]
• I have added a new tag equivalent mark up language using a simple method that enables you to match instances of certain words BUT use a different tag. E.G if you want the words Snowden, NSA and Prism to add the tag Police State to your article you would use this markup language [Snowden, NSA, PRISM]=[Police State] 
• To then add the tag terrorism to the words al-Qaeda, bin-Laden, 9.11, WTC and Taliban you would add this [al-Qaeda, bin-Laden, 9.11, WTC, Taliban]=[Terrorism] To put them together you use a pipe so adding these two would mean an input box filled like this [Snowden, NSA, PRISM]=[Police State]|[al-Qaeda, bin-Laden, 9.11, WTC, Taliban]=[Terrorism]
• I've added an option to set the minimum number of letters a tag must have before it is used as a tag. This applies to stored tags or newly found ones. This allows you to skip tags of one or two letters long even if they are ACRONYMS and you don't want to add them to the NOISE word list.
• I've added option to tell system whether or not to convert plain text links like the wording www.msnbc.com into a proper link msnbc.com This is obviously quite difficult due to all the shortened links you can now get, the lack of protocols e.g //twitter.com/strictlytweets and soon UTF8 characters in URLs.
• I've created a new name function to match names like al-Qaeda or al-Nusra Front or even words with commas in like 1,000 Guineas which would be tagged as 1000 Guineas due to the comma being used as a separator in WordPress.
• I've added a new finished_doing_tagging hook into the plugin so other plugins can act once the tagging is finished. This is most useful when using my AutoTag plugin in conjunction with Strictly TweetBot which allows you to use categories and tags as #hashtags in any tweets that are automatically sent out. Without this hook you can run into problems when you have a large number of tags to search through and it's possible a tweet will get sent with their default #HashTags instead of relevant post tags instead like #Snowden or #NSA on an article about the Police State.
• In my latest paid version I have included a new function which will allow you clean out any HTML added by my own tagging process when it bolds and deep-links tags. I use specific classes on my HTML for anchors and spans so I know how to remove them with regular expressions. Obviously the more articles you have the longer this process to clean up will take. You could obviously clean articles out yourself with a simple UPDATE MySQL statement and a RegExp Replace function call but this saves you from having to.

There is a free version up on the Wordpress site, the latest being version 2.8.8, but this has far less features.

When I bring out new paid only versions I may add a few of the older paid for features into the free versions.

However the whole reason I am doing this is, is due to the fact that although over 180,000 people have downloaded my plugin and obviously think it's useful from Wordpress (and who knows how many from my own website), I have had hardly any donations.

It seems hardly worth my time putting the code out on the web for others when I can just use it myself and gain all the SEO benefits it has brought me (which it has!).

Benefits such as:

• Deep-linking a specified number of the most important words in an article to their "tag listings" page where crawlers and users can find lots more content related to that word / tag.
• Also the deep-linking is done on INPUT not OUTPUT. Which means it's faster to load pages as the formatting is already done. Do something once on INPUT not a THOUSAND times on OUTPUT, especially not when thousands of BOTS and HACKERS come crawling using up CPU and bandwidth!
• Hit Highlighting, bolding or using links, around important words that are key and related to an article is a good way of telling SERP crawlers which words are the most important for an article.Automatically doing this without waiting for 3rd party API's to add new tags relevant to the latest news stories (Peoples names, buildings, companies etc) is a lot faster.

The money I have been donated so far by a few people, usually for adding new features to the plugin has certainly not enough been enough money to warrant me spending the amount of time and effort I have on this plugin so so far - and that's A LOT! This is why I have brought out a paid version as the main version which will always have the latest features in it.

Believe it or not a £2-£5 donation maybe a small sign of gratitude but as this is is proper coding and "Open Source", I thought the idea was to take other peoples code and "change", "edit" or "extend" it. Maybe I was wrong but that's what I did with my first sitemap plugin when all the others were giving me problems. I literally learnt Wordpress AND PHP at the same time just so I could make a fast performing plugin that worked with 50,000+ articles.

Also as I get paid £650 a day contracting a fiver just isn't worth a days work I'm afraid.

Also let me remind those people who still cannot read English:

• I don't work for free.
• I don't add new features into an open source plugin just because you want them. It takes time and effort!
• I don't drop everything to help you configure a plugin which has a simple "Test Config" button in the admin panel that tests everything is set up for you and if not tells you what the problems are and what you need to do to fix them. I do this to save support requests believe it or not.
• I also have a "shit list" for tight wads who have stitched me up in the past. Either by promising me constant work, large payments later on, or payments after the work has been done. If they fail to live up to their side of the agreement or are just plain rude they get their name and comment / email put on my very own "Shit List". You can read some of them here: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html.

The latest is quite funny, not just because of the broken English but because the person seemed to want me to drop everything and fix something that isn't even broken ASAP!

Not only do I not even support or release new versions of the plugin in question "Strictly Google Sitemap" but the problem the person is talking about is a simple case of him not hitting the "Save Settings" button first before hitting the "Build Sitemap" button - DOH!

Have a laugh as you read (his partially corrected by me) broken English request trying to get me to help him out.

hi dear rob i hope you are fine

i have a very bad probelm
my site hosting has limit and i can't build full sitemap with google xml sitemap by arne

i installed your plugin i see very bad problem
when i installed i went in setting of your sitemap
i changed setting when i clicked in manual building sitemap, 
sitemap build but with default settings

dear rob i don't have time to waste!

If only I could build in a Doofus Config Test button to check the user has basic computer skills first before allowing them to use it that would be a money spinner!

I do like the "dear rob I don't have time to waste!" bit at the end. In the real email he left the end "e" of the word waste but I thought I would add it on to help you understand him more easily.

Yes of course, you have no time to waste so I will just drop everything and fix something for your for free. I think not!

This "Tight Wad" list has worked quite well so far and I have had quite a few people contact me and pay me just to take their name off it.

I suppose they have done a Google search for their own name when they were bored one day only to find themselves on my blog in an article about tight arses who talk the talk but can't walk the walk.

Oh well maybe they should have read the specific part of the FAQ page on the WordPress plugin page that states how they should go about debugging any problem and to prevent themselves ending up on my "tightwad list". 

It's quite simple and in the FAQ section.. From the WordPress sites FAQ Section non the website about the plugin.

I have an error.

If you have any error messages installing the plugin then please try the following to rule out conflicts with other plugins:

• Disable all other plugins and then try to re-activate the Strictly Google Sitemap plugin as some caching plugins can cause issues.

• If that worked, re-enable the plugins one by one to find the plugin causing the problem. Decide which plugin you want to use.

• If that didn't work check you have the latest version of the plugin software (from WordPress) and the latest version of WordPress installed

• Check you have JavaScript and Cookies enabled.

• If you can code turn on the DEBUG constant and debug the code to find the problem otherwise contact me and offer me some money to fix the issue :)

• Please remember that you get what you pay for so you cannot expect 24 hour support for a free product. Please bear that in mind if you decide to email me a donation button is on my site and on the plugin admin page.

• If you must email me and haven't chosen to donate even the smallest amount of money please read this: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html.

• If you don't want to pay for support then ask a question on the message board and hop

Or maybe they should have read the "Support" page that clearly states I don't support this Sitemap plugin any-more. The main reason being that I got Cancer at the beginning of 2012.

              

Everything moves so fast it is hard to keep up and build business models on top of 3rd party code - just look at Twitter and their culling of all the DM message responder applications that put a myriad of companies out of business.

However despite these other SEO features not working the core Sitemap functionality still works (on non MU sites) and I use it on all of my own sites.

However if you want support you must donate. Otherwise you go onto the list: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html

I just haven't got time to answer questions in broken English that make so sense as if I "must" help them for some reason. It's 3rd party, open source and free code - sort it out yourself or pay someone to do it for you! Simples!

So there you go a brand new donation only version of AutoTags and some reasons why you shouldn't piss me off :)

If your nice I help - I always like to help others, if I didn't I wouldn't be putting all this helpful technical information on the web on my blog would I?

Just remember I have a full time job, a few companies I am partners in or run on my own, contract work and then when I get spare time I do a bit of coding for the Open Source community. 

However if you know my feelings on Open Source you will know I'm not exactly a fan.

Here is a direct quote from article venting my spleen on the topic.

If you can't code and have put a stupid $50 bid on Rentacoder.com to develop a whole website by next week and expect me to do all your work for you just because you found a piece of my code that does 90% of what you need and you require the other 10% doing for free then you can take a running jump.

People like you are the reason good coders work 50 hour weeks for tech companies and then spent all their free time trying to come up with something that might make some money if the world of programming actually behaved like any other kind of market place.

From the point of view of actually making real money I cannot think of a worse idea for anyone than open source coding. Off the top of my head I cannot think of any other business model that acts in a similar way and leaves the person who produced the goods with bog all and the customer with all their hearts desire at a cost of zilch!

. 

So there you go basically if you want good quality products pay for them.

Otherwise if you get your code for free from unknown sources don't be surprised if they infect your computer with viruses, steal your bank account details, use your mailbox as spam and put crap code into your system.

Googlebot, Sitemaps and heavy crawling

Googlebot over-crawling a site

I recently had an issue with one of my jobboards that meant that Googlebot was over-crawling the site which was causing the following problems:

1. Heavy loads on the server. The site in question was recording 5 million page loads a month which had doubled from 2.4 million within a month.
2. 97% of all their traffic was accounted by Googlebot.
3. The site is on a shared server so this heavy load was causing very high CPU and affecting other sites.

The reasons the site was receiving so much traffic boiled down to the following points.

1. The site has a large number of categories which users can filter job searches by. These categories are displayed in whole and in subsets in prominent places such as quick links and a job browser which allows users to filter results. As multiple categories can be chosen when filtering a search this meant Googlebot was crawling every possible combination in various orders.
2. A new link had been added within the last month to the footer which passed a sessionID in the URL. The link was to log whether users had Javascript enabled. As Googlebot doesn't keep session state or use Javascript it meant the number of crawled URLs actively doubled as each page the crawler hit would find a new link that it hadn't already spidered due to the new SessionID.
3. The sitemap had been setup incorrectly containing URLs that didn't need crawling as well as incorrect change frequencies.
4. The crawl rate was set to a very high level in Webmaster tools.

Therefore a site with around a thousand jobs was receiving 200,000 page loads a day nearly all of them from crawlers. To put this in some perspective other sites with 3000+ jobs, good SEO and high PR usually get around 20,000 page loads a day from crawlers.

One of the ways I rectified this situation was by changing the crawl rate to a low custom crawl rate of 0.2 crawls per second. This caused a nice big vertical drop in the graph and it alarmed the site owner as he didn't realise that there is no relation between the amount of pages crawled by Google and the sites page ranking or overall search engine optimisation.

Top Tips for getting the best out of crawlers

• Setup a sitemap and submit it to Google, Yahoo and Live.
• Make sure only relevant URLs are put in the sitemap. For example don't include pages such as error pages and logoff pages.
• If you are rewriting URLs then don't include the non-rewritten URL as well as this will be counted as duplicate content.
• If you are including URLs that take IDs as parameters to display database content then make sure you don't include the URL without a valid ID. Taking the site I spoke about earlier as an example, someone had included the following

www.some-site.com/jobview.asp

instead of

www.some-site.com/jobview.asp?jobid=35056

This meant crawlers were accessing pages without content and it was a pretty pointless and careless thing to do.

• Make sure the change frequency value is set appropriately. For example on a jobboard when a job is posted its usually posted for between 7 and 28 days. It only needs to be crawled between once a week and once a month depending on the days it was advertised for. It does not need to be crawled every time so setting a value of always is inappropriate as the content will not change every time Googlebot accesses the URL.
• Avoid circular references such as placing links to a site-index or category listings index in the footer of each page on a site. It makes it hard for the bot to determine the site structure as every path it drills down its able to find the parent page again. Although I suspect the bots technology is clever enough to realise its found a link already spidered and not crawl it again I have heard that it looks bad in terms of site structure.
• Avoid dead links or links that lead to pages with no content. If you have a category index page and some categories have no content related to them then don't make the category into a link or otherwise link to a page that can show related content rather than nothing.
• Prevent duplicate content and variations of the same URL being indexed by implementing one of the following two methods.

1. Set your Robots.txt to disallow your non URL rewritten pages from being crawled and then only display rewritten URLS to agents identified as crawlers.
2. Allow both forms of URL to be crawled but use a CANONICAL META tag to specify that you want the rewritten version to be indexed.

• Ban crawlers who misbehave. If we don't spank them when they are naughty they will never learn so punish those that misbehave. Its very easy for an automated process to parse a Robots.txt file therefore there is no excuse for those bots that ignore the commands set out in it. If you want to know those bots who ignore the Robots.txt rules then there are various ways such as parsing your webserver log files or using a dynamic Robots.txt file to record those agents that access it. There are other ways such as using the IsBanned flag available in the Browscap.ini file however this relies on the user-agent being correct and more and more people spoof their agent nowadays. Not only is banning bots good for your servers performance as it reduces load its good for your sites security as bots that ignore the Robots.txt rules are more likely to hack, spam, and scrape your sites content.

If you are having similar issues with over-crawling then I would advise you to first check your sites structure to see if the problem is due to bad structure, invalid sitemap values and over categorisation first before changing the crawl rate. Remember a sites SEO is unrelated to the amount of crawler activity and more is not necessarily better. Its not the number of crawled pages that counts but rather the quality of the content that is found when the crawlers visit that matters.