Twitter Rush caused by Tweet BOTs visiting your site

Twitter Traffic can cause a major hit on your server

If you are using Twitter to post tweets to whenever you blog or post an article you should know that a large number of BOTS will immediately hit your site as soon as they see the link.

This is what I call a Twitter Rush as you are causing a rush of traffic from posting a link on Twitter.

I did a test some months back and I like to regularly test how many hits I get whenever I post a link so that I can weed out the chaff from the wheat and set up rules to ban any BOTS I think are just wasting me money.

Most of these BOTS are also stupid.

If you post the same link to multiple Twitter accounts e.g by using my Wordpress plugin - Strictly Tweetbot then the same BOT will come to the same page multiple times.

Why? Who wrote such crap code and why don't they check before hitting a site that they haven't just crawled that link. I cannot believe the developers at Yahoo cannot write a BOT that works out they have just crawled a page before doing it two more times.

Some BOTS are obviously needed such as the major SERP search engines e.g Google or Bing but many are start up "social media" search engines and other such content scrapers, spammers. hackbots and bandwidth wasters.

Because of this I now 403 a lot of these BOTS or even send them back to the IP address they came from with an ISAPI rewrite rule as they don't provide me with any benefit and just steal bandwidth and cost me money.

RewriteCond %{HTTP_USER_AGENT} (?:Spider|MJ12bot|seomax|atomic|collect|e?mail|magnet|reaper|tools\.ua\.random|siphon|sweeper|harvest|(?:microsoft\surl\scontrol)|wolf) [NC]
RewriteRule .* http://%{REMOTE_ADDR} [L,R=301]

However if you are using my Strictly Tweet BOT plugin which can post multiple tweets to the same or multiple accounts then the new version allows you to pre-post a page which hopefully gets cached by the caching plugin you should be using (WP SUPER CACHE or W3 TOTAL CACHE etc) before the article is made public and the BOT looking at Twitter for URL's to scrape can get to it.

The aim is to get the page cached BEFORE multiple occurrences of BOTS hit the page. If the page is already cached then the load on your server should be a lot less than if every BOT's loading of the page was trying to cache the page at the same time (due to the quickness of their visit). 

However if you are auto blogging and using my TweetBOT you might be interested in Strictly TweetBOT PRO as it had extra features for people who are tweeting to multiple accounts or multiple tweet in different formats to the same account. These new features are all designed to reduce the hit from a Twitter Rush.

The paid for version allows you to do the following:

• Make an HTTP request to the new post before Tweeting anything. If you have a caching plugin on your site then this should put the new post into the cache so that when the Twitter Rush comes they all hit a cached page and not a dynamically created one.
• Add a query-string to the URL of the new post when making an HTTP request to aid caching. Some plugins like WP Super Cache allow you to force an uncached page to be loaded with a querystring. So this will enable the new page to be loaded and re-cached.
• Delay tweeting for N seconds after making the HTTP request to cache your post. This will help you ensure that the post is in the cache before the Twitter Rush.
• Add a delay between each Tweet that is sent out. If you are tweeting to multiple accounts you will cause multiple Twitter Rushes. Therefore staggering the hits aids performance.

I did a test this morning to see how much traffic was generated by a test post. I got almost 50 responses within 2 seconds!

50.18.132.28 - - [30/Nov/2011:07:04:41 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 471 "-" "bitlybot"
50.57.137.74 - - [30/Nov/2011:07:04:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "EventMachine HttpClient"
50.57.137.74 - - [30/Nov/2011:07:04:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "EventMachine HttpClient"
184.72.47.46 - - [30/Nov/2011:07:04:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
204.236.150.14 - - [30/Nov/2011:07:04:44 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 403 471 "-" "JS-Kit URL Resolver, http://js-kit.com/"
50.18.121.55 - - [30/Nov/2011:07:04:45 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
184.72.47.71 - - [30/Nov/2011:07:04:47 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
50.18.121.55 - - [30/Nov/2011:07:04:48 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
184.72.47.71 - - [30/Nov/2011:07:05:11 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
199.59.149.31 - - [30/Nov/2011:07:04:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "Twitterbot/0.1"
107.20.160.159 - - [30/Nov/2011:07:04:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "http://unshort.me/about.html"
46.20.47.43 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 403 369 "-" "Mozilla/5.0 (compatible"
199.59.149.165 - - [30/Nov/2011:07:04:43 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28862 "-" "Twitterbot/1.0"
173.192.79.101 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 403 471 "-" "-"
50.18.121.55 - - [30/Nov/2011:07:05:11 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
46.20.47.43 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 403 369 "-" "Mozilla/5.0 (compatible"
199.59.149.31 - - [30/Nov/2011:07:05:11 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "Twitterbot/0.1"
65.52.0.229 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28863 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
66.228.54.132 - - [30/Nov/2011:07:04:43 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 106183 "-" "InAGist URL Resolver (http://inagist.com)"
199.59.149.165 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28863 "-" "Twitterbot/1.0"
107.20.42.241 - - [30/Nov/2011:07:05:11 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "PostRank/2.0 (postrank.com)"
65.52.0.229 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28862 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
65.52.0.229 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28862 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
110.169.128.180 - - [30/Nov/2011:07:05:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28862 "http://twitter.com/" "User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.18.1"
74.112.131.128 - - [30/Nov/2011:07:05:15 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
74.112.131.131 - - [30/Nov/2011:07:05:16 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
74.112.131.127 - - [30/Nov/2011:07:05:16 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
74.112.131.128 - - [30/Nov/2011:07:05:16 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
74.112.131.131 - - [30/Nov/2011:07:05:17 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
74.112.131.128 - - [30/Nov/2011:07:05:17 +0000] "GET /some-test-url-I-posted/ HTTP/1.0" 200 106203 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
107.20.78.114 - - [30/Nov/2011:07:05:18 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
65.52.54.253 - - [30/Nov/2011:07:05:19 +0000] "GET /2011/11/hypocrisy-rush-drug-test-welfare-benefit-recipients HTTP/1.1" 403 470 "-" "-"
107.20.78.114 - - [30/Nov/2011:07:05:28 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
65.52.62.87 - - [30/Nov/2011:07:05:30 +0000] "GET /2011/11/hypocrisy-rush-drug-test-welfare-benefit-recipients HTTP/1.1" 403 470 "-" "-"
107.20.78.114 - - [30/Nov/2011:07:05:43 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
107.20.78.114 - - [30/Nov/2011:07:06:15 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
199.59.149.165 - - [30/Nov/2011:07:04:45 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28860 "-" "Twitterbot/1.0"
107.20.160.159 - - [30/Nov/2011:07:04:59 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "http://unshort.me/about.html"
107.20.78.114 - - [30/Nov/2011:07:06:15 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
199.59.149.31 - - [30/Nov/2011:07:04:46 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "Twitterbot/0.1"
107.20.42.241 - - [30/Nov/2011:07:05:01 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "PostRank/2.0 (postrank.com)"
107.20.42.241 - - [30/Nov/2011:07:05:07 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "PostRank/2.0 (postrank.com)"
107.20.78.114 - - [30/Nov/2011:07:06:15 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
107.20.78.114 - - [30/Nov/2011:07:06:17 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
107.20.78.114 - - [30/Nov/2011:07:06:17 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 403 - "-" "MetaURI API/2.0 +metauri.com"
50.16.51.20 - - [30/Nov/2011:07:06:21 +0000] "HEAD /some-test-url-I-posted/ HTTP/1.1" 200 - "-" "Summify (Summify/1.0.1; +http://summify.com)"
74.97.60.113 - - [30/Nov/2011:07:07:11 +0000] "GET /some-test-url-I-posted/ HTTP/1.1" 200 28889 "-" "Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0"

Introducing Strictly TweetBOT PRO

Introducing Strictly TweetBOT PRO

By Strictly-Software

Today I created a premium version of my popular WordPress plugin Strictly TweetBOT.

This is the plugin I use on ALL my sites to automatically send out Tweets to various Twitter accounts related on the content of the article.

For example on my horse racing site UK Horse Racing Tipster I set up multiple TweetBOT Accounts that are linked to various Twitter Accounts e.g @ukhorseracetips and @ukautobot and when an article is posted the system uses the words in each accounts content analysis box to decide whether to send the Tweet out or not.

You can choose to
-Always send the Tweet.
-Only send it if one or more words are in the article.
-Only send it if ALL of the words are in the article.
-Don't send the Tweet if any of the words are in the article.

You can also prevent Tweets being sent if they contain "noise" words. This can be a list of swear words or other words you would never want tweeted.

Linking your accounts with Twitter is easy and using the new OAuth Twitter 1.1 API.

To link an account you set it up, save it, then when the page reloads a link will appear saying "Click Here To Authenticate", when you do you will be taken to Twitter where you login and are given a PIN number.

You go back to the plugin admin page and enter the PIN number in the relevant box. Hit save at the bottom and if the details are correct your account will now be linked to Twitter.

If you have already linked a Twitter account then adding another is easy as it will remember your credentials and authenticate it automatically.

The Admin dashboard will contain information on your last posted tweets and there is a "Test Configuration" button that will test all your accounts, links with Twitter and any problems you may have. Excellent for checking your plugin is set-up correctly.

The best thing about this plugin is you can use the articles post tags as #HashTags and with its automatic configuration with Strictly AutoTags it will wait until the tagging is complete before Tweeting rather than Tweeting on publish which would mean some articles wouldn't have tags at that point.

This is possible because of the new finished_doing_tagging hook in Strictly AutoTags which is fired once Tagging is complete. Other plugin authors are free to use this hook as well to ensure actions are only carried out AFTER tagging and not on the "onsave" action/event.


The PRO version of the plugin is only £25 and contains some very useful features especially if you are sending out a lot of Tweets or have an underpowered server. 

It can be bought on my site: Strictly TweetBOT WordPress plugin
Or on Etsy.com in my shop: The Strictly-Software shop.

The PRO version extends the performance features including those which enable you to send an HTTP request to the new article to put it into a cache (depending on the settings of your caching system e.g WP Super Cache, W3 Total Cache) and ensure when the first visitor hits it they get a cached version of the page.

In the PRO version you can do the following:

-Add a delay of X seconds so that after the HTTP request is made no Tweets are sent for a while. This ensures the caching system has time to build and cache the new post.
-Add a query-string to the URL as some caching plugins require special keys or query-string parameters to force the page to be cached.
-Use a special user-agent when making the cache HTTP request. This enables you to track the requests in your Apache Access Log as well as white-listing the user-agent to prevent it from being blocked by .htaccess rules.
-Specify a delay between each Tweet that is sent out from your site. This ensures not too many Tweets hit your account at once and not too much Twitter traffic hits your website all at once. Staggering out your Tweets prevents Twitter Rushes and stops too many new Apache processes from being spawned.

Remember when one Tweet is sent out you will get at least 50 requests to the new post immediately from BOTS who have found the link on Twitter.

This is called a Twitter Rush and can cause major performance problems especially if you are sending out 6+ tweets (6 * 50 = 300 concurrent requests). You can read more about Twitter Rushes here.

Any messages back from Twitter such as posting duplicate Tweets or API errors will be saved and listed in your dashboard so that you can view the status of your last post and all the accounts it tried to tweet to. It will tell you whether the account matched the content analysis or was blocked and the result of any "sleep" or "caching" actions.

The PRO version also allows you to:

-Specify the maximum length a Tweet message can ever have e.g 137. This will ensure all Tweets are sent as sometimes when Tweets are exactly 140 characters long they still get rejected by Twitter.

-Turn off the link between Strictly AutoTags and Strictly TweetBot. If you are having issues with the combination OR just want Tweets to be fired ASAP on publishing whether or not tagging is complete you could turn this option on.

Reasons might be debugging or the use of categories or default hashtags instead of using post tags in your Tweets. If you don't need to use the tags against a post then there is no need to wait in the first place so this option can be disabled.


AutoBlogging - Automatic Posting On WordPress

Combining two of my plugins, Strictly TweetBOT and Strictly AutoTags, make a powerful force for any auto-bloggers out there.

If you are posting your articles by importing them from a feed then these plugins make your site a diamond in the dust.

Not only do you have do nothing but when your feed importer inserts a new article the Strictly AutoTags plugin will do it's job making your article a custom piece of work by:
-Basic content spinning by removing old HTML formatting as B I, FONT and SPAN tags
-Wrapping important tagged content in strong tags to highlight them and tell SERPS like GoogleBOT and YSLURP they are the important words in your article.
-Converting textual links like www.ukhorseracingtipster.com into real clickable links e.g www.ukhorseracingtipster.com.
-Converting a certain number of important tags into links to their relevant tag page.
-Adding rel="nofollow" to existing links in your article that don't already have them as well as those converted to links

Plus the SEO benefits are amazing (as my sites can testify) as the Premium version of the plugin allows you to find certain words and tag others. For example you could find words like al-Qaeda, ISIS, Taliban but add the tag Terrorism. You can add as many of these "Tag Equivalents" as you want.

Also for your "Site Keywords", the words you want associated with your site, for example for my racing site it would be Horse Racing, Racing, Tips, Betting etc, then you can set them to be ranked higher than any other words when it comes to relevancy.

And of course you can specify that words in the Title, H1-6 tags, Anchors, Strong Tags and other content be ranked higher than words outside special format tags.

So the AutoTag plugin does it's magic (read more here) and then once it has completed and you haven't turned the link off between the two plugin the TweetBOT goes to work sending Tweets out to all relevant Twitter accounts.

The Strictly TweetBOT PRO Version is even better than normal because it allows you to stagger your tweeting so they all don't get blasted at once causing a Twitter Rush and because you can format each account differently you could have two accounts both going to the same Twitter Account but with totally different messages.

Plus you can use different tweet shortening functions for each e.g Tweet Shrink or Text Shrink.

For example if the article was about a horse race at Royal Ascot titled "Kingman destroys the pack at Royal Ascot" and the first account was using categories (Ascot, Racing News Horse Racing) as hash tags with the format:

Another Ascot news story %title% which you can read about here %url% %hashtags%

You would get:

Another #Ascot news story Kingman destroys the pack at Royal Ascot which u can rd about here bit.ly/34f53 #RacingNews #HorseRacing

And another account that uses post tags (Kingman, Night of Thunder, Royal Ascot, Ascot and many more) as #hashtags with the format:

Racing news from UK Horse Racing Tipster %url% %title% %hashtags%

Would give you this tweet.

Racing news from UK Horse Racing Tipster bit.ly/34f53 #Kingman destroys the pack at Royal #Ascot #RoyalAscot #NightofThunder #JohnGosden

Putting a time delay of 20 seconds in between each tweet would mean that the first tweet would get sent to your account causing 50+ BOTS and visitors to come and hit your new page (a Twitter Rush) and then 20 seconds later another tweet would be sent causing another lot of visitors. Staggering the tweets prevents Twitter Rushes.

Another new feature is the ability to send an HTTP GET request to your new post hopefully causing it to be cached by WP Super Cache / W3 Total Cache or any other caching plugin.

You have the ability to add another time delay after this request to give the caching plugin time to build the file as well as passing special parameters in the querystring (such as special keys or values required by caching plugins to force a cache).

You can also use a special user-agent to make this request so that you can prevent the request being blocked (for example if you ban blank user-agents like I do) or to identify the request in your log files.

If you don't want to use post tags as #HashTags you can use the articles categories or even specify default hash tags. The system will take the list of possible tags and then order them by size and try to fit as many into the Tweet as possible.

It will now even try and scan the title of the tweet looking for tags contained within the tweet itself and add a # in front of the word as this saves room and allows more hash tags to be used in your %hashtag% parameter.

Buy Strictly TweetBOT PRO

So if you want to aid your WordPress sites performance consider buying the Strictly TweetBOT plugin from my site or Etsy.com or if you download the free version please consider donating some money.

If you want to enhance your auto-blogging capabilities then combining the premium versions of both Strictly TweetBOT PRO and Strictly AutoTags Premium Version will do your site wonders as well as saving you time!

If everyone donated just a single pound for every plugin they had download from WordPress then I would have made hundreds of thousands of pounds by now and could spend my time developing plugins full time AND FOR FREE!

You can also buy a voucher from Etsy.com for £15 that will let you hire me to set the plugin up for you if you are having difficulties or are not a WordPress or Twitter expert. All I would need is access to your admin area to do the work.

Also if you haven't checked lately Strictly AutoTags has a premium version which you can buy for £40 and has LOTS more features than the free WordPress version. It also can be bought with a set-up voucher that you can purchase from Etsy.com.

Also if you could please visit my facebook.com/strictlysoftware fan page and "like" it if you could. Leave comments and pass it onto your friends and fellow WordPress developers.

Latest Version of Strictly AutoTags 2.8.9 - Donation Only

Latest Version of Strictly AutoTags 2.8.9 - Paid Version

By Strictly-Software

I have just released the latest version of my highly popular plugin Strictly AutoTags plugin.

Strictly AutoTags 2.8.9 is a paid for version which means it has features in it that are only available to those people who pay me £40 first.

The new paid for version includes the following features:at

Strictly AutoTags - Version 2.8.9

• Updated storage to handle new data-description and data-pin-desc attributes so they don't get tagged inside images by mistake. The same goes for shortcodes now handled by the Jetpack plugin so that [youtube http://www.youtube.com/watch?v=USbkB6rVbpc] doesn't get accidentally tagged either if one of the words within the [shortcode] contained a word e.g [customField sales]
• I have added a new tag equivalent mark up language using a simple method that enables you to match instances of certain words BUT use a different tag. E.G if you want the words Snowden, NSA and Prism to add the tag Police State to your article you would use this markup language [Snowden, NSA, PRISM]=[Police State] 
• To then add the tag terrorism to the words al-Qaeda, bin-Laden, 9.11, WTC and Taliban you would add this [al-Qaeda, bin-Laden, 9.11, WTC, Taliban]=[Terrorism] To put them together you use a pipe so adding these two would mean an input box filled like this [Snowden, NSA, PRISM]=[Police State]|[al-Qaeda, bin-Laden, 9.11, WTC, Taliban]=[Terrorism]
• I've added an option to set the minimum number of letters a tag must have before it is used as a tag. This applies to stored tags or newly found ones. This allows you to skip tags of one or two letters long even if they are ACRONYMS and you don't want to add them to the NOISE word list.
• I've added option to tell system whether or not to convert plain text links like the wording www.msnbc.com into a proper link msnbc.com This is obviously quite difficult due to all the shortened links you can now get, the lack of protocols e.g //twitter.com/strictlytweets and soon UTF8 characters in URLs.
• I've created a new name function to match names like al-Qaeda or al-Nusra Front or even words with commas in like 1,000 Guineas which would be tagged as 1000 Guineas due to the comma being used as a separator in WordPress.
• I've added a new finished_doing_tagging hook into the plugin so other plugins can act once the tagging is finished. This is most useful when using my AutoTag plugin in conjunction with Strictly TweetBot which allows you to use categories and tags as #hashtags in any tweets that are automatically sent out. Without this hook you can run into problems when you have a large number of tags to search through and it's possible a tweet will get sent with their default #HashTags instead of relevant post tags instead like #Snowden or #NSA on an article about the Police State.
• In my latest paid version I have included a new function which will allow you clean out any HTML added by my own tagging process when it bolds and deep-links tags. I use specific classes on my HTML for anchors and spans so I know how to remove them with regular expressions. Obviously the more articles you have the longer this process to clean up will take. You could obviously clean articles out yourself with a simple UPDATE MySQL statement and a RegExp Replace function call but this saves you from having to.

There is a free version up on the Wordpress site, the latest being version 2.8.8, but this has far less features.

When I bring out new paid only versions I may add a few of the older paid for features into the free versions.

However the whole reason I am doing this is, is due to the fact that although over 180,000 people have downloaded my plugin and obviously think it's useful from Wordpress (and who knows how many from my own website), I have had hardly any donations.

It seems hardly worth my time putting the code out on the web for others when I can just use it myself and gain all the SEO benefits it has brought me (which it has!).

Benefits such as:

• Deep-linking a specified number of the most important words in an article to their "tag listings" page where crawlers and users can find lots more content related to that word / tag.
• Also the deep-linking is done on INPUT not OUTPUT. Which means it's faster to load pages as the formatting is already done. Do something once on INPUT not a THOUSAND times on OUTPUT, especially not when thousands of BOTS and HACKERS come crawling using up CPU and bandwidth!
• Hit Highlighting, bolding or using links, around important words that are key and related to an article is a good way of telling SERP crawlers which words are the most important for an article.Automatically doing this without waiting for 3rd party API's to add new tags relevant to the latest news stories (Peoples names, buildings, companies etc) is a lot faster.

The money I have been donated so far by a few people, usually for adding new features to the plugin has certainly not enough been enough money to warrant me spending the amount of time and effort I have on this plugin so so far - and that's A LOT! This is why I have brought out a paid version as the main version which will always have the latest features in it.

Believe it or not a £2-£5 donation maybe a small sign of gratitude but as this is is proper coding and "Open Source", I thought the idea was to take other peoples code and "change", "edit" or "extend" it. Maybe I was wrong but that's what I did with my first sitemap plugin when all the others were giving me problems. I literally learnt Wordpress AND PHP at the same time just so I could make a fast performing plugin that worked with 50,000+ articles.

Also as I get paid £650 a day contracting a fiver just isn't worth a days work I'm afraid.

Also let me remind those people who still cannot read English:

• I don't work for free.
• I don't add new features into an open source plugin just because you want them. It takes time and effort!
• I don't drop everything to help you configure a plugin which has a simple "Test Config" button in the admin panel that tests everything is set up for you and if not tells you what the problems are and what you need to do to fix them. I do this to save support requests believe it or not.
• I also have a "shit list" for tight wads who have stitched me up in the past. Either by promising me constant work, large payments later on, or payments after the work has been done. If they fail to live up to their side of the agreement or are just plain rude they get their name and comment / email put on my very own "Shit List". You can read some of them here: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html.

The latest is quite funny, not just because of the broken English but because the person seemed to want me to drop everything and fix something that isn't even broken ASAP!

Not only do I not even support or release new versions of the plugin in question "Strictly Google Sitemap" but the problem the person is talking about is a simple case of him not hitting the "Save Settings" button first before hitting the "Build Sitemap" button - DOH!

Have a laugh as you read (his partially corrected by me) broken English request trying to get me to help him out.

hi dear rob i hope you are fine

i have a very bad probelm
my site hosting has limit and i can't build full sitemap with google xml sitemap by arne

i installed your plugin i see very bad problem
when i installed i went in setting of your sitemap
i changed setting when i clicked in manual building sitemap, 
sitemap build but with default settings

dear rob i don't have time to waste!

If only I could build in a Doofus Config Test button to check the user has basic computer skills first before allowing them to use it that would be a money spinner!

I do like the "dear rob I don't have time to waste!" bit at the end. In the real email he left the end "e" of the word waste but I thought I would add it on to help you understand him more easily.

Yes of course, you have no time to waste so I will just drop everything and fix something for your for free. I think not!

This "Tight Wad" list has worked quite well so far and I have had quite a few people contact me and pay me just to take their name off it.

I suppose they have done a Google search for their own name when they were bored one day only to find themselves on my blog in an article about tight arses who talk the talk but can't walk the walk.

Oh well maybe they should have read the specific part of the FAQ page on the WordPress plugin page that states how they should go about debugging any problem and to prevent themselves ending up on my "tightwad list". 

It's quite simple and in the FAQ section.. From the WordPress sites FAQ Section non the website about the plugin.

I have an error.

If you have any error messages installing the plugin then please try the following to rule out conflicts with other plugins:

• Disable all other plugins and then try to re-activate the Strictly Google Sitemap plugin as some caching plugins can cause issues.

• If that worked, re-enable the plugins one by one to find the plugin causing the problem. Decide which plugin you want to use.

• If that didn't work check you have the latest version of the plugin software (from WordPress) and the latest version of WordPress installed

• Check you have JavaScript and Cookies enabled.

• If you can code turn on the DEBUG constant and debug the code to find the problem otherwise contact me and offer me some money to fix the issue :)

• Please remember that you get what you pay for so you cannot expect 24 hour support for a free product. Please bear that in mind if you decide to email me a donation button is on my site and on the plugin admin page.

• If you must email me and haven't chosen to donate even the smallest amount of money please read this: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html.

• If you don't want to pay for support then ask a question on the message board and hop

Or maybe they should have read the "Support" page that clearly states I don't support this Sitemap plugin any-more. The main reason being that I got Cancer at the beginning of 2012.

              

Everything moves so fast it is hard to keep up and build business models on top of 3rd party code - just look at Twitter and their culling of all the DM message responder applications that put a myriad of companies out of business.

However despite these other SEO features not working the core Sitemap functionality still works (on non MU sites) and I use it on all of my own sites.

However if you want support you must donate. Otherwise you go onto the list: http://blog.strictly-software.com/2011/10/naming-and-shaming-of-programming.html

I just haven't got time to answer questions in broken English that make so sense as if I "must" help them for some reason. It's 3rd party, open source and free code - sort it out yourself or pay someone to do it for you! Simples!

So there you go a brand new donation only version of AutoTags and some reasons why you shouldn't piss me off :)

If your nice I help - I always like to help others, if I didn't I wouldn't be putting all this helpful technical information on the web on my blog would I?

Just remember I have a full time job, a few companies I am partners in or run on my own, contract work and then when I get spare time I do a bit of coding for the Open Source community. 

However if you know my feelings on Open Source you will know I'm not exactly a fan.

Here is a direct quote from article venting my spleen on the topic.

If you can't code and have put a stupid $50 bid on Rentacoder.com to develop a whole website by next week and expect me to do all your work for you just because you found a piece of my code that does 90% of what you need and you require the other 10% doing for free then you can take a running jump.

People like you are the reason good coders work 50 hour weeks for tech companies and then spent all their free time trying to come up with something that might make some money if the world of programming actually behaved like any other kind of market place.

From the point of view of actually making real money I cannot think of a worse idea for anyone than open source coding. Off the top of my head I cannot think of any other business model that acts in a similar way and leaves the person who produced the goods with bog all and the customer with all their hearts desire at a cost of zilch!

. 

So there you go basically if you want good quality products pay for them.

Otherwise if you get your code for free from unknown sources don't be surprised if they infect your computer with viruses, steal your bank account details, use your mailbox as spam and put crap code into your system.

New Version of Strictly AutoTags - Version 2.8.6

Strictly Auto Tags 2.8.6 Has Been Released!

Due to the severe lack of donations plus too many broken promises of "I'll pay if you just fix or add this" I am stopping to support the Strictly Auto Tags plugin.

The last free version, 2.8.5 is up on the WordPress repository: wordpress.org/plugins/strictly-autotags

It fixes a number of bugs and adds some new features such as:

1. Updated storage array to store content between important content, bold, strong, headers and links etc. So they don't get tagged inside e.g put bolded words inside an existing h4 etc. 
2. Changed storage array to run "RETURN" twice to handle nested code because of previous change.
3. Fixed bug that wasn't showing the correct value for the minimum number of tags that a post must have before deeplinking to their tag page in admin. 
4. Fixed bug in admin to allow noise words to have dots in them e.g for links like youtube.com 
5. Added more default noise words to the list.
6. Cleaned code that wasn't needed any-more due to changes with the way I handle href/src/title/alt attributes to prevent nested tagging.
7. Removed unnecessary regular expressions which are not needed now. 

Version 2.8.5 of Strictly AutoTags www.strictly-software.com/plugins/strictly-auto-tags

Is going to be a "donate £40+" and get a copy version.

I am going to be sexing this plugin up into more of an SEO, text spinning, content cleaning, auto-blogging tool full of sex and violence in the future and I am running it on my own sites at the moment to see how well it does.

New features in 2.8.6 include.

Set a minimum length of characters for a tag to be used.

Set equivalent words to be used as tags. I have devised a "mark up" code for doing this which will allow you to add as many tag equivalents as you want. For example this is a cut down current version from one of my sites to show you an example using Edward Snowden (very topical at the moment!).

[NSA,Snowden,Prism,GCHQ]=[Police State]|[Snowden,Prism]=[Edward Snowden]|[Prism,XKeyscore,NSA Spying,NSA Internet surveillance]=[Internet Surveillance]|[TRAPWIRE,GCHQ,NSA Spying,Internet surveillance,XKeyscore,PRISM]=[Privacy]|[Snowden,Julian Assange,Bradley Manning,Sibel Edmonds,Thomas Drake]=[Whistleblower]

As you can see from that example you can use the same words multiple times and give them equivalent tags to use. So if the word Snowden appears a lot I will also tag the word "Police State", "Edward Snowden", "Whistleblower" as well as Snowden.

This feature is designed so that you can use related words as tags that maybe more relevant to peoples searches. 

I also have added a feature to convert textual links that may appear from importing or scraping into real links for example www.strictly-software.com will become a real link to that domain e.g http://www.strictly-software.com.

I have added the new attributes data and their derivatives e.g data-description or data-image-description , basically anything that has data- at the front of it, into my list of attributes to store and then replace after auto-tagging to prevent nested tags being added inside them.

I will be extending this plugin lots in the future but only people prepared to pay for it will be able to get the goodies. I am so fed up of open-source coding there is little point in me carrying on working my ass off for free for other peoples benefit any more.

If you want a copy email me and then I will respond.

You can then donate me the money and I will send you a unique copy.

Any re-distribution of the code will mean hacks, DDOS, viruses from hell and Trojans coming out your ass for years!

Regular Expression functions for reformatting content for Wordpress

Using RegEx to reformat Wordpress content

If you read my blog you will know I am "Mr Automate" and I specialise in scraping, reformatting, spinning and automatically posting content to Wordpress, Twitter and other sites.

I use Wordpress for a couple of my blogs and because the well known plugin WP-o-Matic has stopped being supported and the only replacements are paid for plugins like WP Robot I have basically written my own custom version to handle my automated content imports.

A couple of useful Regular Expressions you might like to use which I find particular helpful are below.

I am using PHP as that is the language Wordpress is written in but the core Regular Expressions should be easily transferable between languages.

1. Removing IFRAMES

As YouTube has moved from it's old OBJECT / EMBED nested tag soup to an IFRAME many feeds containing videos will now contain IFRAME's which need to be handled correctly as we all know that viruses can be transfered through rouge IFRAME content.

If you are just importing content from a feed without parsing and checking it then just imagine if for whatever reason the feed you are using one day contains an IFRAME with a SRC pointing to a dodgy virus infected URL. If you don't sanitize the content you will be inserting this dangerous content into your own blog for your visitors to become infected by.

Although most modern browsers and virus checkers are good at picking up on potential malicious URL access it is always wise to run your own checks using a white list of "allowed" domains rather than a "black list" of banned ones.

This code is a basic example of looping through all the IFRAMES on a page, checking their SRC tag for a known white list of acceptable domains and then replacing any IFRAME's that don't match.

// match all XHTML IFRAME tags where $content holds your HTML
$videocount = preg_match_all("@(<iframe[\s\S]+?<\/iframe>)@",$content,$videos,PREG_SET_ORDER);

foreach($videos as $video){

 $object = $video[1];

 // only allow certain domains e.g youtube, dailymotion, vimeo, cnn, fox, msnbc and the bbc
 if(!preg_match("@src=['\"]http:\/\/(www\.)?(?:youtube|dailymotion|vimeo|cnn|fox|msnbc|bbc)\.@i",$object)){
  
  // replace the IFRAME as we don't know where it is pointing to

  $content = preg_replace("@" . preg_quote($object) . "@","",$content);
 }
 
}

2. Re-sizing images

I have already written an article about how you can use regular expressions and a callback function to reformat videos or images so that they fit your template and this follows that theme.

PHP like Javascript and other decent languages offers the ability to create anonymous (unnamed) functions on the fly to be used as callbacks in regular expression replace functions.

Passing a function as a parameter within another function is known as a lambda function and this is very useful in certain situations such as parsing HTML where you want to remove illegal tags and attributes.

This example is pretty niche to my own needs but the idea is useful and can easily be converted to your own reformatting needs.

Basically I show little Google Adsense box adverts at the top right of all my articles and as these are floated to the right I don't want any images within my articles that are less than the size of my content area (minus the size of the advert) to also be aligned to the right as this causes a horrible looking mess.

Therefore I use this code to ensure that the Wordpress class "alignleft" is always inserted on the first image in any article but only if the width of the image is less than 350px. It doesn't matter if the image isn't right at the top of the article (although they usually are) but it helps to ensure a smooth flow and layout.

The Preg_Replace function in PHP is nice in that you can limit the number of replacements to one or more if you so wish and in this case I only want to reformat the first image. You will see that the lambda function I use also contains multiple other regular expressions and there is nothing stopping you from writing complex code within these anonymous functions.

$content = preg_replace_callback("@(<img[^>]+?class=['\"])([\s\S]*?)(['\"][\s\S]*?>)@i",
  create_function(
  '$matches',
  'preg_match("@width=[\'\"](\d+)[\'\"]@",$matches[0],$widthmatch);
  $chk=true;
  if($widthmatch){       
   if(is_numeric($widthmatch[1])){
    if($widthmatch[1] >= 350){         
     $chk = false;
    }       
   }
  }
  if($chk && !preg_match("@alignleft@",$matches[2])){
   $res = $matches[1] . trim( preg_replace("@alignright@","",$matches[2]) . " alignleft") .$matches[3];
  }else{
   $res = $matches[1] . preg_replace("@alignright@","",$matches[2]) .$matches[3];
  }
  return $res;'),$content,1);

As you can see the lambda anonymous function carries out the following logic

• It looks for the first image that has a class attribute on it and passes the match collection into the anonymous function.
• It then checks the image for a width attribute using another regular expression.
• If found it then checks that the value of the width attribute is numeric and if so it tests to see if the value is over my allowed limit of 350px setting a flag if so.
• I then use another regular expression to test for the existence of the standard Wordpress class "alignleft" and if it isn't found then I add it in making sure to replace any "alignright" class if it exists.
• The lambda function then returns the new reformatted content of the image which is then used as the replacement value in the Preg_Replace function.

Just two useful examples which can be expanded upon by anyone interested in Regular Expressions and AutoBlogging.

Remember if you require any custom plugin work for Wordpress, including special reformatting functions, scraping code or content automation then contact me for a quote.