Thursday, 6 October 2011

Twitter Hash Tag Scanner SEO Application

Introducing the first version of my Twitter Hash Tag Scanner Application

The Strictly HashTag Hunter is a Windows form application that allows you to find the most relevant HashTags and Twitter accounts for a variety of specified search terms and keywords.

This application is ideal for people who have just created a Twitter account and want to analyse their own site specific keywords to find #HashTags they should be following or using themselves.

For instance you might have a blog or site that uses an AutoBlogging tool like my Strictly TweetBot Wordpress Plugin and you might want to set up various AutoTweets with relevant HashTags that are related to certain keywords and content snippets.

This tool also helps you find the most important Twitter accounts that you should be following as it analyses those people that are using the keywords or sentences that you enter on Twitter at that point in time to find the most popular HashTags related to those words as well as the accounts that are using them the most.

Obviously the time of day you run your scan will affect the results as different people Tweet at different times of the day but you will see from the results which Twitter accounts have the most followers and therefore worth following for your own account.

The primary aim of this tool is to help you save time trying to work out which #HashTags to use for your own Tweets as well as working out which @accounts to follow for your own Twitter account.

The Strictly Twitter Hash Tag Hunter is built as a windows application that runs on your own desktop and it hooks into Twitters API to obtain the results. It is perfect for SEO and Social Media analysts as well as people with a new Twitter account who don't know which hash tags and accounts they should be following to make an impact on the social scene.



Screen 1 shows how you enter one or more search terms that you want to find information for. These terms can be anything but if you are looking to utilise this tool with my Strictly TweetBot Wordpress Plugin then you should be looking to find the #HashTags and @Accounts to follow on Twitter related to the key terms your website is based on.

For example if you were running a site about Horse Racing and wanted to find out which Twitter @Accounts to follow and which #HashTags to use in your Tweets you would enter a number of search terms like so:

Horse Racing
Kempton Park
fromthestables.com
Free Racing Tips 
Twitter HashTag Hunter Start up screen
Enter each keyword or search term on it's own line.

Once you have entered each term on it's own line you click the "Search" button and the Scanner gets to work analysing your keywords and finding related Twitter information.

For each search term and keyword it will scan the Twitter API for those words looking for the most popular #hashtags that are related to those keywords.

It will also find the Twitter accounts that make the most use of these terms before ordering the accounts by the number of followers each account has and the hash tags by the number of times they are referenced by those accounts.

On completing the Scan

Screen 2 shows the most popular hash tags found for the search terms that were entered.
Twitter HashTag Completion Screen
The most popular hash tags found for the entered search terms and keywords.

Screen 3 shows the most followed Twitter accounts that used the terms you searched for.
Twitter HashTag Completion Screen
The most followed Twitter accounts for the entered search terms and keywords.

Following Accounts or Hash Tags

Once the Twitter Scan has completed and you have looked at the results you can simply click on the Account or Twitter Link column value to open up the desired URL in your default browser.

Screen 4 shows you selecting the desired Account you want to examine on Twitter.

Selecting a Twitter Account
Selecting an account to examine


Screen 5 shows the http://twitter.com page opening in your browser where you can decide whether or not the account or hash tag is worth following.

Folllowing the selected Twitter Account
Viewing the account in Twitter and following them


If you are already logged into Twitter at the time then it's just a simple matter of clicking the "Follow" button in the top right of the screen and your own Twitter account will now be following the account you opened.


About the Twitter Hash Tag Scanner Application

The application is a multi-threaded standalone executable Windows application and it has been built with users and Twitter in mind so that the Twitter API is not overloaded and abused and that you can continue to get all the information you need from their service.

A progress bar keeps you updated with the amount of scans it has carried out as well as the number of accounts and hashtags it has already found matching your search terms.

If for whatever reason Twitter blocks your requests (for example if you were hammering their API with dozens of search terms in one scan) then the application will slow down the amount of the requests it makes and increase the delays between requests. It also has some built in methods for bypassing certain blocking methods as well as the ability to access the information from other sources.

I am hoping to expand this tool over the years and I have had great feedback from both novice users who have found it very useful in deciding who to follow when they first start to use Twitter as well as SEO experts who utilise social media and Twitter all the time for marketing purposes.

As an introductory offer I am offering this application for the same price as a small donation of only £10.00 and you can buy this application from my application order page.

Tuesday, 4 October 2011

Find position of a string within a paragraph using a Regular Expression

How to find a string within another string using a Regular Expression instead of strpos or stripos

I was writing a piece of code in PHP the other day where I had to find a snippet of text within another longer piece of text (e.g an article) that contained a word. I then wanted to take X number of characters from that first point and return a snippet that didn't cut off the last word in the sentence.

At first I was using the PHP functions strpos and stripos but these don't allow you to use Regular Expressions as the search term (needle in the haystack as PHP.net calls the parameters) and therefore it meant that I was returning mismatches due to the search term being contained within other words.

E.G if I was looking for the word wool it would match woollen.

Therefore the answer was to use a custom function that made use of preg_match and a non greedy capture group at the beginning of a pattern that could be passed to the function (without delimiters).

The function is below



/**
 * Function to find the first occurence of a regular expression pattern within a string
 *
 * @param string $regex
 * @param string $str
 * @param bool $ignorecase
 * @return variant
 */
function preg_pos( $regex, $str, $ignorecase ) 
{ 
 // build up the RegEx wrapping it in @ delimiters
 $pattern = "@^(.*?)" . $regex . "@" . ($ignorecase===true ? "i" : "");

 if( preg_match( $pattern, $str, $matches ) ) {
  return strlen( $matches[ 1 ] ); 
 }

 return false; 
} 


As you can see the pattern needs to be passed in without delimiters e.g instead of /\bwool\b/ or @\bwool\b@ just pass in \bwool\b.

I then add a capture group to the beginning that is non greedy so that it finds the first match from the start of the input string ^(.*?) and then if the pattern is found I can do a strlen on the matching group to get the starting position of the pattern.

If you want the pattern to be case-sensitive then you can just pass in TRUE or FALSE as the extra parameter and the ignore flag will be added to the end of the pattern.

An example of this code being used is below. The code is looping through an array of words looking for the first match within a longer string (some HTML) and then taking 250 characters of text from the starting point, ensuring the last word is a whole word match.


// find first occurence of any of the terms I am looking for and then take 250 characters from the first word
// ensuring I get a whole word at the end

$a = explode(" ",$terms);
foreach($a as $w){

 // skip empty or small terms

 if(!empty($w) && strlen($w) > 2){
   
  // get the position of the word ensuring its not within another word - using \b word boundary - notice no RegEx delimiters @regex@ or /regex/
  // also ensure any special characters within the word are delimited to prevent a mismatch
  $pos = preg_pos( "\b" . preg_quote($w) . "\b", $html, true ) ;

  // if pos is false then its empty otherwise 

  if($pos !== false){

   // found the word take 250 chars from the first occurrence

   $text = substr($html, $pos, 250);
   
   // roll back to last space before our last word to ensure we don't get partial words 

   $text = substr($text, 0, strrpos($text," "));
   
   // now we have found a term exit
   break;
  }
 }
}


Also remember to wrap your word in preg_quote so that any special characters that are used by the Regular Expression engine e.g ? . + * [ ] ( ) { } etc are all characters that need to be escaped properly.

I found this function quite useful.

Monday, 26 September 2011

Regular Expression functions for reformatting content for Wordpress

Using RegEx to reformat Wordpress content

If you read my blog you will know I am "Mr Automate" and I specialise in scraping, reformatting, spinning and automatically posting content to Wordpress, Twitter and other sites.

I use Wordpress for a couple of my blogs and because the well known plugin WP-o-Matic has stopped being supported and the only replacements are paid for plugins like WP Robot I have basically written my own custom version to handle my automated content imports.

A couple of useful Regular Expressions you might like to use which I find particular helpful are below.

I am using PHP as that is the language Wordpress is written in but the core Regular Expressions should be easily transferable between languages.


1. Removing IFRAMES

As YouTube has moved from it's old OBJECT / EMBED nested tag soup to an IFRAME many feeds containing videos will now contain IFRAME's which need to be handled correctly as we all know that viruses can be transfered through rouge IFRAME content.

If you are just importing content from a feed without parsing and checking it then just imagine if for whatever reason the feed you are using one day contains an IFRAME with a SRC pointing to a dodgy virus infected URL. If you don't sanitize the content you will be inserting this dangerous content into your own blog for your visitors to become infected by.

Although most modern browsers and virus checkers are good at picking up on potential malicious URL access it is always wise to run your own checks using a white list of "allowed" domains rather than a "black list" of banned ones.

This code is a basic example of looping through all the IFRAMES on a page, checking their SRC tag for a known white list of acceptable domains and then replacing any IFRAME's that don't match.

// match all XHTML IFRAME tags where $content holds your HTML
$videocount = preg_match_all("@(<iframe[\s\S]+?<\/iframe>)@",$content,$videos,PREG_SET_ORDER);

foreach($videos as $video){

$object = $video[1];

// only allow certain domains e.g youtube, dailymotion, vimeo, cnn, fox, msnbc and the bbc
if(!preg_match("@src=['\"]http:\/\/(www\.)?(?:youtube|dailymotion|vimeo|cnn|fox|msnbc|bbc)\.@i",$object)){

// replace the IFRAME as we don't know where it is pointing to

$content = preg_replace("@" . preg_quote($object) . "@","",$content);
}

}


2. Re-sizing images

I have already written an article about how you can use regular expressions and a callback function to reformat videos or images so that they fit your template and this follows that theme.

PHP like Javascript and other decent languages offers the ability to create anonymous (unnamed) functions on the fly to be used as callbacks in regular expression replace functions.

Passing a function as a parameter within another function is known as a lambda function and this is very useful in certain situations such as parsing HTML where you want to remove illegal tags and attributes.

This example is pretty niche to my own needs but the idea is useful and can easily be converted to your own reformatting needs.

Basically I show little Google Adsense box adverts at the top right of all my articles and as these are floated to the right I don't want any images within my articles that are less than the size of my content area (minus the size of the advert) to also be aligned to the right as this causes a horrible looking mess.

Therefore I use this code to ensure that the Wordpress class "alignleft" is always inserted on the first image in any article but only if the width of the image is less than 350px. It doesn't matter if the image isn't right at the top of the article (although they usually are) but it helps to ensure a smooth flow and layout.

The Preg_Replace function in PHP is nice in that you can limit the number of replacements to one or more if you so wish and in this case I only want to reformat the first image. You will see that the lambda function I use also contains multiple other regular expressions and there is nothing stopping you from writing complex code within these anonymous functions.

$content = preg_replace_callback("@(<img[^>]+?class=['\"])([\s\S]*?)(['\"][\s\S]*?>)@i",
create_function(
'$matches',
'preg_match("@width=[\'\"](\d+)[\'\"]@",$matches[0],$widthmatch);
$chk=true;
if($widthmatch){
if(is_numeric($widthmatch[1])){
if($widthmatch[1] >= 350){
$chk = false;
}
}
}
if($chk && !preg_match("@alignleft@",$matches[2])){
$res = $matches[1] . trim( preg_replace("@alignright@","",$matches[2]) . " alignleft") .$matches[3];
}else{
$res = $matches[1] . preg_replace("@alignright@","",$matches[2]) .$matches[3];
}
return $res;'),$content,1);



As you can see the lambda anonymous function carries out the following logic

  • It looks for the first image that has a class attribute on it and passes the match collection into the anonymous function.
  • It then checks the image for a width attribute using another regular expression.
  • If found it then checks that the value of the width attribute is numeric and if so it tests to see if the value is over my allowed limit of 350px setting a flag if so.
  • I then use another regular expression to test for the existence of the standard Wordpress class "alignleft" and if it isn't found then I add it in making sure to replace any "alignright" class if it exists.
  • The lambda function then returns the new reformatted content of the image which is then used as the replacement value in the Preg_Replace function.
Just two useful examples which can be expanded upon by anyone interested in Regular Expressions and AutoBlogging.

Remember if you require any custom plugin work for Wordpress, including special reformatting functions, scraping code or content automation then contact me for a quote.

Wednesday, 31 August 2011

Would you switch back to IE9 now that they have finally made a good browser?

Now that IE9 is standards compliant and actually good will you switch back to using it?

A few months ago I wrote an article about why I hated debugging in IE8 which revolved around their single threaded event model (window.event) and the delay it took before debug messages were outputted to the console.

A member of the Microsoft team commented that he had run the test case I had created in IE9 and that it had run perfectly. No high CPU, no frozen console, no high memory usage or other performance problems at all.

As you cannot install IE9 on WinXP, which is what I use on my home laptop (due to Vista being shite), I haven't had the pleasure of using Internet Explorer 9 a lot until I installed Windows 7 on my work PC.

I have to say that Windows 7 is actually a great operating system and I especially love the way it has incorporated many of the features of clean up and performance tuning tools like Tune Up Utilities and CCleaner into the main OS.

I also have to say that Internet Explorer 9 is the first IE browser I actually like.

Not only is it blindingly fast they have finally listened to their customers who have been complaining for years and made their JavaScript engine standards compliant.

Obviously this makes the Browser / Document mode spaghetti even harder to detect and I haven't been able to find a way as of yet to detect IE 9 browser mode running as IE 8 or 7 on 32 bit machines but that is not a major issue at all.

What I am wondering though is that now that IE9 is actually a good browser, with a good inbuilt development console and element inspection tools, how many developers will actually return to using it either as their browser of choice for surfing the web or for their primary development.

My browser development survey which I held before IE9 was released showed that developers would rather use Chrome or Firefox for surfing and would also always choose the development tools that those browsers bring than use IE 7 or 8.

I know that I changed from using IE to Firefox as my browser of choice some eons back and I changed from Firefox to Chrome for both surfing (speed is key) and developing (no need for Firebug or any other plugin) the other year when Firefox started to suffer major performance problems.

These performance problems are always either to do with having far too many plugins installed. Setting the browser up to check for new versions and any installed plugins on start up which cause long load times, Firebug issues and errors in the Chrome source. This is on top of all the constant problems that Flash seems to bring to any browser.

I use Chrome for surfing due to it's speed but if I leave a few tabs open all night that contain pages running Flash videos then by morning my CPU has flat-lined and memory has leaked like the Titanic.

This is more a problem with Flash than Chrome and I try to keep this browser for surfing as clean and as fast as possible by not installing plugins. Then if I need to hack around or do some hardcore web development I will use Firefox and all the great plugins when I need to.

I don't think I could really get hacking with Chrome alone as when I am really getting stuck into a site I need my key plugins e.g the web developer toolbar, hackbar, header modifier, Request and Response inspectors e.g HTTP Fox, Firebug, ColorZilla, YSlow and all the Proxy plugins that are available.

However for basic development, piping out messages to the console, inspecting the DOM and checking loaded elements and their responses then Chrome has everything Firebug has without all the JavaScript errors.

In fact it's surprising how many developers don't even realise how much is possible with Chrome's inbuilt developer tools, speed tests, header inspectors, element inspectors and DOM modifiers and other great debugging tools that used to be the preserve of Firebug alone.

Which leads me back to IE9.

Yes it's fast and it's developer tools are okay with options to clear cookies, disable JavaScript, inspect the DOM and view the styles but it's no Chrome or Firebug yet;

Therefore what I want to know is how many people (and by that I mean developers) will switch back to using Internet Explorer 9 for pure web surfing?

For sure it's fast, supports the latest CSS standards and all the great HTML 5 features that we have all seen with floating fish tanks and the like. But is this enough for another browser switch?

I have all the major browsers installed at work on Win7 including IE9, Firefox, Chrome, Opera, Safari, SeaMonkey and a number of others I cannot even remember the names of they are that obscure.

Before upgrading I even had Lynx and Netscape Navigator 4 installed as I used to be so anal that any code such as floating DIV's had to work in the earliest browsers possible. However I only use these browsers for testing and I stick to Chrome (for surfing) and Firefox (for hacking) which leaves little room for IE9 at the moment.

I know it's a good browser. It's standards compliant and it no longer crashes when I try to output anything to the console so why shouldn't I try it again.

Maybe one of my readers can help me decide on the browser of choice for surfing and development and how IE9 fits into that toolset.

Maybe I am missing some really cool features that I would use all the time but to be honest I am not into throwing sheep at randoms or constantly voting sites up and down and writing comments on them. Therefore for me to switch back from Chrome to another browser such as IE9 there has to be a really good reason for me to do so.

So does anyone have any reason why I should re-consider using IE9 at work for my web surfing or development?

Wednesday, 3 August 2011

HTML Super Trim Functions

Remove non breaking spaces and other white space either side of strings

Most languages have a Trim function that removes white space from either side of a string.

However a lot of the time these Trim functions will only remove standard white space e.g if you hit the space bar a couple of times and not other forms of white space such as tabs, new lines or HTML space characters such as non breaking spaces whether they are HTML entity encoded: or Numerically encoded e.g

Therefore sometimes you may need a "Super Trim" function that will handle the removal of all types of space characters including HTML entities.

The following have been written in PHP but can easily be converted into C# or VB. The main part to take away is the regular expression used within each function which replaces a string containing one or more space characters whether they be control characters or HTML entities from either side of the string.

// wrapper function to do trim both sides
function HTMLTrim($text){

// call both functions at once
return HTMLLeftTrim(HTMLRightTrim($text));
}

// removes spaces and at the beginning of strings
function HTMLLeftTrim($text){

// remove space to the left of the text
return preg_replace("@^(&#160;|&nbsp;|\s)+(\S+)@","$2",$text);

}

// removes spaces and at the beginning of strings
function HTMLRightTrim($text){

// remove space to the right of the text
return preg_replace("@(\S+)(&#160;|&nbsp;|\s)+$@","$1",$text);

}


You can test this out in a simple PHP page with the following code:


$str = " &nbsp; &nbsp; hello there  &#160;&#160;&#160;   ";

echo "before trim its '" . $str . "'";

echo "<br><br>now its '" . HTMLTrim($str) . "'";


Which returns the following output:

before trim its '     hello there      '

now its 'hello there'

I find it very useful when I am scraping content from the web and need to handle the removal of a mixture of standard spaces and HTML spaces.

Wednesday, 27 July 2011

My JavaScript HTML Encoder Function

Why did I write my JavaScript HTML Encoder object

One of my most popular scripts that continues to get downloaded a lot as well as spark many an email from companies and users is my HTML Encoder script that I wrote many a year back.

You can see the script in action here: www.strictly-software.com/htmlencoder

I was emailed another function that claimed to do the same work earlier and I just thought it would be interesting to other readers to know why I created the script in the first place and why many of the examples out on the net that claim to do the same job are just not up to scratch.

On many sites nowadays people are obtaining content from various sources through XML and RSS feeds and they often use JavasScript and AJAX to do part of the job. I know that when I was creating my football site www.hattrickheaven.com which was basically an exercise in using the Google AJAX API I came across the following problems:

  1. Content is obtained from a multitude of sources and there is no common standard that can be relied on 100%.
  2. Content is often mixed together through feed blenders like Yahoo Pipes or Blastcasta.
  3. Content is often loaded in on the fly using AJAX, a Scraper Proxy to do cross domain content jacks and other formatting duties such as on the fly translations etc.
  4. There is no built in function for HTML encoding in the JavaScript language.

Now I looked at many an HTML Encode solution before writing my own but the main problem I needed to overcome was that content could already be HTML encoded or partially HTML encoded and functions I came across did nothing to handle this problem.

For example if you had this bit of text you will notice that it has an encoded ampersand in the middle but the quotes are not encoded.

"Rob says hello & goodbye"

If you ran this through many of the HTML Encode functions out there including many server side ones it will double encode the & in the & so that you end up with this.

&quot;Rob says hello &amp;amp; goodbye&quot;

Now you may want this to happen but I doubt it.

Not if you want to run all your content through the same function without worrying about double encoding issues like this on your website.

This is why I wrote my HTML Encoder object as it puts into practise the technique I used whilst writing ASP classic sites that supported the UTF-8 character set.

With multi-lingual sites that display Arabic or Japanese character sets you don't want to be using the inbuilt Server.HTMLEncode function on all your textual input as it will triple the size of everything you store by converting every non ASCII character into a &#XXXX; encoded string.

When your database is set up to correctly store your text as Unicode (nvarchar etc) and you are outputting it with the correct code-base then you need something a bit more clever to do the job as you still need to encode characters that can cause you damage such as the naughty 4 " ' < > as well as the ampersand & to make your page validate (URL's, Querystrings etc).

This is obviously for security reasons as you don't want people to be able to malform your input and break your layout or insert XSS hack vectors into your system.

However running the standard HTML Encode function will encode everything as well as cause double encoding issues. Therefore the HTML Encoding needs to be done in stages and the ampersand has to be handled correctly as it makes up part of the HTML encoded &#XXXX; format.

So whilst it comes as quite a surprise to see how popular this script is (#1 on Google for "HTML Encode with Javascript") it is not really surprising when one thinks about the problem this script solves.

Whilst I don't claim to be any kind of coding genius and I am sure the code can be improved it does do what is says on the tin and if you are thinking of using another JavaScript function to HTML Encode your content then compare it first with my tool to see if it handles the double encoding issue as if it doesn't then you might find yourself having problems down the road.

Tuesday, 26 July 2011

Wordpress posts_per_page Query Option Not Working

Fixing the posts_per_page query parameter bug in Carrington Themes

This is just a quick but annoying bug fix for anyone using the Carrington Theme for Wordpress.

If you are creating your own custom templates or pages and want to create your own custom query for the loop then you will probably want to specify how many articles to show.

This is determined by the posts_per_page query filter which determines how many articles are displayed during the loop output e.g

query_posts('cat=234&posts_per_page=5); 

However when I was trying to create a custom template earlier this value was being ignored and I couldn't work out why.

I eventually stumbled across a thread on the Wordpress forums that gave me the solution and it relates to a "feature" in the Carrington theme. Apparently it creates it's own function which ignores any value you may give the posts_per_page query option.

The solution is to disable this "feature" by removing the filter that sets it in motion. Once this is done your loop queries will take into account any value you give the posts_per_page query option.

You can disable this feature on the page you are building by adding the following code before creating your query loop.

remove_filter('pre_get_posts', 'cfct_posts_per_archive_page');


A full example of a custom template page based on the Carrington theme that uses this feature can be seen below.


<?php
/*
Template Name: SomethingTemplate
*/

if (__FILE__ == $_SERVER['SCRIPT_FILENAME']) { die(); }
if (CFCT_DEBUG) { cfct_banner(__FILE__); }

get_header();

?>

<div id="content">

<?php


// get category id by name - if you know it then skip this and add it directly to the query
$id = get_cat_id('Some Category');

// remove the carrington blog filter that overwrites the posts_per_page query filter parameter
remove_filter('pre_get_posts', 'cfct_posts_per_archive_page');

// create a query to filter by the desired category and show the number of posts as specified in the global settings and handle paging
query_posts('cat='.$id.'&posts_per_page='.get_option('posts_per_page') . '&paged='.$paged);


if (have_posts()) {
while (have_posts()) {
the_post();?>
<h1><a href="<?php the_permalink(); ?>" title="This is my blog article: <?php htmlspecialchars(the_title()) ?>"><?php the_title(); ?></a></h1>

<?php the_content(); ?>

<?php
}
}

?>

<div class="pagination">
<span class="previous"><?php previous_posts_link('« Previous') ?></span>
<span class="next"><?php next_posts_link('Next »') ?></span>
</div>

</div>