Tuesday, 30 October 2012

New version of the SEO Twitter Hunter Application

Introducing  version 1.0.4 of the Twitter Hashtag Hunter Application

I have just released the latest version of the popular windows application that is used by SEO experts and tweeters in combination with my Strictly Tweetbot Wordpress plugin to find new @accounts and #hashtags to follow and use.

Version 1.0.4 of the Twitter HashTag Hunter application has the following features:
  • A progress bar to keep you informed of the applications progress in scanning.
  • More detailed error reporting including handling the fail whale e.g 503 service unavailable error.
  • More HTTP status code errors including 400, 404, 403 and the 420 Twitter Scan Rate exceeded limit.
  • Clickable URL's that open the relevant Twitter account or Hash Tag search in your browser.
  • Multiple checks to find the accounts follower numbers to try and future proof the application in case Twitter change their code again.
  • A new settings tab that controls your HTTP request behaviour.
  • The ability to add proxy server details e.g IP address and Port number to scan with.
  • The ability to change your user-agent as well as a random user-agent switcher that picks between multiple agent strings for each HTTP request when a blank user-agent is provided.
  • An HTTP time-out setting to control how long to wait for a response from the API.
  • A setting to specify a wait period in-between scans to prevent rate exceeded errors.
  • A setting to specify a wait period when a "Twitter scan rate exceeded" error does occur.
  • Extra error messages to explain the result of the scan and any problems with the requests or settings.
The main new feature of 1.0.4 is the new settings panel to control your scanning behaviour. This allows you to scan through a proxy server, specify a user-agent, set delay periods in-between scans and the "Twitter Scan Rate exceeded limit" error which occurs if you scan too much.

Changing the Scanner Settings

For Search Engine Optimisation (SEO) experts or just site owners wanting to find out who they should be following and which #hashtags they should be using in their tweets this application is a cheap and useful tool that helps get your social media campaign off the ground by utilising Twitters Search API.

You can download the application from the main website www.strictly-software.com.

Monday, 22 October 2012

Fixing Postie the Wordpress Plugin for XSS Attacks that don't exist

Fixing the "Possible XSS attack - ignoring email" error message in Postie

As you may know if you read my earlier post on fixing the Wordpress plugin Postie when it wouldn't let me pass multiple categories in their various formats in the subject line a new version of Postie has come out since.

However I have been regularly noticing that emails that should be appearing on my Wordpress site when they are posted by email using Postie haven't been.

Today I looked into why and when I ran the manual process to load emails by pressing the "Run Postie" button I was met with an error message that said

possible XSS attack - ignoring email

I looked into the code and searched for the error message which is on line 38 of the file postie_getmail.php and it gets displayed when a Regular Expression runs that is supposed to detect XSS attacks.

The code is below

if (preg_match("/.*(script|onload|meta|base64).*/is", $email)) {
 echo "possible XSS attack - ignoring email\n";
 continue;
}

I tested this was the problem by running the script manually in the config area of Postie and outputting the full email before the regular expression test.

As the email is base64 encoded (well mine is anyway) the full headers are shown at the top of the encoded email e.g

Return-Path:
X-Original-To: xx12autopost230.sitename@domain-name.com
Delivered-To: xx12autopost230.sitename@domain-name.com
Received: from smtp-relay-2.myrelay (smtp-relay-2.myrelay [111.11.3.197])
by domain-name.com (Postfix) with ESMTP id 8497724009C
for ; Mon, 22 Oct 2012 05:49:32 +0000 (UTC)
Received: from xxxxxxx (unknown [11.1.1.1])
by smtp-relay-2.myrelay (Postfix) with ESMTP id 9E3B495733
for ; Mon, 22 Oct 2012 06:45:30 +0100 (BST)
MIME-Version: 1.0
From: admin@sitename.com
To: xx12autopost230.sitename@domain-name.com
Date: 22 Oct 2012 06:46:28 +0100
Subject: Subject: [My Subject Category1] [My Subject Category2] Title of Email
 October 2012
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
Message-Id: <20121022054530 .9e3b495733=".9e3b495733" smtp-relay-2.myrelay="smtp-relay-2.myrelay">

PHA+PHN0cm9uZz5ZZXN0ZXJkYXlzIG1lbWJlcnMgaGFkIGFjY2VzcyB0byA0NSB0aXBzIGFj
cm9zcyA4IGRpZmZlcmVudCBzeXN0ZW1zLjwvc3Ryb25nPjwvcD48cD5JZiB5b3UgaGFkIHBs
YWNlZCBhIEJldGZhaXIgbWluaW11bSBiZXQgb2YgJnBvdW5kOzIuMDAgb24gZWFjaCBiZXQg
PHN0cm9uZz5vbiBFVkVSWSBzeXN0ZW08L3N0cm9uZz4gKExheXMsIFBsYWNlIERvdWJsZXMs
IFdpbnMgZXRjKSB0aGF0IGhhZCBhbiBTUCBsZXNzIHRoYW4gMTkvMSBhdCB0aGUgdGltZSBJ

I've just shown a bit of the message which is base64 encoded.

As you can see if you do a search for one of the strings he is searching for as a word not in a scriptual context e.g base64 not base64("PHA+PHN0cm9"); 

The word base64 appears in the headers of the email e.g:

Content-Transfer-Encoding: base64

Therefore the regular expression test fails and Postie displays the "possible XSS attack - ignoring email" error message.

Therefore just doing a basic string search for these words:

script, onload, meta and base64

Will mean that you could find yourself having emails deleted and content not appearing on your Wordpress site when you expect it to. All due to this regular expression which will be popping up false positives for XSS attacks when none really exist.

Also these words could legitimately appear in your HTML content for any number of reasons and not just because they are used in the email headers so a better regular expression is required to check for XSS attacks.

How to fix this problem

You could either remove the word base64 from the regular expression or you could delete the whole test for the XSS attack.

However I went for keeping a test for XSS attacks but making sure they were checking more thoroughly for proper usage of the functions rather than simple tests for occurrences of the word.

The regular expression is more complicated but it covers more XSS attack vectors and I have tested it myself on my own site and it works fine.

You can replace the code that is causing the problem with the code below.


if(preg_match("@((%3C|<)/?script|<meta|document\.|\.cookie|\.createElement|onload\s*=|(eval|base64)\()@is",$email)){
      echo "possible XSS attack - ignoring email\n";
      continue;
}

Not only does this mean that it won't fall over when the words are mentioned in headers but it actually looks for the correct usage of the hack and not just the word appearing. E.G instead of looking for "script" it will look for

<script %3Cscript </script %3Cscript 

This includes not only the basic <script but also urlencoded brackets which are another common XSS attack vector. You could include other forms of encoding such as UTF-8 but it all depends on how complicated you want to make the test.

As you may know if you have read my blog for a long time hackers are always changing their methods and I have come across clever two stage SQL injection attacks which involve embedding an encoded hack string first and then a second attack that's role is to unencode the first injection and role out the hack.

The same can be done in XSS attacks, e.g encoding your hack so it's not visible and then decoding it before using an eval statement to execute it. However I am keeping things simple for now.

I have also added some more well known XSS attack vectors such as:

eval( , document. , .createElement and .cookie 

As you can see I have all prefixed or suffixed them with a bracket or dot which is how they would be used in JavaScript or PHP.

Notice however that I haven't prefixed createElement and cookie with the word document. This is because it is all too easy to do something like this:

var q=document,c=q.cookie;alert(c)

Which stores the document object in a variable called "q" and then uses that to access the cookie information. If you have a console just run that piece of JavaScript and you will see all your cookie information.

This regular expression still also tests for:

<script, base64(, <meta, onload= and onload =

but as you can see I have prefixed the words with angled brackets or suffixed them with rounded brackets, dots or equal signs (with or without a space).

This has solved the problem for me and kept in the XSS attack defence however if you are passing HTML emails containing JavaScript to your site just beware that if you use any of these functions they might be flagged up. 

I have tested each XSS attack vector but let me know of any problems with the regular expression.

Also I have removed the .* before and after the original test as it's not required. Also it just uses up memory as its looking for any character that may or may not be there and the longer the string it's searching the more memory it eats up.

I have updated my own version of this file and everything has gone onto the site fine since I have done so.

If anyone else is having problems with disappearing posts driven by Postie then this "might be the cause."

You can see my Wordpress response here: Wordpress - Postie Deletes Email but doesn't post

Monday, 15 October 2012

Twitter changes their format for status feeds

Tweets not showing up? Twitter changes their feed format for status feeds

You may have noticed over the couple of weeks that the usual Tweets from my Twitter account @strictlytweets have not been showing in the right hand side bar.

This is because Twitter have changed the URL that you must request to get those tweets from.

http://twitter.com/statuses/user_timeline/strictlytweets.json?count=10

To their new API which uses a secure URL (e.g from http to https)

https://api.twitter.com/1/statuses/user_timeline.json?screen_name=strictlytweets&count=10


Now if you have a callback function it would be called in the same way with a parameter passed in the URL for the name of the function to be called.

Although I am not sure whether Twitter have changed their name of their standard function which formats Tweets into links from twitterCallback to twitterCallback2 if you take a look at the one blogger uses you can see they are calling it twitterCallback2.

You can see their script here:

http://twitter.com/javascripts/blogger.js

Therefore if you have a special function that you need to call to format your tweets in your own style, cache on your own server or scroll them you will still need to add the name of your callback function to the script source and make sure it accepts a JSON hash table of Tweet objects.

You will also need to reference this script before calling the new script URL that gets the statuses.

If you look at this blog you can see that the first script loads in bloggers own formatting of the Tweets in  through a call to blogger.js and the second one gets the new Twitter feed status format in the new URL format using JSON.


<script type="text/javascript" src="http://twitter.com/javascripts/blogger.js"></script>
<script type="text/javascript" src="https://api.twitter.com/1/statuses/user_timeline.json?screen_name=strictlytweets&callback=twitterCallback2&count=6"></script>


Twitter are constantly changing and like most sites are moving to secure URL's e.g https and using JSON instead of XML/RSS to return data to the user.

So if you have a Twitter account and want to show your own Tweets on your site and it hasn't been working lately this is the reason why.

Just switch over to the new https format and you will be okay as long as you also implement some form of caching to handle the rate limit which is only 150 requests per hour per IP address NOT per domain. Or if you authenticate each request with OAuth you can make 350 requests per hour.

As my IP address is also shared by a lot of other sites that use this feature the Tweets may still not appear until I come up with a form of caching (which I haven't got round to doing) but the logic would be something like this.
  • Make the request to the new API URL.
  • Format the HTML using your callback function.
  • Make an AJAX request to a server side script (PHP/ASP/.NET etc) that stores the formatted HTML as a text/html file.
  • If you get back a 400 Status Code (Bad Request) which is the code Twitter send you when you exceed their rate limit. Then you can display this HTML file instead.


You can read more about rate limits here.

Monday, 24 September 2012

Using String Builders to speed up string concatenation

Using String Builders to speed up string concatenation

If you are using a modern proper language then a string builder object like the one in C# is a standard tool for adding strings without the overhead of concatenation which can be a performance killer.

The reason is simple.

When you do this

a = "hello I am Rob";

a = a + " and I would like to say thank you";

a = a + " and good night";


A lot of languages have to make a copy of the string built so far and store it in memory before creating the new string.

This means that the longer the string gets the more memory is used up as two copies have to be held at the same time before being joined together.

I have actually seen ASP classic sites crash with out of memory errors caused by people using string concatenation to build up large RSS feeds.

The reason I am mentioning this is because of a comment I was given about my popular HTML Encoder object that handles double encoding, numerical and entity encoding and decoding with partial and fully encoded strings.

I have updated the numEncode function after the comment from Alex Oss to use a simple string builder which in JavaScript is very simple.

You just have an empty array, push the new strings into it (at the end of the array) and then join it together at the end to get the full string out. You can see the new function below.


// Numerically encodes all unicode characters
numEncode : function(s){ 
 if(this.isEmpty(s)) return ""; 

 var a = [],
  l = s.length; 
 
 for (var i=0,len=l.length;i "~"){ 
   a.push("&#"); 
   a.push(c.charCodeAt()); //numeric value of code point 
   a.push(";"); 
  }else{ 
   a.push(c); 
  } 
 } 
 
 return a.join("");  
}, 

You can download the latest version of my HTML Encoder Script for JavaScript here.

However in old languages like ASP classic you are stuck with either string concatenation or making your own string builder class.

I have made one which can be downloaded from my main website ASP String Builder Class.

You will notice that it ReDim's the array in chunks of 128 (which can be changed) and once 128 elements have been used it then ReDim's by another large chunk.

A counter is kept so we know how many element we actually have added and once we want to return the whole string we can either just RTRIM it (if we are joining with a blank space) or ReDim it back down to the right array size before joining it together.

This is just an example of how a string builder class is used and you could make a similar one in JavaScript that lets you access specific elements, the previous or next slot, update specific slots and set the delimiter like this ASP version.

Most modern languages have a String Builder Class but if you are using old languages or scripting languages like PHP or ASP classic then adding strings to an array before joining them together is the way to go for performances sake.

Friday, 31 August 2012

New Version of the Twitter Hash Tag Hunter Application

Twitter HashTag Hunter Version 1.0.3 Released

Version 1.0.3 of the popular Twitter HashTag hunter Application has just been released.

This version has had an update to keep on top of the ever changing Twitter API and fixes a bug that was returning a follower count of 0 for some people.

You can find more out about the application over on my main site: Twitter HashTag Hunter Application.

Any existing users who want a new copy please email me with your reference/invoice number or details that prove that you bought the application previously and I will email you a new version.

Wednesday, 29 August 2012

Shrinking an MS SQL Database MDF file after a TRUNCATE or big DELETE

Shrinking an MS SQL Database MDF file after a TRUNCATE or big DELETE

I am not a DBA and we don't have a dedicated one at our company therefore when I had a database file with a huge table containing 80 million rows of data taking up 35 GB I needed to remove it.

I re-planned my usage of the table and decided that it would only keep a days worth of data instead of every day (going back 3 years) and re-jigged some code about.

I then set up an nightly job with a TRUNCATE TABLE statement to remove the data quickly at midnight each day.

However just by doing this alone does not reduce the size of the database file (MDF) and you will have reserved and unused space that you may want to reclaim or get rid of.

The current sizes of the database file before running any clean ups was 35 GB and as I was using simple mode for the transaction log the size of that file was negligible. Therefore after much research I had to go through the following routine to reduce the size of the database file.

A lot of people will warn you not to Shrink a database file as it will only grow again and cause disk fragmentation which is correct however if you have just removed a large table like I have then this is one case where a DBCC SHRINKFILE command is useful.

Therefore this is the approach I followed this morning.

I first ran this SQL to find out current size of tables and indexes plus any reserved and unused space within each database table. I also list out the tables in order of reserved size to see which has the most to be reclaimed and whether or not it matched the table I thought was causing problems - which it did.

CREATE TABLE #t (name SYSNAME, rows CHAR(11), reserved VARCHAR(18), 
data VARCHAR(18), index_size VARCHAR(18), unused VARCHAR(18))

DECLARE @Sizes TABLE(name SYSNAME, rows int, reserved int, Data int, index_size int, unused int)

-- ensure correct values are being returned by using  @updateusage see http://msdn.microsoft.com/en-us/library/ms188776.aspx
EXEC sp_msforeachtable 'INSERT INTO #t EXEC sp_spaceused ''?'', @updateusage = N''TRUE'';'

INSERT INTO @Sizes
SELECT Name, Rows, CAST(SUBSTRING(Data, 1, LEN(Data)-3) as INT), CAST(SUBSTRING(Reserved, 1, LEN(Reserved)-3) as INT),
  CAST(SUBSTRING(Index_Size, 1, LEN(Index_Size)-3) as INT), CAST(SUBSTRING(Unused, 1, LEN(Unused)-3) as INT)
FROM #t

DROP TABLE #t

SELECT CAST(SUM(Data) as varchar)+' KB' as 'Data Size',
  CAST(SUM(Reserved) as varchar)+' KB' as 'Reserved Size',
  CAST(SUM(Index_Size) as varchar)+' KB' as 'Index Size',
  CAST(SUM(Unused) as varchar)+' KB' as 'Unused Size'
FROM @Sizes


SELECT *
FROM @Sizes
ORDER BY Reserved DESC

Then I ran this script that I found to find out if there is space that can be removed.

SELECT name ,size/128.0 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS int)/128.0 AS AvailableSpaceInMB
FROM sys.database_files;

This showed me that there was indeed a good 30 GB or so that could be removed.

Once I had worked out the new size of the data file in megabytes that I wanted to reduce the MDF file of my database to I ceased all activity on the database to help speed up the process and prevent any errors.

I then ran the following command: DBCC SHRINKFILE (details here)

To find the logical file name that you need to shrink either an MDF (database) or LDF (log file) run this command.

sp_helpdb [database name]

Once everything was ready I ran the following command to shrink a database to 6 GB (you have to pass the value in as MB) which left some room for the database to grow into.

DBCC SHRINKFILE([database name], 6144);

After running the command I did not see an immediate difference when I checked the properties of the database in MS Management Studio.

Therefore I ran a job to ReIndex my tables and rebuild rebuild statistics.

If you don't already have a nightly/weekly MS Agent job set up to do this for you then the following script of mine might be useful for you: ReIndex or ReOrganize your MS SQL indexes.

This script will allow you to set a percentage e.g 30% and if the index is fragmented over that percentage then a full REBUILD of the Index is carried out otherwise a REORGANIZE is performed.

After the rebuilding is done a call to EXEC sp_updatestats is made to update the systems statistics.

If you are going to use this SQL code check that the parameters within the stored procedure are to your liking and turn the DEBUG flag on if you require it.

The script will show you the fragmentation level before and after the script has run so that you can see how much difference it has made.

I found that after doing this when I went to database > properties > files I could now see the new sizes.

Logical NameFile TypeFilegroupInitial SizeAutogrowthPathFile Name
MyDBRows DataPRIMARY6144By 1 MB, unrestricted growthC:\Program Files 9x86)\Microsoft SQL Server\MSSQL.1\MSSQL\DATAMyDB.mdf
MyDB_logLogNot Applicable7377By 10 Percent, restricted growthC:\Program Files 9x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Data_logsMyDB_log.ldf

This clearly shows that my database MDF file has been reduced from 35 GB to 6 GB.

If anyone else has any comments or notes on best practise I would be glad to hear them.

Sunday, 19 August 2012

PayPal and Instant Payment Notification (IPN) Messages

Blocking bad bots and issues with PayPal's IPN service

If you are someone like me who likes to ban as much bad BOT traffic as possible then you might have added a rule in your .htaccess or httpd.ini file that blocks blank user-agents. Something like this.


# Block blank or very short user-agents. If they cannot be bothered to tell me who they are or provide gibberish then they are not welcome!                                        
RewriteCond %{HTTP_USER_AGENT} ^(?:-?|[a-z1-9\-\_]{1,10})$ [NC]
RewriteRule .* - [F,L]


This is because many bad bots wont supply a user-agent because of the following reasons:
  1. They think websites will check for user-agent strings and by not supplying one they will slip under the radar.
  2. The people crawling are using simple methods e.g they have just grabbed bits of code off the web that don't actually set a user-agent and used them as is.
  3. They are using simple HTTP functions like file_get_contents('http://www.mysite.com'); to get the HTTP response and not passing in any context data such as user-agent, timeouts, headers etc.
  4. They are making requests from websites that use CURL or WGet from places like WebMin to test Cron jobs or run crawl scripts and not supplying the user-agent parameters which are available.
Obviously it's always best to supply a user-agent as it identifies (or masks) your real code in any log file and prevents you getting blocked by people who ban blank user-agent strings.

However there come times when even big companies like PayPal send information without user-agent details.

One instance of this is the IPN notification messages they send (if set up) on sites that are using PayPal as their payment gateway.

When a customer makes a payment or cancels a subscription then PayPal will send a POST request to a special page on the website that then checks the validity of the request to make sure it comes from PayPal. If so it sends back the data along with a special key so that PayPal can confirm they made the request and the website owner is sure it's not a spoofer trying to get free items.

Once this handshake is done the relevant IPN data such as payment information can be obtained from the PayPal IPN Response.

However today I received an email in my inbox that said.

Dear Rob Reid,

Please check your server that handles PayPal Instant Payment Notifications (IPN). Instant Payment Notifications sent to the following URL(s) are failing:

http://www.mysite.com/?myIPN_paypal_handler

If you do not recognize this URL, you may be using a service provider that is using IPN on your behalf. Please contact your service provider with the above information. If this problem continues, IPNs may be disabled for your account. 

Thank you for your prompt attention to this issue.


Yours sincerely, 

PayPal 


After checking my PayPal account and the IPN history page I could see that a message was stuck in the queue an being re-sent every so often:


1W03166E61323432D
18/08/2012 09:01 BST
Original
---
http://www.mysite.com/?myIPN_paypal_handler
403
Resending
14
0U508347GT8609TY0
Transaction made

Therefore I checked my website logs for any instance of this request and I found that they were all returning 403 / Forbidden status codes.



66.211.170.66 - - [18/Aug/2012:08:54:47 +0000] "POST /?myIPN_paypal_handler HTTP/1.0" 403 417 "-" "-" 0/55005
66.211.170.66 - - [18/Aug/2012:09:27:11 +0000] "POST /?myIPN_paypal_handler HTTP/1.0" 403 417 "-" "-" 0/55010

As you can see - no user-agent!

Therefore I changed my .htaccess file to allow blank user-agents and re-sent the IPN message from the PayPal console and low and behold it was allowed.

Obviously if you are not using PayPal and their IPN system or not using any Payment service then you might consider to keep blocking blank agents due to the amount of bandwidth you will save (lots from my own stats).

Or you should check that payment systems own requests to your site to ensure you are not blocking them from doing anything important like sending payment information to your website.

Just be-warned in case you get a similar email from PayPal.