Googlebot, Sitemaps and heavy crawling

Googlebot over-crawling a site

I recently had an issue with one of my jobboards that meant that Googlebot was over-crawling the site which was causing the following problems:

1. Heavy loads on the server. The site in question was recording 5 million page loads a month which had doubled from 2.4 million within a month.
2. 97% of all their traffic was accounted by Googlebot.
3. The site is on a shared server so this heavy load was causing very high CPU and affecting other sites.

The reasons the site was receiving so much traffic boiled down to the following points.

1. The site has a large number of categories which users can filter job searches by. These categories are displayed in whole and in subsets in prominent places such as quick links and a job browser which allows users to filter results. As multiple categories can be chosen when filtering a search this meant Googlebot was crawling every possible combination in various orders.
2. A new link had been added within the last month to the footer which passed a sessionID in the URL. The link was to log whether users had Javascript enabled. As Googlebot doesn't keep session state or use Javascript it meant the number of crawled URLs actively doubled as each page the crawler hit would find a new link that it hadn't already spidered due to the new SessionID.
3. The sitemap had been setup incorrectly containing URLs that didn't need crawling as well as incorrect change frequencies.
4. The crawl rate was set to a very high level in Webmaster tools.

Therefore a site with around a thousand jobs was receiving 200,000 page loads a day nearly all of them from crawlers. To put this in some perspective other sites with 3000+ jobs, good SEO and high PR usually get around 20,000 page loads a day from crawlers.

One of the ways I rectified this situation was by changing the crawl rate to a low custom crawl rate of 0.2 crawls per second. This caused a nice big vertical drop in the graph and it alarmed the site owner as he didn't realise that there is no relation between the amount of pages crawled by Google and the sites page ranking or overall search engine optimisation.

Top Tips for getting the best out of crawlers

• Setup a sitemap and submit it to Google, Yahoo and Live.
• Make sure only relevant URLs are put in the sitemap. For example don't include pages such as error pages and logoff pages.
• If you are rewriting URLs then don't include the non-rewritten URL as well as this will be counted as duplicate content.
• If you are including URLs that take IDs as parameters to display database content then make sure you don't include the URL without a valid ID. Taking the site I spoke about earlier as an example, someone had included the following

www.some-site.com/jobview.asp

instead of

www.some-site.com/jobview.asp?jobid=35056

This meant crawlers were accessing pages without content and it was a pretty pointless and careless thing to do.

• Make sure the change frequency value is set appropriately. For example on a jobboard when a job is posted its usually posted for between 7 and 28 days. It only needs to be crawled between once a week and once a month depending on the days it was advertised for. It does not need to be crawled every time so setting a value of always is inappropriate as the content will not change every time Googlebot accesses the URL.
• Avoid circular references such as placing links to a site-index or category listings index in the footer of each page on a site. It makes it hard for the bot to determine the site structure as every path it drills down its able to find the parent page again. Although I suspect the bots technology is clever enough to realise its found a link already spidered and not crawl it again I have heard that it looks bad in terms of site structure.
• Avoid dead links or links that lead to pages with no content. If you have a category index page and some categories have no content related to them then don't make the category into a link or otherwise link to a page that can show related content rather than nothing.
• Prevent duplicate content and variations of the same URL being indexed by implementing one of the following two methods.

1. Set your Robots.txt to disallow your non URL rewritten pages from being crawled and then only display rewritten URLS to agents identified as crawlers.
2. Allow both forms of URL to be crawled but use a CANONICAL META tag to specify that you want the rewritten version to be indexed.

• Ban crawlers who misbehave. If we don't spank them when they are naughty they will never learn so punish those that misbehave. Its very easy for an automated process to parse a Robots.txt file therefore there is no excuse for those bots that ignore the commands set out in it. If you want to know those bots who ignore the Robots.txt rules then there are various ways such as parsing your webserver log files or using a dynamic Robots.txt file to record those agents that access it. There are other ways such as using the IsBanned flag available in the Browscap.ini file however this relies on the user-agent being correct and more and more people spoof their agent nowadays. Not only is banning bots good for your servers performance as it reduces load its good for your sites security as bots that ignore the Robots.txt rules are more likely to hack, spam, and scrape your sites content.

If you are having similar issues with over-crawling then I would advise you to first check your sites structure to see if the problem is due to bad structure, invalid sitemap values and over categorisation first before changing the crawl rate. Remember a sites SEO is unrelated to the amount of crawler activity and more is not necessarily better. Its not the number of crawled pages that counts but rather the quality of the content that is found when the crawlers visit that matters.

Using Google APIS

Creating a content rich site using Google APIs

I recently had a domain renewal notification about a domain I had bought but never got round to using for its original purpose. I didn't want the domain to go to waste so I thought about creating a site as a way for me to play with Googles APIS. They offer a wide range of objects and frameworks which let you add content to a site very quickly such as the ability to search feeds, translate content on the fly from one language to another, search blogs, news and videos and much much more.

Hattrick Heaven

The site I created is a football site called www.hattrickheaven.com its main purpose apart from an example of Googles APIs in action is to display the football league tables of all the countries in the world. I found that on some other sites it was quite a few clicks from the home page to drill down to the league standings so my site was a way to offer this data straight away. As well as the league tables I have links to the latest news, football related blogs, discussion boards and videos.

Google Search APIS

To make use of Googles APIs you just need to setup an access key which is linked to the domain and is free to use. This is passed along in the URI when you reference the main Google JavaScript file. This file is all you need to include as it handles the loading in of all the other libraries with commands such as:

// load google APIS I want to use
google.load("feeds", "1");
google.load("language", "1");
google.load("search", "1");

As well as loading in Googles own framework with this main google.load method you can also reference other well used frameworks such as JQuery, YUI, prototype, dojo, Mootools and others or you can do what I chose to do and reference the relevant framework directly from Googles CDN (Content Delivery Network ).

Targeting Visitors with Googles APIs

One of the very cool features I liked about Googles API is that you get as standard geographical information about your users such as Longitude, Latitude, Country, Region and City. This was great for me as it meant I can use this information to default the search criteria for my feed, blog, news and video searches. If your in the UK and live near a major footballing city such as Liverpool, Manchester or London then HattrickHeaven.com will default the searches with the names of the clubs related to those towns such as Man UTD and Man City for Manchester.

Below is an example from the site of my Visitor object which uses the google.loader object to set up details about the current users location and browser language.

(function(){

V = H.Visitor = {

 sysDefLanguage : "en", // the language I wrote the system in and that transalations will be converted from en=English

 Latitude : google.loader.ClientLocation.latitude,

 Longitude : google.loader.ClientLocation.longitude,

 CountryCode : google.loader.ClientLocation.address.country_code,

 Country : google.loader.ClientLocation.address.country,

 Region : google.loader.ClientLocation.address.region,

 City : google.loader.ClientLocation.address.city,

 BrowserLanguage : (navigator.language || navigator.browserLanguage || this.sysDefLanguage), // the language currently set in users browser

 Language : "",

 isEnglish : false

}

 // set visitors language making sure "en-gb", "en-us" is converted to "en"
V.Language = H.ConvertLanguage(V.BrowserLanguage);
V.isEnglish = (V.Language=="en") ? true : false; // check for English

})();

Translating Content

Another cool feature is the ability to translate content from one language to another on demand. I make use of this on www.hattrickheaven.com to translate the headers and content that is delivered through Googles feed and search objects if the users language is different from that set for the page (at the moment its all English). You can see this in action on a specific page I created http://www.hattrickheaven.com/spanish-news which converts the content from English into Spanish once its been inserted into the DOM. The code to do this is very simple you just pass the text to convert, the language code for the language the text is written in, the language code for the language to translate the text into and a callback function to run once the translation has completed.

google.language.translate(txt, langFrom, langTo, function(result){
// On translation this method is called which will either run the function
// defined in destFunc or set the innerHTML for outputTo 
self.TranslateComplete(result, destFunc, outputTo);
});

On www.hattrickheaven.com I have created my own wrapper object for the site which encapsulates a lot of the various Google options and makes it very easy for me to specify new content to search, translate and output. I have options which control the number of results to show, whether to ignore duplicate URLs and whether to show just the link or to show the snippet of content underneath. For example the following is the code I use on the page

<script type="text/javascript">
// load google APIS I want to use
google.load("feeds", "1");
google.load("language", "1");
google.load("search", "1", H.Visitor.Language);

H.newsPageSteup = function(){
//set up 2 feed objects for the sidebar content
var blog = new H.Feeder();
var wnews = new H.Feeder();

blog.OutputNo = 12; // output 12 links
blog.FeedType = "blogs"; // set the type of feed
blog.getFeedOutputElement = "blogs"; // the node ID to output results
blog.findFeeds(); // find some relevant blogs, translate if necessary and output

wnews.FeedType = "news";
wnews.searchQuery = "World Cup 2010 News"; // overwrite the default search terms
wnews.ShowSnippet = true; // show the content snippet under the link
wnews.OutputNo = 5; // output 5 news articles
wnews.getFeedOutputElement = "worldcupnews"; // where to output news results
wnews.findFeeds(); // run news feed search, translate if necessary and output

// set up a search control to output a search box, paging for results etc
var news = new H.SearchControl();
news.controlType = "news"; // tell object to search for news
news.getFeedOutputElement = "news"; // where to output results in DOM
news.searchSetup(); // run search, translate and output results

// if visitor is not English then I want to translate some headers on this page
if(!V.isEnglish){
 var sobj = new H.Search();
 var arr = ["WorldCupNewsHeader","NewsHeader","BlogsHeader"];
 sobj.TranslateContents(arr);
}

}
// On load run my initialise function
google.setOnLoadCallback(H.newsPageSteup,true);
</script>

As you will see if you take a look at the site its very easy to get some rich content up with very few lines of code. The only issue I currently have is that this functionality is all being done client side with Javascript which leads to two problems.

1. Roughly 10% of visitors (ignoring bots) have Javascript disabled by default. This means that apart from the league tables the site will look pretty bare.

2. Because the content is all loaded in using Javascript its only visible in the DOM after the page has loaded it means that for SEO purposes the source code is going to be pretty empty. I have a few ideas floating around regarding this and I will give more details if any of them come to fruition.

All in all I am pretty impressed with the framework especially its simplicity and hopefully others will feel the same way once they get stuck into developing with it.

Related Links

http://www.hattrickheaven.co.uk/league-listings

http://developer.yahoo.com/performance/rules.html

http://code.google.com/apis/ajaxsearch/samples.html

Optimizing a query with Forced Parameterization

SQL 2005 Forced Parameterization Overview

SQL 2005 has a feature which will take AdHoc queries containing literal values and remove those values replacing them with parameters. This means that the query plan which gets cached can be re-used for similar queries that have different values which can aid performance as it will reduce compilation time. This is called FORCED PARAMETERIZATION however the default mode is SIMPLE which will only cache and re-use AdHoc plans containing the same literal values. This cool feature is useful for systems where stored procs and prepared statements aren't being used as these forms of query will already benefit from cached plan re-use. You should read BOL about the suitability of this feature as certain constructs and features will prevent it from being utilized. It may also result in unsuitable query plans being re-used in certain cases rather than a new plan being generated for a query.


Using Forced Parameterization

I came across this feature on the wonderful SQLServerCentral.com site on a discussion about dynamic SQL. I read up about it and thought it might come in useful on a site that has intermittent problems on a results page when hit by concurrent crawlers which results in high CPU on the SQL Server. The page in question is a results page that allows filtering and ordering by multiple columns, both directions and also uses paging. However the query is an AdHoc query that is built up on the client using literal values that are appended to a string rather than using parameters and something like ADO's sqlCommand object to pass those parameter values to the database.

I looked into rewriting this query to make use of parameters but the WHERE clause was being cached and re-used by multiple queries on the same page so it wasn't a simple change therefore it seemed like a good candidate for FORCED PARAMETERIZATION.


Testing the theory

Before enabling this feature for the live site I wanted to prove on our development box that
a) Simple mode just wasn't good enough and was causing multiple plans to be compiled and cached for this query.
b) Forced mode would enable one plan to be cached and re-used for multiple parameter variations of this query.


Enabling Forced Parameterization

I had some trouble with this due to a number of other sites/blogs showing the incorrect way to set this but the correct way is:

-- enable forced mode
ALTER DATABASE YOUR_DATABASE SET PARAMETERIZATION FORCED

-- enable simple mode
ALTER DATABASE YOUR_DATABASE SET PARAMETERIZATION SIMPLE

You can also use the Management Console and the properties option under the relevant database to switch the setting over manually. When switching from one mode to another I made sure to clear the current cache by running the following:

DBCC DROPCLEANBUFFERS
DBCC FREEPROCCACHE

I then used the following query which makes use of SQL 2005's DMV's (data management views) to check the queries that were currently cached and the type of method being engaged.

SELECT *
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS qt
INNER JOIN sys.dm_exec_cached_plans as cp on qs.plan_handle=cp.plan_handle
WHERE cp.plan_handle=qs.plan_handle
AND (ObjType = 'Adhoc' OR ObjType='Prepared')

Notice that I am filtering on AdHoc and Prepared and ignoring Stored Procs, Triggers and Views. I initially tried filtering by the dbid (database id) column to narrow down the results to my particular database however for AdHoc and Prepared queries this column contains NULL values.

The columns I am particular interested in are the plan_handle which is the ID of the query plan related to the SQL statement and execution_count which tells me how many times the SQL related to the plan has been executed.

A very simple version of the query in question which conveys the point I am trying to make is below.

SELECT * FROM JOBS WHERE Row between 1 AND 20

With SIMPLE mode enabled a test page running my query with different criteria values for my BETWEEN statement caused multiple records to be returned from sys.dm_exec_cached_plans each with a different plan_handle value and the value for ObjType set to AdHoc. This tells me that the plans are not being re-used by similar queries when they could be. Running the same query with the same values does increment the execution_count against the relevant plan record which shows that a very simple form of re-use is being utilized. However as I want the best performance as possible this isn't good enough.

With FORCED mode enabled and the cache cleared the test pages with different criteria for my BETWEEN statement results in a single record to be returned with one plan_handle. The value for ObjType has now changed from AdHoc to Prepared and looking at the start of the text column which contains the SQL I can see the following before the start of my SQL:

(@0 int,@1 int,@2 int,@3 int,@4 varchar(8000),@5 int,@6 int)

Which is the database creating the parameters required to replace the literal values from my query. The values from my BETWEEN clause have been changed from literals to parameters e.g:

--Simple Mode Enabled
WHERE Row BETWEEN 1 AND 20
--Forced Mode Enabled
WHERE Row BETWEEN @5 and @6

Re-running the test page with different values for these parameters causes the execution_count to increment which informs me that the query plan is being cached and re-used across similar queries which is what I wanted to prove.


Enabling Forced Parameterization for a single query

There maybe situations where you want to enable this feature for one or more queries rather than set it on for the whole database. You can do this by setting up a plan guide for the query in question. An easy way to do this is to run the following SQL making sure to replace the first parameter value with the SQL you want to parametrize.

DECLARE @stmt nvarchar(max);
DECLARE @params nvarchar(max);
EXEC sp_get_query_template
N'SELECT * FROM JOBS WHERE Row between 1 AND 20;',
@stmt OUTPUT,
@params OUTPUT;
EXEC sp_create_plan_guide
N'MyPlanGuide_JobResults',
@stmt,
N'TEMPLATE',
NULL,
@params,
N'OPTION(PARAMETERIZATION FORCED)';

You can then set you database to SIMPLE mode but still benefit from having parametrized queries that match the template you have just created in the previous plan. Some useful SQL related to managing query plans is below:

--View all plans in your system
SELECT * FROM sys.plan_guides

--Disable the plan guide.
EXEC sp_control_plan_guide N'DISABLE', N'MyPlanGuide_JobResults';
GO

--Enable the plan guide.
EXEC sp_control_plan_guide N'ENABLE', N'MyPlanGuide_JobResults';
GO

--Drop the plan guide.
EXEC sp_control_plan_guide N'DROP', N'MyPlanGuide_JobResults';

System Tables - sys.processes

Analyse current processes with sysprocesses

The following SQL is based on an article about the sys.processes system view on SQLServerCentral I read today and is another good example of using SQL system views and the new DMV's (Data Management Views).

I have combined some of the example code from the article into a helpful query for analysing your current processes to find long running queries that maybe causing issues with your system. Read the comments within the code for more details.

-- Using the sys.processes system table to find current process details

DECLARE @oldStats TABLE( os_thread_id int, kernel_time bigint, usermode_time bigint)

/* Insert current threads

The KPID is useful in that it helps us tie up what has been passed to the operating system to run commands and is actually working.
Although the SPID is constant throughout the life of the connection a KPID is allocated to each task that needs to be carried out.

The KPID maps back to an actual windows thread and so it is possible using performance monitor to get actual physical statistics
about a task instead of the purely logical statistics which SQL shows through the CPU column.

The KPID is the actual o/s thread id and you can use the "Thread" performance counter using "ID Thread" and "% Processor Time" to
match the thread to the actual cpu stats.
*/
INSERT INTO @oldStats
SELECT os_thread_id, kernel_time, usermode_time
FROM sys.dm_os_threads
WHERE os_thread_id IN (SELECT KPID
FROM sys.sysprocesses
WHERE kpid <> 0
 AND spid>50)

-- wait for 2 seconds
WAITFOR DELAY '0:0:2'

/* Compare previous data to our current processes to see which task are consuming
the most CPU.

If records appear with a KPID of 0 and Physical Time of NULL then it means the
O/S thread is no longer active.

Investigate processes that have high physical times, high CPU, long wait times, and blocked
*/
SELECT sp.KPID, sp.SPID, sp.CPU AS LogicalCPU
,(new.kernel_time + new.usermode_time) - (old.kernel_time + old.usermode_time) AS PhysicalTime
,waittime,lastwaittype,blocked
,blockingSQL = CASE WHEN blocked > 0 AND blocked <> sp.SPID THEN  (SELECT SUBSTRING((SELECT TEXT FROM fn_get_sql(sql_handle)), stmt_start/2,
CASE stmt_end
WHEN -1 THEN LEN(CONVERT(VARCHAR(8000), (SELECT TEXT FROM fn_get_sql(sql_handle)))) - (stmt_end/2)
WHEN 0 THEN LEN(CONVERT(VARCHAR(8000), (SELECT TEXT FROM fn_get_sql(sql_handle))))
ELSE stmt_end /2
END
) FROM sys.sysprocesses WHERE SPID = sp.blocked) ELSE NULL END
,last_batch,open_tran,sp.status,loginame,hostname,cmd
,(SELECT SUBSTRING((SELECT TEXT FROM fn_get_sql(sql_handle)), stmt_start/2,
CASE stmt_end
WHEN -1 THEN LEN(CONVERT(VARCHAR(8000), (SELECT TEXT FROM fn_get_sql(sql_handle)))) - (stmt_end/2)
WHEN 0 THEN LEN(CONVERT(VARCHAR(8000), (SELECT TEXT FROM fn_get_sql(sql_handle))))
ELSE stmt_end /2
END
) FROM sys.sysprocesses WHERE SPID = sp.SPID) as TSQL
FROM sys.sysprocesses SP
LEFT OUTER JOIN
@oldStats old
ON SP.kpid = old.os_thread_id
LEFT OUTER JOIN
sys.dm_os_threads new
ON sp.kpid = new.os_thread_id
ORDER BY PhysicalTime DESC

Store and Analyse Blocked Processes

To view your blocked processes in more detail either set up a loop with a WAITFOR DELAY or an MS Agent job that runs once a minute to log into a table the output from the following SQL. The SQL make use of a recursive CTE to link together all the processes affected by a blocking action which is useful for seeing the action that has caused the blocking and all the processes being affected by the blocking. You can view all databases on the server or filter by a particular database name or partial name.

DECLARE @DatabaseName nvarchar(255) --leave null to use current DB OR 'ALL' For all DBS
DECLARE @PROCESSES TABLE(SPID int, blockingSPID int, databaseName nvarchar(255), programName nvarchar(500), loginName nvarchar(255), ObjectName nvarchar(max), Definition nvarchar(max))
INSERT INTO @PROCESSES
SELECT s.spid, BlockingSPID = s.blocked, DatabaseName = DB_NAME(s.dbid),
s.program_name, s.loginame, ObjectName = OBJECT_NAME(objectid,s.dbid),
Definition = CAST(text AS VARCHAR(MAX))
FROM sys.sysprocesses s
CROSS APPLY
sys.dm_exec_sql_text (sql_handle)
WHERE s.spid > 50 AND
1 = CASE
WHEN @DatabaseName IS NULL AND s.dbid = db_id() THEN 1
WHEN @DatabaseName = 'ALL' THEN 1
WHEN COALESCE(@DatabaseName,'')<>'' AND DB_NAME(s.dbid) LIKE @DatabaseName  + '%' THEN 1
END


;WITH Blocking(SPID, BlockingSPID, DatabaseName, BlockingStatement, RowNo, LevelRow)
AS
(
SELECT s.SPID, s.BlockingSPID, s.DatabaseName, s.Definition,
ROW_NUMBER() OVER(ORDER BY s.SPID),
0 AS LevelRow
FROM @PROCESSES s
JOIN @PROCESSES s1 ON s.SPID = s1.BlockingSPID
WHERE s.BlockingSPID = 0
UNION ALL
SELECT r.SPID,  r.BlockingSPID, r.DatabaseName, r.Definition,
d.RowNo,
d.LevelRow + 1
FROM @PROCESSES r
JOIN Blocking d ON r.BlockingSPID = d.SPID
WHERE r.BlockingSPID > 0
)
SELECT * FROM Blocking
ORDER BY RowNo, LevelRow

Write your own script compressor

Compression Techniques

I have just finished writing my own script compressor to handle the compression on one of my larger systems. You may ask why I didn't just use one of the existing free compressors out there like jsmin or YUI compressor. Well the answer is two fold:

1. I like to write my own code for the simple reason I learn more about my craft doing it this way and although it takes time and usually blood sweat and tears I always come out the other side with more knowledge about development practises in general. See this link about whether to use a framework or write your own for more details.

2. The existing compression tools didn't do all I wanted them to do and also I found certain issues with certain syntax such as complex regular expression literals and conditional comments.

This article is just an overview of some of the things I discovered during the building of my script in case other people want to write their own.

The main features of my compressor are:

• Takes a directory as its source and will output all compressed files to a destination folder replicating the structure of the source folder.
  
• Handles multiple file types e.g JS, PHP, ASP, INC, HTML, CSS
  
• A file such as an ASP file that contains in-line JS, HTML, CSS and ASP will have each "section" compressed according to that sections code type.
  
• The usual compression technique of removing comments, excess white space and empty lines is carried out.
  
• JS code is also compressed by having function parameters and local variables "minified" by replacing variable names with one character names. 
  
• Global objects and common functions are also replaced with short versions.
  
• Debug functions are removed.
• JS functions are saved one per line so even the compressed file is nicely formatted.
• Adds missing terminators ; to aid compression of functions to one line.
• Corrects HTML so that its XHTML valid.
• Skips files that are already compressed.
• Creates a log file detailing the files compressed and the compression rate.

Building a Compressor

I found out during this task that getting a fully working Javascript compressor without the aid of a Java engine is not as simple as it may at first sound. Especially if you wanted like I did each function to appear on its own line and you are handling code that may not be correctly formatted in the first place (e.g missing terminators ;). 

If you want to just remove excess white space and comments then that's fine but doing a "minifier" that renames variables with single letter names is a bit trickier as you need to identify all your functions and parameters correctly for this to work. Remember there are a multitude of ways of defining functions in JavaScript as well so its not just a case of looking for the word function e.g:

function myFunc(var1, var2){ return }

var myFunc = function(var1, var2){ return }

SomeObject.prototype.myFunc = function(var1, var2){ return }

myFunc : function(var1, var2)

(function(){  code }())

Also unless you create a global system that checks which JavaScript functions are being referenced by each file then you can only really minify local function variables and parameters. Otherwise you may change the name of a variable that is being referenced by a file you don't know about causing an error.

If you want to create a very simple minification process you can concentrate on some common global objects such as the window, document and navigator objects. You can also create yourself a small function for document.getElementById and then update all references to that e.g

var _w=window,_d=document,_n=navigator;

//create wrapper function
_g = function(i){
 return _d.getElementById(i);
}

So for every time your site references document.getElementById which is probably quite a lot you have saved yourself 18 characters. Add to that the common use of window and document you will save a fair few bytes just by these changes alone.

Store and Replace

I found the best approach was a layered approach to parsing each file as files such as PHP or ASP will contain a mixture of both server side and client side script as well as HTML and maybe CSS defined in style tags. With the help of some good regular expressions I would look for each type of code using their identifying markers for example look for the open and close style tags. I would then store each block of code in an array, parse it accordingly and then when rebuilding the compressed file re-insert in order. This got round issues where you have ASP script inside in-line JS script blocks within HTML inside an ASP page. Each section would have its own function that compressed according to that languages syntax but all would first store any string and regular expression literals in another array so that they stayed unaffected during any compression as you don't want to be removing white space and changing symbols within string literals.

Fun with regular expressions

Most of my compression was carried out using regular expressions but some tasks such as identifying literals and comments were done in loops. This is pretty easy when your looking for string literals as you know they are going to be enclosed within double or single quotes and you can combine this with a check for single and multi line comments at the same time. However in JS regular expressions can also be defined as literals such as

var re = /^\S\s+[^/].*?>/gi

You cannot just start at the first / and then stop at the next unescaped / as you can use unescaped slashes within character groups e.g [/] so you would end prematurely. You also cannot stop at the last / you come across as you may have multiple statements on the same line such as:

var re = /^\S\s+[^/].*?>/gi, str = str.replace(/##BR##/gi,"<br />");

Plus as you can see the replacement value on the second statement has a forward slash inside it so you would cut off half the replacement value causing a syntax error. I first took the decision that so what a bit more than I intend gets stored as a literal until its put back in but if that extra bit of code also contains variable names that you want to minify you will run into problems.

I got round this problem by first using a regular expression to identify unescaped forward slashes within expressions and replacing them with a placeholder. I could then use another pattern to match string functions such as replace, match, search, split, compile and another for literals and regular expression functions such as test and exec. Once the literal is stored I can put the escaped characters back.

Negative Matches

I also found that the previous technique of using a placeholder value before carrying out a regular expression match was very useful when dealing with complex negative matches. If you have a long string of text and you are trying to carry out a replacement except in a certain instances then this is a good way of having to avoid a complex negative pattern match. For example in my JS compression function I add in extra terminators to the end of lines to make sure I can get a whole function on to the same line. However doing  this sometimes causes issues when terminators are put in places they shouldn't be so at the end I run some corrections which remove terminators from places they shouldn't be such as inside certain brackets. However there are cases where a terminator can appear inside an open bracket legitimately such as a for expression without all the sections. Therefore instead of using a complex negative match to do the excess terminator replacement I use a placeholder e.g

// put a placeholder in for the terminator I want to keep
strJSCompressed = strJSCompressed.replace(/for(;/,"##_FOR_TERM_##");

// carry out the replacement of terminators
strJSCompressed = strJSCompressed.replace(/([\{\(\[,><\|&])(;)/,"$1"); 

Then once the replacements have been carried I put back in all the original values that the placeholders were storing e.g

// put the placeholder back in to my code
strJSCompressed = strJSCompressed.replace(/##_FOR_TERM_##/,"for(;/");

This is a very useful technique when you want to avoid complex regular expressions that involve negative matches as you should know by now complex patterns and long strings combine to cause high CPU!

Another good example is HTML comments. I want to strip all HTML comments apart from the following derivatives:

Server Side Includes e.g <!-- #virtual="/somefile.inc"-->

IE Conditional Comments e.g <!--[if lt IE 7]> OR <![endif]--> 

Server Side META Includes e.g <!--METADATA TYPE="typelib" Blah -->

As you can see trying to write one regular expression that would handle multiple pattern matches within a file that strips all HTML comments apart from those that start with # [ or METADATA would involve some hardcore matching. Its very easy though to match each individual comment type first and replace it with a placeholder, then do my replace for everything between <!-- AND --> and then put the placeholder values back in.

A tidy page is a godly page

As well as carrying all the usual compression functions I also incorporated a number of replacements to tidy up the code. If your going to loop through each page in a system then this seems like a good place to do such things as:

• Make my HTML XHTML compliant by encoding characters, expanding attributes, making sure all attributes are quoted and that my tags are lower case and some other HTML related tweaks.
  
• Removing comments from within SCRIPT blocks as they are not required anymore as well as shortening SCRIPT tags down to the minimum e.g remove the language and type attributes. Obviously this breaks your XHTML compliance but then again you can't have everything.
  
• Combine multi-line string literals together into one variable.
  
• Combine variable declarations (server and client side) into one declaration.
  
• Remove certain function calls such as calls to my custom ShowDebug function that outputs messages for client and server side script. I always build my codebase with the debug statements built in rather than add them in later as it makes debugging quicker and easier. However on a production system these function calls are expensive and unnecessary and should be removed.
  
• Remove excess white space, usually TABS within dynamic SQL strings. Obviously I don't do UPDATES or DELETES only SELECTS but I usually format my code with TABS.

Why Compress Server-Side Code?

Well yes I know that even interpreted languages such as ASP gets compiled into a token based language which is cached by the web server so there is not much scope for compression but the smaller I can make the file size then the better in terms of storage and maybe caching. Plus the main point of the server-side compression was to remove all my ShowDebug function calls to aid performance.

So can I get a copy?

Not at the moment as I am in the process of testing it on a live system to iron out any bugs. At the moment the code is a script that I point at a directory and run. I am hoping to make a C# based windows application version of it and then I might put a copy up on the site.

Should I use a framework?

When to use a framework and when to write your own code

I am always asked at work why I don't just use a framework such as jQuery, prototype, YUI, MooTools etc rather than spend time writing my own code and its a fair point. I have spent time looking at the major frameworks and its all good code written by clever people and if you haven't got the time to spend then I would definitely recommend using a library. Then again if John Reisig had thought like that then millions of people would be using YUI instead of jQuery and Microsoft would be packaging another library with Visual Studio to handle selectors instead.

Libraries are good for many reasons they hide browser incompatibilities from the developer and its good for a team of developers to stick to a standard code base rather than all adding their own functions and bloating a site up with several versions of the same addEvent or toggleClass function.

The downside is that most libraries will contain lots of code that is never even used by the developer. If you're not even going to be using selectors to return DOM objects in your JavaScript and are just looking for a shorter version of document.getElementById then using $('#blah') is not the way to go. 

The other good thing about writing your own code is that you get to understand the language of your trade a whole lot better than if you just relied on a library. There is no better way in my opinion for learning anything that being thrown in at the deep end and having to sink or swim so to speak. Yes it takes much longer as you will have to read up about the early browser wars and compatibility issues, learn about objects and their properties and understand event models and script syntax but it will make you a much better programmer and when bugs appear due to a new version of Internet Explorer you won't have to wait for an update to your framework to be released.

As with all things its swings and roundabouts and just because I like to write my own code and know why things work the way they do does not mean I won't use a library. However having spent the time researching the language for my own code has given me invaluable knowledge and it helps being able to step through something like jQuery and actually understand what its doing and why rather than just knowing that it works.

Problems with Firebug and raised errors

Firebug raising false errors or incorrect line numbers

I have noticed recently that Firebug seems to be giving me unhelpful error reports and it seems to be happening more and more. By unhelpful I mean in the way Internet Explorer used to when an error was raised with an invalid or incorrect line number. I always thought Firebug was pretty spot on for error reporting but it seems that within the last month or so I have been getting quite a lot of errors in the console that say the problem is in file X and on line 120 but in fact after stepping through the problem the problem is actually in file Z and on line 45. Sometimes the error details that are reported in the console do not correspond exactly to those in the standard error console and a lot of the time errors that appear in Firebugs console do not appear in the standard error console.

Possible Causes to spurious error messages

Conflict with Firefox Add-Ons

I have found problems in the past with certain Firefox add-ons that raise errors constantly or intermittently. One such add-on is the user-agent switcher which always raises a 

UserAgentButton is null error and another add on which is a brilliant add-on for Search Engine Optimisation called the SEO Toolbar keeps raising the following error s4fToolbar_cache is undefined.

Both of these errors disappear once the add-on has been disabled or in the case of the SEO toolbar you can turn it off without having to restart your browser. I have quite a number of add-ons on my work PC and due to my experience of errors related to add-ons I wouldn't rule out some sort of clash that is generating incorrect error details. 

Syntax Problems

I have found that sometimes when there are compilation errors usually due to missing or incorrect brackets then the error can be reported incorrectly. If a function hasn't been closed correctly due to a missing bracket then the calling function can sometimes be identified as the erroring function. However a large number of the errors I am getting invalid details for do not meet this criteria.

Firefox or Firebug installations

Within the last couple of months I have had upgrades of both Firefox and Firebug so it maybe that some problem within the application is causing the invalid error details.

If anyone else has found similar issues recently with their Firebug installation it would be good to hear about it. As my colleagues also use Firefox and Firebug but without the various add-ons I have installed do not have these problems it means at the moment I am of the opinion the problem is most likely due to some sort of clash between add-ons as I cannot see such a problem not being spotted and resolved before any upgraded version of Firebug being released. I came to expect this sort of behaviour from IE and it was just understood by everyone that you couldn't get a proper error message when you needed one so its a tad annoying to say the least for me to start experiencing this issue with Firefox.