Compression and Unpacking Javascript

Reverse Engineering the Strictly Compressor with the Unpacker Tool

I have just put up a cut down version of my own compressor tool on my website. The compressor has a number of advanced options which allow you to customise the compression process as well as take care of some very common global objects and function calls such as window and document. If you choose these options the tool will add in aliases for these objects and then replace any references to the object within the code with the alias instead. An example of the compression and then its reverse engineering with the unpacker tool is below.

Example Code

The following code is a dummy script that does nothing apart from show the process at work.

/* A make believe example object
and calling function */
var myObj = {

 myFunc : function(test1,divID){
  // throw in a regular expresion literal
  var re = /^\/firefox.+\d+?/gi;
  var test2 = "IE";
  var some = "vars", anothervar=100;

  var somestring = "this is a string "
    +"that continues "
    +"on a few lines";
    
  if(document.getElementById(test1).value==test2){
   window.open("someurl.htm","width=500");
  }else{
   if(/firefox/i.test(navigator.userAgent)){
    document.getElementById(divID).innerHTML="you are using Firefox";
   }
  }
  return;
 }
}
myObj.myFunc(document.getElementById('div1').innerHTML,"myDiv");

I will run it through the compressor tool selecting the advanced options:

• Minify Global Objects.
• Create a Get function.
• I have used the default value of G for my get function.

The compressed output which is a 37.76% reduction in size is below.

var _w=window,_n=navigator,_d=document;
G=function(i){return document.getElementById(i)}
var myObj={
myFunc:function(a,b){var re=/^\/firefox.+\d+?/gi,c="IE",d="vars",e=100,f="this is a string that continues on a few lines";if(G(a).value==c){_w.open("someurl.htm","width=500")}else{if(/firefox/i.test(_n.userAgent)){G(b).innerHTML="you are using Firefox"}};return}};myObj.myFunc(G('div1').innerHTML,"myDiv");

Notice how the compressor has added the following lines to the top of the code.

var _w=window,_n=navigator,_d=document;
G=function(i){return document.getElementById(i)}

Now if you are sensible and working with lots of compressed scripts you wouldn't want to have these 2 lines in each script and should place them in a central file that is included on all pages so all your scripts can reference them. Removing these two lines gives us a compression rate of 52.11% which if you compare it against YUI ( 39% ) and JSMin ( 27.7% ) is pretty good. Even with these 2 extra lines added to the output we are compressing on the same level as YUI. Larger files will do much better and I have had compression ratios of 60-70% on certain files so far.

Some other things to note about the compressed output are:

• Function parameters have been renamed to use single letters. I don't rename those that are already less than 2 characters in length.
• Local variables are also renamed to use single letters. I start at a and increment up to z and then if needs be into double letters e.g aa to zz and beyond.
• Multiple variable declarations have been combined into one.
• Strings on multiple lines have been joined together.
• Comments have been removed.
• Unneccessary terminators have been removed.
• Global objects window, document and navigator have been renamed to use short aliases.
• Any reference to document.getElementById has been replaced with a call to the new function G.

Now we have a compressed version of the file lets see what happens if we run this compressed code through my unpacker tool. Don't be fooled by the name the unpacker not only unpacks javascript packed with Dean Edwards packer but also reformats compressed code and if you have used my compressor to do the original compression you get an added bonus that it will reverse engineer the Get function and also the global object minification to make the resulting code more understandable.

The unpacked code is below.

var myObj = {
 myFunc: function (a, b) {
  var re = /^\/firefox.+\d+?/gi,
  c = "IE",
  d = "vars",
  e = 100,
  f = "this is a string that continues on a few lines";
  if (document.getElementById(a).value == c) {
   window.open("someurl.htm", "width=500")
  } else {
   if (/firefox/i.test(navigator.userAgent)) {
    document.getElementById(b).innerHTML = "you are using Firefox"
   }
  };
  return
 }
};
myObj.myFunc(document.getElementById('div1').innerHTML, "myDiv");

Notice how the extra code that my compressor put in before has been removed and how any global objects that were being referenced through aliases and the Get function have now been replaced with their original values. Not only does this aid readability, it also helps you understand the codes purpose when you can see the original object name that is being referenced.

Now even if you cannot get your hands on the original uncompressed version of a script you can get a pretty readable version back out from a compressed and minified version with just a few clicks of a button.

Tips for compression

Put minified references to global objects and your Get function in a global file that all your scripts can reference.

When coding your global objects build in a minified name alongside your standard name e.g

var _S = System = { }

The same goes for your important and most frequently referenced functions. You will get most benefit from minifying those functions that you reference constantly throughout your site e.g functions to get and set elements, add and remove events, traverse the DOM etc e.g

G = getEl = function(i){ return document.getElementById(i) }
O = getObj = function(i){ return (typeof i == "string") ? getEl(i) : i }

This is good for multiple reasons including allowing you to reference your objects and functions with both names. For example if the compression fails to rename any references to the longer name then the code will still work as the function can still be referenced. You may also have files elsewhere that have not been through the compression process and this way they will also still work.

Always make sure you correct any syntactical errors and make sure any nested single line IF/ELSE statements are wrapped in brackets to avoid any problems. Although perfectly legal to not wrap single line conditionals in brackets its a lot more readable if each branch is contained correctly. You can use Douglas Crockfords JSLint tool to do this online if you need to.

Write your own script compressor

Compression Techniques

I have just finished writing my own script compressor to handle the compression on one of my larger systems. You may ask why I didn't just use one of the existing free compressors out there like jsmin or YUI compressor. Well the answer is two fold:

1. I like to write my own code for the simple reason I learn more about my craft doing it this way and although it takes time and usually blood sweat and tears I always come out the other side with more knowledge about development practises in general. See this link about whether to use a framework or write your own for more details.

2. The existing compression tools didn't do all I wanted them to do and also I found certain issues with certain syntax such as complex regular expression literals and conditional comments.

This article is just an overview of some of the things I discovered during the building of my script in case other people want to write their own.

The main features of my compressor are:

• Takes a directory as its source and will output all compressed files to a destination folder replicating the structure of the source folder.
  
• Handles multiple file types e.g JS, PHP, ASP, INC, HTML, CSS
  
• A file such as an ASP file that contains in-line JS, HTML, CSS and ASP will have each "section" compressed according to that sections code type.
  
• The usual compression technique of removing comments, excess white space and empty lines is carried out.
  
• JS code is also compressed by having function parameters and local variables "minified" by replacing variable names with one character names. 
  
• Global objects and common functions are also replaced with short versions.
  
• Debug functions are removed.
• JS functions are saved one per line so even the compressed file is nicely formatted.
• Adds missing terminators ; to aid compression of functions to one line.
• Corrects HTML so that its XHTML valid.
• Skips files that are already compressed.
• Creates a log file detailing the files compressed and the compression rate.

Building a Compressor

I found out during this task that getting a fully working Javascript compressor without the aid of a Java engine is not as simple as it may at first sound. Especially if you wanted like I did each function to appear on its own line and you are handling code that may not be correctly formatted in the first place (e.g missing terminators ;). 

If you want to just remove excess white space and comments then that's fine but doing a "minifier" that renames variables with single letter names is a bit trickier as you need to identify all your functions and parameters correctly for this to work. Remember there are a multitude of ways of defining functions in JavaScript as well so its not just a case of looking for the word function e.g:

function myFunc(var1, var2){ return }

var myFunc = function(var1, var2){ return }

SomeObject.prototype.myFunc = function(var1, var2){ return }

myFunc : function(var1, var2)

(function(){  code }())

Also unless you create a global system that checks which JavaScript functions are being referenced by each file then you can only really minify local function variables and parameters. Otherwise you may change the name of a variable that is being referenced by a file you don't know about causing an error.

If you want to create a very simple minification process you can concentrate on some common global objects such as the window, document and navigator objects. You can also create yourself a small function for document.getElementById and then update all references to that e.g

var _w=window,_d=document,_n=navigator;

//create wrapper function
_g = function(i){
 return _d.getElementById(i);
}

So for every time your site references document.getElementById which is probably quite a lot you have saved yourself 18 characters. Add to that the common use of window and document you will save a fair few bytes just by these changes alone.

Store and Replace

I found the best approach was a layered approach to parsing each file as files such as PHP or ASP will contain a mixture of both server side and client side script as well as HTML and maybe CSS defined in style tags. With the help of some good regular expressions I would look for each type of code using their identifying markers for example look for the open and close style tags. I would then store each block of code in an array, parse it accordingly and then when rebuilding the compressed file re-insert in order. This got round issues where you have ASP script inside in-line JS script blocks within HTML inside an ASP page. Each section would have its own function that compressed according to that languages syntax but all would first store any string and regular expression literals in another array so that they stayed unaffected during any compression as you don't want to be removing white space and changing symbols within string literals.

Fun with regular expressions

Most of my compression was carried out using regular expressions but some tasks such as identifying literals and comments were done in loops. This is pretty easy when your looking for string literals as you know they are going to be enclosed within double or single quotes and you can combine this with a check for single and multi line comments at the same time. However in JS regular expressions can also be defined as literals such as

var re = /^\S\s+[^/].*?>/gi

You cannot just start at the first / and then stop at the next unescaped / as you can use unescaped slashes within character groups e.g [/] so you would end prematurely. You also cannot stop at the last / you come across as you may have multiple statements on the same line such as:

var re = /^\S\s+[^/].*?>/gi, str = str.replace(/##BR##/gi,"<br />");

Plus as you can see the replacement value on the second statement has a forward slash inside it so you would cut off half the replacement value causing a syntax error. I first took the decision that so what a bit more than I intend gets stored as a literal until its put back in but if that extra bit of code also contains variable names that you want to minify you will run into problems.

I got round this problem by first using a regular expression to identify unescaped forward slashes within expressions and replacing them with a placeholder. I could then use another pattern to match string functions such as replace, match, search, split, compile and another for literals and regular expression functions such as test and exec. Once the literal is stored I can put the escaped characters back.

Negative Matches

I also found that the previous technique of using a placeholder value before carrying out a regular expression match was very useful when dealing with complex negative matches. If you have a long string of text and you are trying to carry out a replacement except in a certain instances then this is a good way of having to avoid a complex negative pattern match. For example in my JS compression function I add in extra terminators to the end of lines to make sure I can get a whole function on to the same line. However doing  this sometimes causes issues when terminators are put in places they shouldn't be so at the end I run some corrections which remove terminators from places they shouldn't be such as inside certain brackets. However there are cases where a terminator can appear inside an open bracket legitimately such as a for expression without all the sections. Therefore instead of using a complex negative match to do the excess terminator replacement I use a placeholder e.g

// put a placeholder in for the terminator I want to keep
strJSCompressed = strJSCompressed.replace(/for(;/,"##_FOR_TERM_##");

// carry out the replacement of terminators
strJSCompressed = strJSCompressed.replace(/([\{\(\[,><\|&])(;)/,"$1"); 

Then once the replacements have been carried I put back in all the original values that the placeholders were storing e.g

// put the placeholder back in to my code
strJSCompressed = strJSCompressed.replace(/##_FOR_TERM_##/,"for(;/");

This is a very useful technique when you want to avoid complex regular expressions that involve negative matches as you should know by now complex patterns and long strings combine to cause high CPU!

Another good example is HTML comments. I want to strip all HTML comments apart from the following derivatives:

Server Side Includes e.g <!-- #virtual="/somefile.inc"-->

IE Conditional Comments e.g <!--[if lt IE 7]> OR <![endif]--> 

Server Side META Includes e.g <!--METADATA TYPE="typelib" Blah -->

As you can see trying to write one regular expression that would handle multiple pattern matches within a file that strips all HTML comments apart from those that start with # [ or METADATA would involve some hardcore matching. Its very easy though to match each individual comment type first and replace it with a placeholder, then do my replace for everything between <!-- AND --> and then put the placeholder values back in.

A tidy page is a godly page

As well as carrying all the usual compression functions I also incorporated a number of replacements to tidy up the code. If your going to loop through each page in a system then this seems like a good place to do such things as:

• Make my HTML XHTML compliant by encoding characters, expanding attributes, making sure all attributes are quoted and that my tags are lower case and some other HTML related tweaks.
  
• Removing comments from within SCRIPT blocks as they are not required anymore as well as shortening SCRIPT tags down to the minimum e.g remove the language and type attributes. Obviously this breaks your XHTML compliance but then again you can't have everything.
  
• Combine multi-line string literals together into one variable.
  
• Combine variable declarations (server and client side) into one declaration.
  
• Remove certain function calls such as calls to my custom ShowDebug function that outputs messages for client and server side script. I always build my codebase with the debug statements built in rather than add them in later as it makes debugging quicker and easier. However on a production system these function calls are expensive and unnecessary and should be removed.
  
• Remove excess white space, usually TABS within dynamic SQL strings. Obviously I don't do UPDATES or DELETES only SELECTS but I usually format my code with TABS.

Why Compress Server-Side Code?

Well yes I know that even interpreted languages such as ASP gets compiled into a token based language which is cached by the web server so there is not much scope for compression but the smaller I can make the file size then the better in terms of storage and maybe caching. Plus the main point of the server-side compression was to remove all my ShowDebug function calls to aid performance.

So can I get a copy?

Not at the moment as I am in the process of testing it on a live system to iron out any bugs. At the moment the code is a script that I point at a directory and run. I am hoping to make a C# based windows application version of it and then I might put a copy up on the site.