Saturday 31 August 2013

Displaying Apache Server Status HTML

Display Your Apache Server Status As HTML Webpage

Diagnosing and fixing Apache issues can be a fucking nightmare (excuse the french). However one thing you might want to do is allow yourself access to view key Apache stats on a webpage rather than have to SSH in and use the console.

If you read some articles on the web to set this up it's as simple as adding the following lines to your main apache2.conf or httpd.conf file.

Depending on your server you will need to go into /etc/apache2/apache2.conf or etc/httpd/conf/httpd.conf and then adding the following lines.

# Allow server status reports, with the URL of http://dv-example.com/server-status
# Change the ".dv-example.com" to match your domain to enable.
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from mywebsite.com

Or if you wanted to only allow your own IP address to access the page you could do something like this.

# Allow server status reports, with the IP addresss you want to allow access
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from 86.42.219.12

However when I tried either of these approaches I was just met with an error message:

Forbidden

You don't have permission to access /server-status on this server.




The Fix
At the bottom of my main apache2.conf file I noticed the lines
# Include generic snippets of statements
Include /etc/apache2/conf.d/

Therefore I went into that folder and was met with 3 other files:

apache2-doc
charset
security

The top of the security file had these lines.


# Disable access to the entire file system except for the directories that
# are explicitly allowed later.
#
# This currently breaks the configurations that come with some web application
# Debian packages. It will be made the default for the release after lenny.
#
#
# AllowOverride None
# Order Deny,Allow
# Deny from all
#

So it seems that the main file was loading in these other files and the security file was blocking access to all other directories. By adding my rule to the top conf file the security file just over-ruled it again which caused the 403 error. Therefore I added the rules to the bottom of the security file and hey presto it worked.

Now I can access one of my domains to find my apache info. However I found that due to most of my sites on this virtual server using WordPress their ISAPI rules prevent a simple mysite.com/server-status from working.

WordPress is obviously trying to resolve the URL to a page, post or category list.

Therefore if you have got this problem you might need to use a domain not full of rules or create a rule that bypasses the WordPress guff.

Once you get it working you will get Apache info such as the following.
Current Time: Saturday, 31-Aug-2013 08:28:06 BST
Restart Time: Saturday, 31-Aug-2013 08:15:13 BST
Parent Server Generation: 0
Server uptime: 12 minutes 53 seconds
Total accesses: 622 - Total Traffic: 4.2 MB
CPU Usage: u26.65 s3.24 cu0 cs0 - 3.87% CPU load
.805 requests/sec - 5.5 kB/second - 6.9 kB/request
3 requests currently being processed, 4 idle workers

Plus lots of info about current HTTP requests, CPU usage and session info.

It is well worth doing if you need info about your server and you are away from your SSH console.

Thursday 22 August 2013

Handle jQuery requests when you want to reference code that hasn't loaded yet

Handle jQuery requests when you want to reference code that hasn't loaded yet

As you should be aware it is best practise to load your JavaScripts at the bottom of your HTML for performance and to stop blocking or slow load times.

However sometimes you may want to reference an object that is not yet loaded higher up in the page.

If you are using a CMS or code that you cannot change then you may not be able to add your event handlers below any scripts that maybe needed to use them. This can cause errors such as:

Uncaught ReferenceError: $ is not defined 
Uncaught ReferenceError: jQuery is not defined

If you cannot move your code below where the script is loaded then you can make use of a little PageLoader function that you can pass any functions to and which will hold them until jQuery (or any other object) is loaded before running the functions.

A simple implementation of this would involve a setTimeout call that constantly polls the check function until your script has loaded.

For example:


PageLoader = { 
 
 // holds the callback function to run once jQuery has loaded if you are loading jQuery in the footer and your code is above
 jQueryOnLoad : function(){}, 

 // call this function with your onload function as the parameter
 CheckJQuery : function(func){
  var f = false;

  // has jQuery loaded yet?
  if(window.jQuery){
   f=true;
  }
  // if not we store the function if first time in loop otherwise and set a timeout
  if(!f){
   // if we have a function store it until jQuery has loaded
   if(typeof(func)=="function"){    
    PageLoader.jQueryOnLoad = func;
   }
   // keep looping until jQuery is in the DOM
   setTimeout(PageLoader.CheckJQuery,200);
  }else{
   // jQuery has loaded so call the function
   PageLoader.jQueryOnLoad.call();    
  }
 }
}


As you can see the object just holds the function passed to it in memory until jQuery has been loaded in the DOM. This will be apparent because window.jQuery will be true.

If the object isn't in the DOM yet then it just uses a setTimeout call to poll the function until it has loaded.

You could increase the length of time between the polls or even have a maximum limit so that it doesn't poll for ever and instead after 10 loops returns an error to the console. However this is just a simple example.

You would call the function by passing your jQuery referencing function to the CheckJQuery function like so.


<script>
PageLoader.CheckJQuery(function(){
 $("#myelement").bind('click', function(e) { 
  // some code
  alert("hello");
 });
});
</script>


It's just a simple way to overcome a common problem where you cannot move your code about due to system limitations but require access to an object that will be loaded later on.

Thursday 15 August 2013

MySQL Server Has Gone Away Fix For Wordpress 3.6

MySQL Server Has Gone Away Fix For Wordpress 3.6

I have been constantly having issues with the bag of XXXX that is Wordpress and I really, really hate relying on someone else's code especially when there is so much I would do to improve it.

Adding new indexes and writing my own plugins for heavy task can only do so much to tweak the performance of a 3rd party CMS system.

However one of the problems I have been having lately is the dreaded "MySQL Server Has Gone Away" error littering my error log.

The symptoms I have been getting include:

  • Trying to load a page but it just spinning away.
  • Checking the console with a TOP command to see very low server loads (0.00 to 0.03)
  • No Apache / MySQL processes running.


I first thought I had solved the problem some months back when I de-activated the Apache Caching plugins my server was using.

I did this because of the high number of Apache related errors in the error log files like this:

[Thu Feb 02 16:30:57 2012] [error] (103)Software caused connection abort: cache: error returned while trying to return mem cached data


These errors related to the times I was getting the slow page loads, high disk swapping and low server loads.

Apache was obviously causing me issues with it's own caching and a restart always fixed it.

As I was using WP Super Cache there was no need for duplicate caching and there are far too many levels on a LAMP set-up where caching can be enabled.

For me removing the Apache caching seemed to fix the issue, for a while at least.

However I keep getting intermittent issues where the same symptoms are present except instead of Apache errors in the error log I am getting lots of MySQL Server Has Gone Away errors like this:

[Wed Aug 14 23:37:29 2013] [error] [client 173.203.107.206] WordPress database error MySQL server has gone away for query UPDATE wp_posts SET robotsmeta = '' WHERE ID = 42693 made by WPOMatic->runCron, WPOMatic->processAll, WPOMatic->processCampaign, WPOMatic->processFeed, WPOMatic->processItem, WPOMatic->insertPost, wp_insert_post, do_action('wp_insert_post'), call_user_func_array, RobotsMeta_Admin->robotsmeta_insert_post


Therefore I looked into the MySQL Server Has Gone Away error and came across the Robs Note Book workaround for WordPress.

The fix he uses is pretty simple and involves using a custom wp-db.php file for your WordPress installation.

As this is a core file that handles all database queries it is a bit annoying in that it may need constant updates when new versions come out.

As the highest version of the workaround on his site is for WordPress 2.8.1 and I was on WordPress version 3.6 I had to create my own workaround for the file.

However the fix is pretty simple to implement and just involves using a new function with a number of retries for the failed query with a server re-connect between each loop iteration.

Basically all the fix does is something I do in my own projects many times when I encounter, lock timeouts or deadlocks e.g retry the query X number of times before quitting with an error.

You can see one such example I use to handle LOCK TIMEOUTS in MS SQL here.

Therefore if you need to apply the same fix in a future version of WordPress you can just follow these steps yourself. Download the existing copy of wp-db.php from the wp-includes folder, back it up and then make a copy before applying these changes.

1. In the db_connect() function that attempts to connect to your database ensure that the initialquery flags are set on and then off are placed around the first query run on the class initialise. This happens to currently be the $this->set_charset function which sets the connections correct character set.

The code should be wrapped around this function call and also before the $this->select function which selects the database to use.

$this->initialquery=1;

$this->set_charset( $this->dbh );

$this->ready = true;

$this->initialquery=0;

$this->select( $this->dbname, $this->dbh );


2. You need to create the following query and put in the file somewhere. This query attempts to run the query and then retries a number of times with a re-connect to the database in-between.

This is what handles the "MySQL Server Has Gone Away" error as it re-connects if the connection is no longer present.

You can change your retries to any number you want. I use 3 re-attempts as I don't see the point of any further retries as if you can't re-connect to your database after 3 goes you certainly have an issue.

function queryWithReconnect($query)
{
 // set this to the number of re-attempts you want to try
 $maxcount = 3;
 $cnt = 1;

 // loop until we reach our $maxcount value OR we get a good query result
 while($cnt < $maxcount)
 {
  // close our connection and then re-connect to our database - this is what fixes the MySQL Has Gone Away error
  // as if it has gone away we are re-attempting the connection.
  @mysql_close($this->dbh);

  // re-connect to our database
  $this->db_connect();

  // re-run our query and store the result in a global variable
  $this->result = @mysql_query($query, $this->dbh);
  
  // if we dont have an error from the server quit now, otherwise carry on!
  if (!mysql_error($this->dbh))
  {
   // return 0 so we know the query ran ok
   return 0;  
  }

  // if we are here we had an error so increment our $cnt loop counter
  $cnt+=1;
 }
 
 // we have looped up to our $maxcount number and all attempts at running the query failed so quit with a failure code!
 return 1;  
}


3. In the main query() function which runs ALL queries in WordPress you need to ensure that the call to our new function that re-attempts the query 3 times is placed just after the initial attempt at running the query.

Find the code that checks for a MySQL error and place our call to the new function inside it and above the code that clears any insert_id that may have been stored.

// If there is an error then take note of it..
if ( $this->last_error = mysql_error( $this->dbh ) ) {
 
 // If its the first initial query e.g in the db_connect() function OR we have a MySQL error then call our queryWithReconnect
 // function until it either passes OR fails after X attempts.

 if (($this->initialquery)||($this->queryWithReconnect($query)!=0)) {
  
  // Clear insert_id on a subsequent failed insert.
  if ( $this->insert_id && preg_match( '/^\s*(insert|replace)\s/i', $query ) )
   $this->insert_id = 0;

  $this->print_error();
  return false;
 }
}


Since putting this re-try code in my wp-db.php file I have had no MySQL Gone Away Errors but it is too early to tell if my main issue of low server loads, no processes running and no website activity is solved yet.

However even if it hasn't this is a good trick to use for your own WordPress code.

If you want to download the latest version of the wp-db.php workaround file which works with WordPress 3.6 then you can get it from the link below.

Just re-name it from wp-db.txt to wp-db.php and then copy it into the wp-includes folder of your site. Be sure to make a backup first in-case it all goes tits up!

Download WordPress 3.6 Fix For MySQL Server Has Gone Away Error - wp-db.php

http://www.strictly-software.com/scripts/downloads/wp-db.txt

Wednesday 14 August 2013

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

To celebrate it's 10th birthday the worlds most infamous censored site, The Pirate Bay, has introduced its own Internet browser to enable people to access its website even if it's being blocked by your ISP.

Most big ISP's have blocked The Pirate Bay for its users claiming it breaches copyright by allowing people to download Torrents of films and music.

If you don't know what a torrent it it's a movie split into thousands of small pieces. Each piece is stored on various computers so that each user is not in theory holding a full version of a film that may be breaching copyright. When you download  the torrent the files are all put together and downloaded from their various locations.

However most ISP's still see this as copyright violation and even though The Pirate Bay is just like Google in the fact they are just a search engine and don't actually host the films or music they have been attacked from all quarters.

Is Your ISP Blocking You?

You can quickly test whether you are being blocked by your ISP by clicking these links which all point to the Pirate Bay Website.

http://thepiratebay.org

http://thepiratebay.sx

https://piratereverse.info

Check More Pirate Bay Proxies

If you want a PHP script that will scan a number of known Pirate Bay Proxies then you can download this one I quickly knocked up from here: Pirate Bay Proxy Checker Script.

Just change the file extension to .php and either run it from your local computer or server. If you are running Windows I recommend downloading WAMP so that you can run an Apache server on your PC.

I also recommend changing the port number the server runs on to 8080 or 8888 so it can run side by side with IIS. This article will explain how to set WAMP up.


Use Pirate Bay Tor Browser

If you are being blocked from the Pirate Bay by your ISP then you have a number of choices.

1. Find a working mirror to the site - e.g search for Pirate Bay Proxies on DuckDuckgo.com or try this page: http://torrentproxies.com/ to find some not blocked by your ISP. Many mirrors will first open up in something called adf.ly with a JavaScript countdown. After a few seconds a link will appear saying "click here to continue". On clicking the page will change (usually to an advert with a blue bar at the top) and in the top right corner will be another countdown saying "Please Wait". Usually after 5 seconds it will say "SKIP AD" and on clicking it you will end up at the Pirate Bay mirror site.

2. Use a proxy to access the Pirate Bay. There are many free proxies out there and you can find a good list at http://nntime.com/proxy-list-01.htm. Use a tool like FoxyProxy to manage your proxy lists and toggle between them. Or you can go into your browsers network / proxy settings and manually enter the IP address and port number you want to use.

3. Download the new Pirate Bay Browser or the Mozilla TOR Browser. Both of which access the TOR network to help disguise your internet footprint by bouncing your HTTP requests through a series of servers.  As the TOR website explains.


"Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location. Tor works with many of your existing applications, including web browsers, instant messaging clients, remote login, and other applications based on the TCP protocol."


With all the outrage surrounding the NSA spying on ALL American citizens and the recent revelations that they are using this information not only to catch terrorists but to catch petty criminals, drug dealers, tax avoiders and to spy on and blackmail politicians and judges it is in everyone's best interests that they tighten up on their Internet Security.

As this article I wrote explains there are a number of measures you can take to reduce your Internet footprint and whilst you might be totally invisible you can blend into the background.

So do yourself a favour and use the Pirate Bay browser or the TOR Mozilla Browser. Not only does making use of the TOR network help protect your privacy but the Pirate Bay Browser will let you access a number of Pirate Bay Mirror sites and bypass any censorship that your ISP or country may have introduced.

Download the TOR Browser


Download the TOR Browser


Once installed when you open the .EXE you will connect to the TOR network.

Joining the TOR network

Once connected you can use the Firefox browser to surf the web. All your Pirate Bay Mirror sites and other Torrent sites are linked at the top in the icon bar. Pick one and then carry out your search for movies, applications or other content.

As you can see it's just a search engine like Google which makes it very unfair that the Pirate Bay is being blocked by ISP's as Google also indexes illegal content such as porn and copyrighted material.

This is a search for the US TV show Dexter.

The Pirate Bay Browser

Like most search engines once you run a search you get your results.

The Pirate Bay Search Results

As you can see from the results on the right there are two columns, seeders and leachers.

You want to choose an item with as many seeders (uploaders) and as few leachers (downloaders) as possible to get a quick download.

You can also tell from the names of the file what sort they are e.g if the file has the word CAM in it then it's a poor quality camera in a cinema job. If it's HDTV quality it will be a bigger better quality file and BRRip is a BluRay copy.

Clicking on the file you want will open up the result page. Hopefully you will get comments telling you the quality of the file. Click on "Get This Torrent" to download your torrent file.

Downloading a Torrent from the Pirate Bay

When you want to download a torrent from the Pirate Bay you will first need to have a torrent client. There are many out there including:
www.utorrent.com/downloads/win

bitlord.soft32.com/

www.bittorrent.com/
Once you have one installed your application of choice you can just click on the torrent in your search results and it will open up in the client and start downloading.

As you can see whilst you download the file you will also be uploading at the same time. This way you are not just taking without giving back the files you have already leached. You can control the bandwidth ratio between upload and download as well as set limits on the rate you upload.

Downloading a Torrent in UTorrent

However bewarned, ISP's are also on the lookout for people downloading (leaching) and uploading (seeding) torrents so you need to protect yourself.

Some of the ways include:
-Using someone else's WIFI or network.

-Using an anonymous proxy that doesn't leak identifying information or even better a VPN. A paid for tool like BTGuard which is both a proxy and encryption tool which helps prevent your ISP throttle your traffic.

-Using a block list which will mean that all traffic is bounced around ISP routers so that they cannot scan the traffic.

-Setting your download port to a common port for other traffic such as 80 or 8080  (HTTP) so that it doesn't look obvious you are downloading P2P data. Most people use a "random port" but it is better to look like you are downloading HTTP content especially if you are encrypting your traffic as well.

-Limiting your upload rate to a minimum. Although this violates the spirit of P2P (sharing) the people going after you for stealing copyrighted material are more interested in those spreading (uploading) the content rather than downloading it.

-Ensuring you force outgoing traffic to be encrypted. This will help prevent your ISP see that you are using BitTorrent traffic and may prevent them throttling your bandwidth.

-Set a download cap on your traffic. Even if you are encrypting your traffic some ISP's may see the amount you are downloading and throttle it if they think you are up to no good.

-Using an application like PeerBlock which will block traffic from known bad IP addresses such as P2P blocklists, known bad IP addresses, spyware, FBI and copyright monitoring sites and so on. It is worth downloading and just sits in the background running as you do you're downloading.
As with most security measures a wide range of different measures is much better than just relying on one system.

Hopefully this article will help you make use of The Pirate Bay if you have been blocked so far from accessing it.