Debugging Memory Problems on Wordpress

High Memory Consumption on Wordpress

This year I have taught myself PHP and setup a couple of Wordpress blogs to get my head round the sort of code PHP developers hack together. Although PHP has a lot of good qualities such as the variety of functions compared to other scripting language such as ASP, the ability to write object orientated code and the amount of available help on the web it does have some downsides which include the ability for poor coders to write poor code without even realising it.

I won't go into a diatribe about coding practice as there will always be good coders and bad coders but my main problem with Wordpress is that although rich in functionality and easy to use the code behind the scenes does not lend itself well to performance.

My current blog is getting about 1000 human visitors a day and four times as many bots. It's not a big site and runs on a LINUX box with 1GB RAM which should be enough especially since I have already helped my performance by doing the following:

-Adding the Super Cache Plugin which GZips up content

-Tuning the MySQL and PHP configuation files

-Banning over 40% of undesirable bot traffic using MOD Rewrite

-Disabling unused plug-ins and comparing new ones before installing them so that performance is the key factor I use when deciding which one to use.

I am also looking into using a PHP accelerator which will store the complied version of the source code so that it's not re-interpreted on every load.

However I am still experiencing intermittent memory issues and the problem is trying to detect the source of them. Earlier today I added some debug functions to the Wordpress source code that outputted the Memory_Usage value at key points. The results are quiet shocking as it shows that when the home page is loaded it consumes 24MB of memory!

The file in question is located in your root directory wp_settings.php.

The output is here:

START OF SETTINGS FILE PHP

Memory Limit: 128M - Current Memory Usage: 768 kb - Peak Usage: 768 kb

Before Require Compat, Functions, Classes - Current Memory Usage: 1.25 mb

After require_wp_db - Current Memory Usage: 3 mb

Before Require Plugin, Default Filters, Pomo, l10n - Current Memory Usage: 3 mb

After Required Files loaded - Current Memory Usage: 4 mb

Before Requires of main files Formatting, Query, Widgets, Canonical, Rewrite and much more 30 files! - Current Memory Usage: 4.5 mb

After ALL Requires - Current Memory Usage: 15.25 mb

Before muplugins_loaded action fired - Current Memory Usage: 15.25 mb

Before require vars - Current Memory Usage: 15.25 mb

Before create_initial_taxonomies - Current Memory Usage: 15.25 mb

After create_initial_taxonomies - Current Memory Usage: 15.25 mb

Before include active plugins - Current Memory Usage: 15.25 mb

Before require pluggable - Current Memory Usage: 21.5 mb

Before wp_cache_postload and plugins_loaded - Current Memory Usage: 22.25 mb

After ALL Plugins loaded and action fired - Current Memory Usage: 22.75 mb

Before action sanitize_comment_cookies - Current Memory Usage: 22.75 mb

Before create global objects WP_Query WP_Rewite WP and WP_Widget_Factory - Current Memory Usage: 22.75 mb

Before action setup_theme - Current Memory Usage: 22.75 mb

Before wp-init and do_action(init) - Current Memory Usage: 23.75 mb

End of Settings - Current Memory Usage: 24 mb

As you can see the inclusion of 30 core include files that Wordpress loads on every page adds a wopping 10MB in one go is a major factor in the large size as well as the loading and activation of other plug-ins.

If I didn't have Caching enabled then you can imagine how much trouble I would be in. If I was suddenly hit with a spike in traffic of only 42 concurrent users my 1GB of RAM would have been eaten up straight away. Plus in reality it wouldn't even need 42 users as not all my RAM will be consumed by PHP / APACHE and we need to factor in all the MySQL queries that run behind the scenes.

In fact from using the Wordpress function get_num_queries() which logs each query run by the wp_db->query() method it shows that my home page makes 32 calls to the database!

Now in no developers world should these be good stats to report on and it just goes to show the sort of battle someone has to fight when making a Wordpress blog run under high loads on a small budget. Yes throwing resources at the problem is a form of answer but a better one is to resolve the underlying issues. The problem is that it's very hard to with someone else's code.

One of the big downsides to using other peoples code is that when it goes wrong you are big trouble. You can either wait for an upgrade from the developer which may or may not come within any sort of agreeable timescale or you try to make sense of the code they wrote. The issue with the core Wordpress codebase is that if you start hacking it about then you leave yourself in a position of having to ignore future updates or having to redevelop it yourself for time memorial.

I don't know which of these two undesirable options I am going to take but I know what I would be looking at if I had to redevelop the wordpress system.

1. I would most definitley consider running it from a different database instead of MySQL which has a lot of good features but also a lot of configurable options I don't want to have to worry about.

If I am joining a lot of records together in a GROUP_CONCAT or CONCAT I don't want to have to worry about calculating the size of the group_concat_max_len first I just want all my string to return.

Also as well as all the missing DML such as CTE's I really miss the very useful Data Management Views that SQL Server 2005-2008 has as they make optimising a database very easy. I have seen a lot of Plugins that use very poor SQL techniques and some time spent on proper indexing would speed things up a lot. Having to sift through the slow query log and then run an EXPLAIN on each one is a time consuming job whereas setting up a scheduled job to monitor missing indexes and then list all suggestions is very easy to do in MSSQL.

2. I would definitely change the database schema that Wordpress runs on and one of the major changes would be to horizontally partition the main wp_posts table as this is referenced a hell of a lot by plug-ins and other internal Wordpress code and the majority of the time the wide columns such as content and excerpt are never required. A narrow table containing the post id, post type, modified date and maybe the post title would help things a lot.

3. All of the queries run in the system are single executions. This means extra network traffic and extra resources. Returning multiple recordsets in one hit would save a lot of time and I know it's hard in a CMS to combine queries but with some clever thinking it could be done.

4. A lot of performance problems come from calling the same function multiple times within loops rather than calling it once, storing the value and then re-using it. A good example is all the calls to get the permalink structures when creating links.

5. All the checks to see whether functions, classes and constants exist must be consuming some resources. It is hard in a CMS to know what has and hasn't been included but maybe some of these could be left out.

6. Caching should be built into the Wordpress core code as standard. When new posts and pages are saved there should be the option to create a physical hard copy of the html instead of using the usual dynamic page. Any sidebar functionality could also be calculated and added there and then as does it really matter if a tag or category cloud is slightly out of date or do links to other blogs and sites really need to be loaded from the DB every time? The answer is obviously no.

Anyway those are just six suggestions I have just conjured up through a cursory examination of the codebase. I am sure there are lots more potential ideas and I see from reading up on the web that a lot of people have experienced similar issues with performance since upgrading to Wordpress 2.8.

If anyone else has some suggestions for reducing the memory footprint of a Wordpress blog please let me know as I really don't like the idea of a simple blog using up so much memory.

Problems with LINUX, Apache and PHP

LINUX Apache Server stopped serving up PHP pages

By Strictly-Software

When I logged into my hosted LINUX web server earlier tonight I was met with a message saying I should install a number of new packages.

I usually ignore things like this until it gets to a point where someone forces me to do purely for reasons that will shortly become obvious.

The packages were the following:

• apache2.2-common
• apache2-mpm-worker
• apache2
• usermin
• libapt-pkg-perl
• apt-show-versions
• webmin
• webmin-virtual-server

I have no idea what most of them do but they had been sitting around for a long time waiting for me to install them and tonight was the night. These are always nights I dread!

Shortly after doing the updates I noticed that my WordPress sites had stopped working and all the PHP files were being served up as file attachments with a content type of application/x-httpd-php instead of being parsed, executed and then delivered as text/html.

At first I thought it was something to do with the SQL performance tweaks I was doing but I soon remembered about the updates and I went off hunting the web for a solution.

It's nights like these that make me wish I was back doing carpet fitting, finishing the day at 3 pm and then going down the pub. Much more enjoyable than spending Friday nights scratching my head wondering what the hell I had done to bring down my websites at 3 am.

To cap off the nightmare I had just spent ages writing a detailed message to post on an APACHE web forum only for my session to timeout and then for the site to refuse to log me in.

They then decided to block me for failing 5 login attempts in a row. Obviously I couldn't get back my message so I was pretty pissed right now!

For some reason APACHE had stopped recognising PHP file extensions and I still don't know what had happened under the covers but after a long hunt on Google I came across the solution.

The libapache2-mod-php5 module had somehow disappeared so I had to re-install it with the following lines:

sudo apt-get purge libapache2-mod-php5
sudo apt-get install libapache2-mod-php5
sudo a2enmod php5
sudo /etc/init.d/apache2 restart

I also added the following two lines to /etc/php5/apache2/php.ini

AddHandler application/x-httpd-php .php
LoadModule php5_module modules/libphp5.so

I then cleared my browser cache and low and behold my site came back.

Maybe this info might come in handy for anyone else about to upgrade packages on their server or serve as a reminder of what happens when you try to behave like a sysadmin and have no real idea what your doing!

It also should make you glad that we live in the days where a Google search can provide almost any answer you are looking for. I doubt I would have owned, let alone found, a book that would have been of any use at 3am on a Saturday morning.

So despite all their snooping, minority report advertising and links to the alphabet agencies they are good for something.

Tune Up Utilities 2010

Tune Up Utilities 2010

I don't usually promote products other than the odd Google advert and in reality that seems to be a waste of time considering I make literally pennies. However one of the products that I do admit to liking and help advertise is the TuneUp Utilities application which takes the best features of all the other clean up and performance tuning apps that are available and puts them into one nice easy to use interface.

You have all the features that other tune up tools such as CCleaner, Comodo and Defraggler have and a lot more. It also combines network and browser optimisations that well known plugins such as FireTune use as well as TCP/IP optimisers such as the DrTCP. It also puts all the important windows configuration options that manage memory, errors, start ups, background processes, programs, display settings and a whole lot more into the same application which means you should never really need another tune up utility once this has been installed.

A break down of some of the features:

• Disk Defragmenter
• Disk Cleaner
• Registry Cleaner
• Memory optimiser
• TCP / IP optimiser
• Internet Browser optimiser
• File Shredder
• Program manager (add / remove programs, shortcuts, start ups etc)
• Windows Settings manager
• Process manager
• One Click maintenance
• Maintenance Scheduler

It also has some features to increase your computers security, manage network traffic and important computer configuration options that can help improve performance. In fact it seems to have wrapped up all the important settings that you would ever need or want to change in a very nice user interface.

If you don't want to spend time going through the many levels of available settings there is the easy option of "One Click maintenance" that will check your system for potential issues and then offer solutions in an easy to understand dialog.

Another feature I quite like, which is probably more to do with the name than any perceived benefit it may give is "Turbo Mode". This is a big round button that you can press when you want a performance boost and it basically just ensures that the process your working on at the time gets the necessary resources and disables CPU and memory hogging processes such as disk de-fragmentation.

The price of the application is £29.99 but you can install it on up to 3 computers and this is a small price to pay for the convenience of having every feature wrapped up in one easy to use interface.

If you are looking for a program that can aid your PC's performance then I would suggest taking a look at Tune Up Utilities first but if you are not interested in paying for your tools and are willing to spend the time tweaking your settings manually then have a read of the following articles which discuss PC, Network and browser optimisation techniques:

http://blog.strictly-software.com/search?q=PC+tuning

Are you a spammer

Spam Checker Tool

If you post comments on blogs and sites around the web you might find that your time and effort is being wasted especially if you have been marked as a comment spammer.

Most blogs and sites that accept comment use a tool called Akismet to check whether a comment is spam or not. It's a free service that comes with a simple API that allows users with a key to check content and user details for spam. The plugin is easy to use and many popular blogging tools use it to check comments for spam.

The reason it's so popular is that it uses a variety of methods to identify spam such as using feedback from site owners who mark spam and ham that get through the automatic filter so that other users gain the benefit of a large community analysis. On top of that the automatic filtering uses the usual blacklists of known spammer IP addresses, bad emails, dodgy websites and also the usual content analysis that ensures your comment section doesn't get filled with adverts for Viagra.

So the service is well worth using if you want to block spammers filling your comment section up with bullshit replies and backlinks. However if you are a "commenter" and you're worried that you might be getting blocked from posting due to being unfairly marked as a spammer then the best way to find out is to use the same service yourself to see if it regards you as a spammer.

Therefore I created this Spam Checker Tool today which allows you to check whether your details (Name, Website, Email, IP) are blacklisted by Akismet as well as checking out a potential comment before you post it.

Just complete the fields you want to test and hit submit. The site will then report whether Akismet regards your content as spam or not.

All fields are optional but at least one value must be supplied to perform a spam analysis.

www.strictly-software.com/comment-spam-test

Note:

If Akismet clears your good name and doesn't regard you as spam it doesn't automatically mean that your comment will be posted.

Most blogging software including Wordpress allows the administrators to set up their own rules regarding comment spam. These rules usually consist of blacklists of words and a number of links to count therefore keep your comments free of profanity and the number of links down to avoid being flagged for review.

Most of all make your comment relevant to the post your commenting on. Auto posting generic comments such as "Great post, keep it up" and "I found this site is it also yours" will not get you very far unless all filtering has been disabled.

Strictly AutoTags Wordpress Plugin

Wordpress Plugin - Strictly AutoTags

I have been doing a lot of work with Wordpress lately and have been working on a number of custom plugins that I already use with my own Wordpress blogs. One of these plugins that I find very useful is AutoTags which automatically detects words within new posts to use as tags.

Unlike other smart tag plugins that add lots of functionality to manage existing tags this plugin only attempts to find relevant and useful words within posts to use as tags by utilising the power of regular expressions and some simple logic to determine which words and sentences are relevant to the post.

I have been using this plugin on a couple of my own blogs for a while now and as well as helping me build up a nice big list of taxonomies it's not overly complicated in what it does so it doesn't throw any curve balls my way very often.

My posts are usually quite lengthy so I choose to set my MaxTag option to five which means that only the five most relevant tags are added and single or double occurrences are usually skipped which helps make the tags that are added a lot more relevant to the content.

Keeping in the spirit of all things open source I thought I would release the code to the public. It's my first Wordpress plugin and I have only just learned PHP so I cannot claim it to be a work of perfection or anything but I would like to hear feedback from anyone who uses it.

Also if any PHP or Wordpress developers do take a look at the source code I would be interested to hear any advice on the format the plugin takes. I started using a template I downloaded from the web but it wasn't very encapsulated so I decided to put the whole thing in a class. I don't know if this is the best way or not so please share your experience.

You can check out the plugin either at my own site - Strictly AutoTags or at the plugin directory hosted by Wordpress.