Debugging Memory Problems on Wordpress

High Memory Consumption on Wordpress

This year I have taught myself PHP and setup a couple of Wordpress blogs to get my head round the sort of code PHP developers hack together. Although PHP has a lot of good qualities such as the variety of functions compared to other scripting language such as ASP, the ability to write object orientated code and the amount of available help on the web it does have some downsides which include the ability for poor coders to write poor code without even realising it.

I won't go into a diatribe about coding practice as there will always be good coders and bad coders but my main problem with Wordpress is that although rich in functionality and easy to use the code behind the scenes does not lend itself well to performance.

My current blog is getting about 1000 human visitors a day and four times as many bots. It's not a big site and runs on a LINUX box with 1GB RAM which should be enough especially since I have already helped my performance by doing the following:

-Adding the Super Cache Plugin which GZips up content

-Tuning the MySQL and PHP configuation files

-Banning over 40% of undesirable bot traffic using MOD Rewrite

-Disabling unused plug-ins and comparing new ones before installing them so that performance is the key factor I use when deciding which one to use.

I am also looking into using a PHP accelerator which will store the complied version of the source code so that it's not re-interpreted on every load.

However I am still experiencing intermittent memory issues and the problem is trying to detect the source of them. Earlier today I added some debug functions to the Wordpress source code that outputted the Memory_Usage value at key points. The results are quiet shocking as it shows that when the home page is loaded it consumes 24MB of memory!

The file in question is located in your root directory wp_settings.php.

The output is here:

START OF SETTINGS FILE PHP

Memory Limit: 128M - Current Memory Usage: 768 kb - Peak Usage: 768 kb

Before Require Compat, Functions, Classes - Current Memory Usage: 1.25 mb

After require_wp_db - Current Memory Usage: 3 mb

Before Require Plugin, Default Filters, Pomo, l10n - Current Memory Usage: 3 mb

After Required Files loaded - Current Memory Usage: 4 mb

Before Requires of main files Formatting, Query, Widgets, Canonical, Rewrite and much more 30 files! - Current Memory Usage: 4.5 mb

After ALL Requires - Current Memory Usage: 15.25 mb

Before muplugins_loaded action fired - Current Memory Usage: 15.25 mb

Before require vars - Current Memory Usage: 15.25 mb

Before create_initial_taxonomies - Current Memory Usage: 15.25 mb

After create_initial_taxonomies - Current Memory Usage: 15.25 mb

Before include active plugins - Current Memory Usage: 15.25 mb

Before require pluggable - Current Memory Usage: 21.5 mb

Before wp_cache_postload and plugins_loaded - Current Memory Usage: 22.25 mb

After ALL Plugins loaded and action fired - Current Memory Usage: 22.75 mb

Before action sanitize_comment_cookies - Current Memory Usage: 22.75 mb

Before create global objects WP_Query WP_Rewite WP and WP_Widget_Factory - Current Memory Usage: 22.75 mb

Before action setup_theme - Current Memory Usage: 22.75 mb

Before wp-init and do_action(init) - Current Memory Usage: 23.75 mb

End of Settings - Current Memory Usage: 24 mb

As you can see the inclusion of 30 core include files that Wordpress loads on every page adds a wopping 10MB in one go is a major factor in the large size as well as the loading and activation of other plug-ins.

If I didn't have Caching enabled then you can imagine how much trouble I would be in. If I was suddenly hit with a spike in traffic of only 42 concurrent users my 1GB of RAM would have been eaten up straight away. Plus in reality it wouldn't even need 42 users as not all my RAM will be consumed by PHP / APACHE and we need to factor in all the MySQL queries that run behind the scenes.

In fact from using the Wordpress function get_num_queries() which logs each query run by the wp_db->query() method it shows that my home page makes 32 calls to the database!

Now in no developers world should these be good stats to report on and it just goes to show the sort of battle someone has to fight when making a Wordpress blog run under high loads on a small budget. Yes throwing resources at the problem is a form of answer but a better one is to resolve the underlying issues. The problem is that it's very hard to with someone else's code.

One of the big downsides to using other peoples code is that when it goes wrong you are big trouble. You can either wait for an upgrade from the developer which may or may not come within any sort of agreeable timescale or you try to make sense of the code they wrote. The issue with the core Wordpress codebase is that if you start hacking it about then you leave yourself in a position of having to ignore future updates or having to redevelop it yourself for time memorial.

I don't know which of these two undesirable options I am going to take but I know what I would be looking at if I had to redevelop the wordpress system.

1. I would most definitley consider running it from a different database instead of MySQL which has a lot of good features but also a lot of configurable options I don't want to have to worry about.

If I am joining a lot of records together in a GROUP_CONCAT or CONCAT I don't want to have to worry about calculating the size of the group_concat_max_len first I just want all my string to return.

Also as well as all the missing DML such as CTE's I really miss the very useful Data Management Views that SQL Server 2005-2008 has as they make optimising a database very easy. I have seen a lot of Plugins that use very poor SQL techniques and some time spent on proper indexing would speed things up a lot. Having to sift through the slow query log and then run an EXPLAIN on each one is a time consuming job whereas setting up a scheduled job to monitor missing indexes and then list all suggestions is very easy to do in MSSQL.

2. I would definitely change the database schema that Wordpress runs on and one of the major changes would be to horizontally partition the main wp_posts table as this is referenced a hell of a lot by plug-ins and other internal Wordpress code and the majority of the time the wide columns such as content and excerpt are never required. A narrow table containing the post id, post type, modified date and maybe the post title would help things a lot.

3. All of the queries run in the system are single executions. This means extra network traffic and extra resources. Returning multiple recordsets in one hit would save a lot of time and I know it's hard in a CMS to combine queries but with some clever thinking it could be done.

4. A lot of performance problems come from calling the same function multiple times within loops rather than calling it once, storing the value and then re-using it. A good example is all the calls to get the permalink structures when creating links.

5. All the checks to see whether functions, classes and constants exist must be consuming some resources. It is hard in a CMS to know what has and hasn't been included but maybe some of these could be left out.

6. Caching should be built into the Wordpress core code as standard. When new posts and pages are saved there should be the option to create a physical hard copy of the html instead of using the usual dynamic page. Any sidebar functionality could also be calculated and added there and then as does it really matter if a tag or category cloud is slightly out of date or do links to other blogs and sites really need to be loaded from the DB every time? The answer is obviously no.

Anyway those are just six suggestions I have just conjured up through a cursory examination of the codebase. I am sure there are lots more potential ideas and I see from reading up on the web that a lot of people have experienced similar issues with performance since upgrading to Wordpress 2.8.

If anyone else has some suggestions for reducing the memory footprint of a Wordpress blog please let me know as I really don't like the idea of a simple blog using up so much memory.