Test of WordPress Caching Plugins –

There’s been a lot of talk about which is the best caching plugin for WordPress. There’s no doubt in my mind which is best, and that’s W3 Total Cache. I’ve reached this conclusion based on logic, theory and a knowledge of how it should be done. W3TC performs best in theory, but is it the best when tested and what are the pros and cons of the different types of caching tools available? That’s what I’d like to take a closer look at here. I’m going to look at both performance and load times. For that I’m going to use two tools in particular, Pingdom Tools and AB Test (Apache Benchmark test). However, I’ve also used other tools to assess load times on the website both during periods of heavy use and during single visits.

I’d like to begin by describing the different plugins, before I reveal the test results for the three.

The test

The test was carried out on a site which most closely resembles a ‘real-world’ website, with the caveat that the site doesn’t get crawled or found by Google. This is the newest English language version of WordPress for which I’ve bought the currently popular Nevada theme. I’ve set the theme up with standard content supplied by the developers through XML import. Permalinks are activated.

Pingdom Tools and Apache Benchmark are used to look at load times and performance on the site, respectively. In addition to these two tests I’ve also used Gomez Networks, Load Impact and Google Chrome to achieve a valid end result which best approximates reality – FastCGI functioned as the engine for PHP solution. For the Apache Benchmark test I ran the following which means that 1000 tests were run with 10 simultaneous users:

ab -n 1000 -c 10 http://www.domain.com/

Each testing tool was run 10 times, both with and without a caching plugin, and it’s the average result which will be shown at the end. In total, 250 tests were run with intervals of ten minutes and a single AB test on a static html file (where I had copied the html code from the site being tested) to see how much the server could actually handle under those conditions.
It’s important to point out that the tests only look at HTML caching/Page Caching because this is the element which the plugins in question have in common. Some of them also have CDN, Gzip, and Browser Caching but these won’t be considered as they’re not shared by all plugins.

Tests to be divided up as follows:

  • Test without plugins of any type (50 tests)
  • Test using caching plugins and their standard settings (50 tests)
  • Test using caching plugins in their optimized versions (50 tests)

W3TC (W3 Total Cache)

http://wordpress.org/extend/plugins/w3-total-cache

W3 Total Cache is a plugin that has pretty much everything in terms of speed optimization. Among other attributes, it has HTML caching, which we’re testing here, and minification, Object Cache, Database Cache, Gzip, browser caching, CDN, Varnish, minification of js and css files, minification of html code, auto caching and a lot more.
W3 Total Cache creates its html caching files by taking PHP output and putting it into two html files on the server, one compressed and one uncompressed. When this URL is requested a small rewrite in htaccess checks to see if the browser can support compressed files, and if the cached file exists, which, if it does, is then displayed to the user. If it doesn’t, it will continue on to WordPress and show that page’s content. If caching is permitted, W3TC will then create a cached version of the requested page and display it to the next user who requests it.
If you want to set up the whole plugin with all its features, it will typically take 1 to two hours, depending on website type, content, activity, theme etc. But when you’ve set up the plugin and all its features your website will run pretty well. But it’s no substitute for normal optimization of, for example, pictures or the usual requirement that your theme and plugins have been well coded.

WP Super Cache

http://wordpress.org/extend/plugins/wp-super-cache/

WP Super Cache is a page caching plugin which offers some different options including page caching and CDN. It’s easy to set up and doesn’t require much tech knowhow from the administrator. It uses three different caching methods with the default setting being the next best of the three options (via PHP). Unlike W3TC it doesn’t create a compressed version of the cached file by default, but of course this is something you can ask it to do.
Setting up the entire plugin takes about twenty minutes. It’s a small plugin that’s quick to install, and pretty much anyone can do it.

Quick Cache

http://wordpress.org/extend/plugins/quick-cache/

Quick Cache is an even smaller plugin which only handles page caching. There aren’t many settings to speak of but on the other hand it’s incredibly easy to use. The only thing you need to do is to activate the plugin and switch ‘page caching’ on.
Other settings deal primarily with when page caching is permitted and when cached files should be deleted. There is also an option for browser caching but the developers recommend that you don’t activate this.

Shared Features of the tested plugins:

Common to all is, of course, caching of the html on which the homepage is built. That’s about the only thing which all the plugins have in common. So only testing can show the delay-reducing effect on the server before the html code is relayed to the user. Which brings us to testing of load times and performance.

The first test

The first test is carried out on a static html file copied from the code which the home page of the website throws out. This is to see how much the server can handle and how quickly. The result shows that the server should be able to handle around 4,738.14 requests per second and that 1,000 requests from 10 simultaneous users takes on average 0.208 ms. That’s quite fast and puts almost no strain on the server.

Four variables have been measured:

  • Waiting time on the server before file is passed on to the user
  • Number of requests the server can handle per second
  • Waiting time as measured by other testers (for example, Pingdom’s yellow bar)
  • Full load time for the website

The last of these, full load time for the website, isn’t really something we should look at in this test. That’s because there are so many factors (for example, js, css, visuals) which determine how quickly the website loads that it doesn’t give a fair reflection of each plugin’s ability. The only common denominator shared by all plugins tested is page caching / HTML caching and therefore this is the only element we can use to compare performance for these plugins and decide which is best.

files-on-website

Plug and Play Settings

This test was carried out using nothing more than the default settings of each plugin, in other words just switching them on. It didn’t take more than five minutes to install and set up each plugin. In short, one attribute was chosen and the plugin was activated.

Wait time

Wait, time to first byte, execution time. You can call it many things but your main concern is to keep it to a minimum. Firstly, to ensure a faster response time for the end user and, secondly, to lighten the server load, thereby allowing the server/webpage to handle more visitors simultaneously. An added bonus is the reduced risk of server overload and crash due to too many requests. I’m sure you’ve seen it when a product with broad appeal is shown on tv – you visit the website only to find lots of people have thought the same as you and the site is either intolerably slow or fails to load.

wait

On the accompanying green diagram, with measurements provided by external tools (Pingdom tools, for example), you can see that it took the server around 426ms to generate the html to be relayed to the user. With caching plugins this improves considerably and the best in this regard was W3 Total Cache with just 27.9 ms on average.
The reason that it’s faster than WP Super Cache or Quick Cache, for example, is quite simple. W3TC has created a static HTML file which, by means of a simple rewrite in the htaccess file, is shown to the user, if, that is, a static version exists. In contrast, WP Super Cache and Quick Cache both need to begin by going through WordPress in order to fetch the cached file (html and php) which they have previously generated.
Quick Cache is slightly faster than Super Cache. Though I haven’t investigated all the reasons for this, one of them is the fact that Super Cache processes a lot more data than Quick Cache, having as it does large files which check if the page should be fetched as a cached version and if the cached version of the page should be saved.

Number of requests and upload time on the server

The above principle holds true when looking at the number of requests the server can handle – the more the better. With a base measurement of 11.7 pages a second for the website without any caching plugins, a test of W3 Total Cache makes it possible for the server to handle no fewer than 3,636.4 requests per second. Quick Cache, with 691.1, was again better than WP Super Cache, with 334.5. Twice as good, in fact, and for the same reasons I’ve mentioned above.
The time which the server uses to create/generate/fetch the html which the user’s browser then processes must be as low as possible. The end result is that the server can handle lots more traffic simultaneously. With a corresponding decrease in the risk of crashing or requests timed out. W3TC does this in 0.238 ms – that is, not even one millisecond. Which means the website will load faster for the individual user and at the same time be able to handle a sudden spike in the number of visitors.

time-per-request

requests-per-second

The complete load time for the website

In a way the results from this part of the test are not especially usable because, as mentioned before, there are too many contributing factors at play to be able to draw any concrete conclusions from the data. Sometimes pictures, js and css files can be slow to load and other times fast. Even in the 120 tests I carried out I got results that ranged from up to 2300ms down to 1000ms (one second). The main culprit after plug and play optimization was almost always javascript files, which with a little optimization should bring the full load time down and give more stable measurements.
So even though full load time is best for W3TC we can’t declare it the winner on this basis alone. We need to look elsewhere to see or test the effect of html caching. And we can only do that after we’ve optimized js, css and pictures – otherwise they’re simply too unstable.

loadtime

Conclusion of plug and play tests (5 minute installation and setup)

W3 total cache is the clear winner in the plug and play test, mainly because it uses, as standard, the best method of caching and displaying page versions to users. It does this by completely bypassing PHP and WordPress and sending a static html to the user. Apache is super fast at handling static html files, so that’s definitely the way to go if you’re thinking about page caching. The only way to make it faster would be to completely drop the rewrite and instead generate files with their names and location already fixed, as we saw in the first test of the static html file.
The reason W3TC doesn’t quite get top marks in performance terms is precisely because it is a rewrite which the server must negotiate in order to reach the correct cached file. This results in the number of requests being 1000 fewer and processing time being around 0.080ms slower.
The two other plugins, in contrast, start up the PHP engine and WordPress and carry out a huge amount of data processing and finally end by displaying the cached html to the user. That’s the major drawback with these two plugins – that by default they run through PHP and WordPress.
Luckily though, WP Super Cache has another option, one which resembles the way W3TC does things. That’s what I’d like to test now.

Optimized settings for HTML/Page Caching

In the last test I looked at the default settings of each plugin. I used at most five minutes installing and setting up each one, simply switching on html/page caching.
But in this test I’d like to see what happens when I use twenty minutes (max.) setting up WP Super Cache in particular. The aim is that it performs best in terms of html/Page Caching – still the attribute which all plugins have in common.
I’m not going to look at Quick Cache as the previous tests have exhausted its possibilities. W3TC has already been optimally set up, by default, when it comes to page caching, so we’ve already covered it in prior tests.
I should mention in passing that W3TC also has the ability to minify html code but because the other plugins don’t support this it doesn’t fall within the compass of these tests.

WP Super Cache optimized

Under the heading ‘advanced’ you should take the following steps to best exploit the potential offered by Super Cache:
Choose “use mod_rewrite to serve cache files”
Tick “compress pages…”
And save.
If the htaccess file hasn’t been changed then you need to follow the instructions which follow after you have saved the file. This will require you to manually insert the rewrite code in the htaccess files.
Test results after setting up Super Cache optimally:

Test Results Combined

As you can see on the orange and blue graph, the improvement in Super Cache is quite significant. Performance has gone from 334.5 requests per second right up to 2268.5 per second, almost an eight-fold improvement. Wait time has been reduced from a little over 3 milliseconds to under 0.5. Overall waiting time has dropped from 52.6 to 32.3. The full load time, which again we shouldn’t emphasize too much, has gone from 1832 to 1581. All in all quite a big improvement for WP Super cache.

Conclusion (The WP Caching Plugin Test Winner)

I conclude that W3 is the winner, with Super Cache in second place and Quick Cache coming last. The perceptive reader will have spotted that Super Cache didn’t quite become as fast as W3TC even when using the same technique. There’s a reason for that.

In the same way that W3TC wasn’t quite as good as in the first test because of rewrites in the htaccess files, the same thing happens to WP Super Cache. The difference, however, is that WP Super Cache creates around 60 lines of rewrite while W3TC only creates about 20.

To explain briefly, the more lines you have in htaccess, the worse the performance and speed. Not least when it involves mod_rewrite, which rewrites use and which is a slow and unwieldy module in Apache.

It’s not just about load time

Yes, there are more things than load time to be considered when choosing a caching plugin. Of course it’s great when your website loads quickly for users but it’s also very important that performance is good. The faster the performance, the quicker the load time.

Reader request

At the request of regular readers I carried out a further test, one in which I used 1 hour to optimize all the functions available in W3TC. Or, more accurately, the only things I chose to optimize were minification of html, css and javascript, Browser caching and CDN. Even though after optimization I could still have further optimized theme, pictures and content etc., W3TC still managed to come down to an average load time of 537ms, with a variation of 100ms during the test, indicating that website stability was also improved. It needs to be tested some more to find out exactly how quick it’s become.

I’m sure many of you are thinking – “Yeah, well, you don’t really need to optimize anymore.” To that my answer is both yes and no. It all depends on your website, what you want to use it for and not least how many visitors you get. Of course, the more visitors you get the more essential it becomes that each of them can access your site as quickly as possible.

Leave a Reply

Your email address will not be published. Required fields are marked *