What is the current status of web servers…Is Apache 2.x “fast enough?”
Automattic uses Lightspeed (for PHP), nginx (for static content), and Apache (for media uploads). For WordPress-generated content, all server options are approximately the same speed.

What about APC?
Automattic uses beta versions of APC, and provides a 3-5x performance increase. It’s tied closely to the PHP version, so Automattic recently switched from PHP 4 to PHP 5.

Databases?
MySQL scales well and is easy enough to use that there’s little reason to consider other DBs for WordPress content. Other applications may have different needs. Note: FriendFeed uses MySQL to store schema-less data. Single-table key lookups in MySQL are faster than getting the data from Memcached.

Caching?
Automattic uses Batcache for full-page caching (.002 to .003 second), Memcached persistent object cache, very limited MySQL query cache (never larger than 256MB), sufficiently large key buffer.

HyperDB?
HyperDB solves DB scaling problems.

Backups
User-data backed up every hour, if something changed. Every blog backed up every 12 hours. Dedicated MySQL slaves do LVM snapshots for backups.

Peter at MySQL Performance Blog pointed out this sweet perl script to analyze MySQL’s slow query logs. (This is supposedly a PHP port.)

The script does a good job of aggregating similar queries (those that only differ in their query values) and displaying overall stats for them. The following two queries are showing up a lot in my WPMU installation because I also have it set to log queries that don’t use indexes. They’re not necessarily slow (MySQL < 5.1 logs execution times to the nearest second, with a one second minimum), but the fact that they’re appearing a lot probably means the value isn’t being cached in Memcached.

The first SQL line shows the prototype for the query, the second is a real example from the log.

### 90379 Queries
### Total time: 90556, Average time: 1.00195841954436
### Taking 1  to 6  seconds to complete
### Rows analyzed 744 - 587874
SELECT COUNT(ID) FROM wp_XXX_posts WHERE post_status = 'XXX' and post_type = 'XXX';
 
SELECT COUNT(ID) FROM wp_7_posts WHERE post_status = 'publish' and post_type = 'post';

### 8768 Queries
### Total time: 9033, Average time: 1.03022354014599
### Taking 1  to 4  seconds to complete
### Rows analyzed 0 - 199
SELECT option_name, option_value FROM wp_XXX_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;
 
SELECT option_name, option_value FROM wp_4_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;

The above graph is far from typical, but I love that the box (the top one in this picture) can do the job when it needs to. This activity is a result of bulk record imports, web activity results in relatively little database traffic due to my use of Memcached and Batcache.

Via an email from the Amazon Web Services group today:

…we are excited to share some early details with you about a new offering we have under development here at AWS — a content delivery service.

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

• Lets developers and businesses get started easily – there are no minimum fees and no commitments. You will only pay for what you actually use.
• Is simple and easy to use – a single, simple API call is all that is needed to get started delivering your content.
• Works seamlessly with Amazon S3 – this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
• Has a global presence – we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

You’ll start by storing the original version of your objects in Amazon S3, making sure they are publicly readable. Then, you’ll make a simple API call to register your bucket with the new content delivery service. This API call will return a new domain name for you to include in your web pages or application. When clients request an object using this domain name, they will be automatically routed to the nearest edge location for high performance delivery of your content. It’s that simple.

We’re currently working with a small group of private beta customers, and expect to have this service widely available before the end of the year.

Steve Souders: 10% of the problem is server performance, 90% of problem is browser activity after the main html is downloaded. He wrote the book and developed YSlow, so he should know.

JavaScripts are downloaded serially and block other activity. Most JavaScript functions aren’t used at OnLoad. We could split the JS and only load essential functions up front, and load all the rest later. How much might that help? He says 25% to 50%. This quickly gets complex, but he’s got a simple plan that considers three questions:

• Is the script URL on the same host as the main HTML?
• Should the browser indicate it’s busy, or not?
• Does script execution order mater?

And at that point things started to get too interesting to take publishable notes. I clearly need to pay more attention to this guy.

Stats he mentioned without being specific about the source:

• Google: 200ms longer download time cut revenue by 20%
• Yahoo: 100ms of latency costs … big.

When doing a bulk insert/update/change to a MySQL table you can temporarily disable index updates like this:

ALTER TABLE $tbl_name DISABLE KEYS

…do stuff…

ALTER TABLE $tbl_name ENABLE KEYS

From the docs:

ALTER TABLE ... DISABLE KEYS tells MySQL to stop updating non-unique indexes. ALTER TABLE ... ENABLE KEYS then should be used to re-create missing indexes. MySQL does this with a special algorithm that is much faster than inserting keys one by one, so disabling keys before performing bulk insert operations should give a considerable speedup. Using ALTER TABLE ... DISABLE KEYS requires the INDEX privilege in addition to the privileges mentioned earlier.
While the non-unique indexes are disabled, they are ignored for statements such as SELECT and EXPLAIN that otherwise would use them.

The WordPress team has taken up the issue of performance optimization pretty seriously, and I look forward to the fruits of their efforts, but I’m also casting a critical eye on my own code. Thanks to caching and a hugely optimized query architecture, Scriblio is now performing better than ever, and I’m now looking at the next tier of problems.

First among them is a WordPress query that runs to find which posts have pingbacks or trackbacks waiting to be processed. As it was originally written, it looked like this:

$trackbacks = $wpdb->get_results("
	 SELECT ID 
	 FROM $wpdb->posts 
	 WHERE CHAR_LENGTH(TRIM(to_ping)) > 7 
	 AND post_status = 'publish'
	");

The problem is that the query requires a full table scan, and it can’t be optimized simply by creating an index on to_ping because CHAR_LENGTH(TRIM(to_ping)) > 7 requires inspecting every row and applying the functions before the result can be tested. As written, the query could take as long as 15 minutes to execute on a busy table of 350,000 rows.

Once I read through the code and determined that it’s very unlikely that cruft would be left in to_ping, or that any that was there would get removed once processed through the ping functions, I decided to try the query below. (Hat tip also to Andy Skelton for sanity checking my reading of the code.)

$trackbacks = $wpdb->get_results("
	 SELECT ID 
	 FROM $wpdb->posts 
	 WHERE to_ping <> '' 
	 AND post_status = 'publish'
	");

A trac ticket has been created and patch added, but despite the optimized query, a table as large as I’m working with probably needs an index to keep things running smoothly. So here’s what I’ve got:

KEY ping_status ( to_ping(1), post_status )

I’ve not yet confirmed that this is the most optimal index, but queries now take a fraction of a second to complete.

After reporting weirdness last week I finally sat down with a completely clean and virgin install of WordPress 2.3.2 and traced what happens when you make a permalink request for a non-existent URL.

Here are two sets of URLs to use as examples and context:

• These are valid URLs:
  • http://site.org/archives/101
  • http://site.org/page-name
• These are _not_ valid URLs:
  • http://site.org/archivezorz/101
  • http://site.org/favicon.ico

Valid URLs get parsed, the expected MySQL queries get executed, and the results are processed and returned to the browser. Normal. The problem is that invalid URLs that get sent through WordPress still result in a query like the following being executed on the database. What?

SELECT SQL_CALC_FOUND_ROWS  test_posts.* FROM test_posts  WHERE 1=1  AND post_type = 'post' AND (post_status = 'pub
lish' OR post_status = 'private')  ORDER BY post_date DESC LIMIT 0, 10

That is, even after a URL is sent through WP->parse_request() and found to be invalid/404, WordPress marches on to WP->query_posts() and hits the database with a generic request for the X most recent posts. And because this is executed for every 404, it actually results in a lot of database activity.

In most cases MySQL has cached the result, and so it poses a minimal load on the server. And even if the cache is stale, for most sites it’s not a particularly resource intensive query.

But, if you’ve got 350,000 rows in the posts table, it’s incredibly resource intensive to order all those posts on the post_date (datetime) column. I’ve seen hundreds of them pile up and take forever to complete after writes to the table. It’s sufferable if write activity on the posts table is very low, but that’s not something I want to hope for.

So my question to the wp-hackers community is: Do we actually want to execute that query for every 404 under normal circumstances? If not, is the following (or something like it) the stupidest solution?

In wp-includes/classes.php:

	function query_posts() {
		global $wp_the_query;
		$this->build_query_string();
 
		// return if the request URI is a 404
		if( $this->did_permalink && $this->query_vars['error'] == '404' )
			return;
 
		$wp_the_query->query($this->query_vars);
	}

The above works, but there’s probably a better way to write it.

Yes, I’m still trying to squeeze more performance out of MySQL. And since small changes to a query can make a big difference in performance…

Here are two really easy things to be aware of:

• Never do a COUNT(*) (or anything *, says Zach). Instead, replace the * with the name of the column you’re searching against (and is hopefully indexed). That way some queries can execute entirely in the keycache (while * forces MySQL to read every matching row from the table).
• When joining two large tables, but only searching against one, put the join statement at the end. Why join the two entire tables when you only have to join the matching rows?

I mention these because, well, I’ve known them forever, but upon seeing them again I realized I hadn’t really obeyed those simple rules in some of my queries.

Separately, there’s some pretty good info on what server variables affect what at mysqlperformanceblog too.

mysql, optimization, query optimization, performance, tips

I’ve been working on MySQL optimization for a while, and though there’s still more to done on that front, I’ve gotten to the point where the the cumulative query times make up less than half of the page generation time.

So I’m optimizing code when the solution is obvious (and I hope to rope Zach into giving the code a performance audit soon), but I’m also looking at optimizing how PHP works.

Once upon a time, most of us ran PHP as a CGI, and every time a request came in, the PHP interpreter would have to launch, read/compile/execute the script, then spit out the result and shutdown. Now (hopefully) everybody’s running PHP as an Apache module, so all the time spent launching the interpreter, allocating memory and other resources for it, and then shutting it down and cleaning up after it, is done just once for each thread of Apache.

It might not sound like much, but I had a chance to compare CGI vs. module performance recently and found that a fairly simple, but frequently accessed script running as a CGI completely swamped a server as a CGI (creating a load average over 20), but was hardly noticed when running as a module.

But even as a module, the PHP scripts still need to be interpreted and compiled before they can be executed. And because of the way PHP works, this is done every time the page/script is requested.

Java programmers, among others, criticize PHP for this, but that small inefficiency is part of what makes PHP so easy to use (and popular). And that ease of use means people are building some really interesting apps worth scaling.

Anyway, there’s a solution to eliminate that inefficiency in PHP: intermediate code caching.

By caching the executable code generated by the interpreter, then the using the cached copy instead of the source script for the next request, you can enjoy the benefits of PHP’s easy development and compiled code’s fast execution time. A number of projects all promise anywhere from double to 10X jump in performance.

• ionCube PHP Accelerator
  This has the most recognizable “brand,” and some hosting providers even offer it, but it’s offered only in binary form and the license seems intentionally ambiguous (um, not free?).
• Turck MMCache for PHP
  Has a good performance chart, and is GPL’d, but development appears to have ended in 2003.
• afterBURNERr*Cache
  Yeah, I had to point to a page in the Wayback Machine, this project is dead.
• Zend Platform Performance Suite
  A commercial suite that includes piles of goodies, if you pay.
• APC: Alternative PHP Cache
  It’s under active development and includes Rasmus Lerdorf, the guy who created PHP, among the project leads. License: BSD-style (non-copyleft)

I haven’t actually tried any of these yet, but I’m looking for information and suggestions, and I’m likely to try APC, maybe even Zend soon. Just as soon as I make an app compelling enough (and large enough) to need it.

php, caching, acceleration, zend, apc, intermediate code cache, optimization, scaling, web applications

Ryan Boren wrote about using memcached with WordPress almost a year ago:

Memcached is a distributed memory object caching system. WordPress 2.0 can make use of memcached by dropping in a special backend for the WP object cache. The memcached backend replaces the default backend and directs all cache requests to one or more memcached daemons. You must have a memcached daemon running somewhere for this to work. Unless you’re managing the server on which your blog is running, you probably can’t run a memcached daemon, making this backend useless to you. The memcached backend is targeted at ISPs and those running WPMU. If you are using WPMU and distributing DB requests across multiple servers, memcached will come in very handy. Using memcached for a single blog isn’t really worth it. In my tests, it was sometimes slower than using the default object cache backend.

The plugin is here, a bug-fix note is here.

WordPress, memcached, optimization, caching

I’ve been following WP2.1 development, but Aaron Brazell’s post in the development blog wrapped up a lot of questions all at once.

The short story is that 2.1 is going to bring some really good changes that will allow more flexibility and better optimization of WPopac. Of the four changes Brazell names, the last two, the addition of the post_type column and a change in usage of the post_status column, are where the money is.

I’m awaiting the final release of 2.1 before building the necessary changes into WPopac, but the benefits will be worth it.

WordPress 2.1, WPopac, optimization, development

Update: bugfix release b2v6 available.

Some conversations with Chow Kah Soon, who’s site is full of diversions from work , finally convinced encouraged me to solve some small problems that were giving him big trouble. Chow Kah Soon is in the lucky, but rare, position of having over 20,000 unique daily visitors to his site, so he’s sort of my designated stress-tester.

After looking at the logs he shared with me, the table structure, and the queries in bsuite, it was pretty clear that I needed to make some changes to the indexes.

Optimizing the indexes was easy, but how can I make bsuite update those indexes on existing installs? That’s where these docs explaining WordPress’ rather unique dbDelta function come in:

Instead of using a normal $wpdb query to make our table, there is a special function in WordPress that we’ll use that can create a table if it isn’t there, or change the structure if it doesn’t match the current table. This will come in very handy if we ever need to make changes to the MySQL structure in the future.

The function’s a little finicky, and it took me a couple minutes to realize that it was tripping up on the backtick quotes (`) I had wrapped around my table and field names. But with that out of the way, it solves the big problem of how to easily add indexes to existing tables. What it doesn’t solve, sadly, is how to delete existing indexes that aren’t in the new table definition.

With the dbDelta function implemented, I also finally got around to implementing an auto-install hook that creates or re-configs the tables upon plugin activation. Here’s the example from the docs:

add_action('activate_pluginurl','jal_install');

But I had to Google for this support forum post to understand what the “pluginurl” should be.

…you simply have to use the path of your plugin relative to /wp-content/plugins. Eg: you have a plugin in /wp-content/plugins/wp-spamfilter/wp-spamfilter.php then you have to call add_action as following:

add_action(‘activate_wp-spamfilter/wp-spamfilter.php’, ’somefunction’);

Yes, all of these updates will be released soon. I promise.

bstat, bsuite, Chow Kah Soon, documentation, KahSoon.com, mysql, optimization, update, wordpress

I have over 1000 posts here at MaisonBisson, but even so, the table with all those posts is under 3MB. Now I’ve got a project with 150,000 posts — yes, 150,000 posts! — and the table is about 500MB. An associated table, structured sort of like WP’s postsmeta, has over 1.5 million records and weighs in at over 100MB (not including the 150MB of indexes).

Up to now I’ve been a “throw more hardware at it” sort of guy — and in a server with only 1GB of RAM, that’s probably the best solution — but I also think it’s time I learned some MySQL optimization tricks. Zach’s been pushing me to get mytop installed for quite a while, and I finally got around to it.

mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL 3.22.x, 3.23.x, and 4.x server. It runs on most Unix systems (including Mac OS X) which have Perl, DBI, and Term::ReadKey installed. And with Term::ANSIColor installed you even get color. If you install Time::HiRes, you’ll get good real-time queries/second stats.

mysql, mysql optimization, optimization, server overload, big databases, mytop, server sizing, database optimization, db