What is the current status of web servers…Is Apache 2.x “fast enough?”
Automattic uses Lightspeed (for PHP), nginx (for static content), and Apache (for media uploads). For WordPress-generated content, all server options are approximately the same speed.

What about APC?
Automattic uses beta versions of APC, and provides a 3-5x performance increase. It’s tied closely to the PHP version, so Automattic recently switched from PHP 4 to PHP 5.

Databases?
MySQL scales well and is easy enough to use that there’s little reason to consider other DBs for WordPress content. Other applications may have different needs. Note: FriendFeed uses MySQL to store schema-less data. Single-table key lookups in MySQL are faster than getting the data from Memcached.

Caching?
Automattic uses Batcache for full-page caching (.002 to .003 second), Memcached persistent object cache, very limited MySQL query cache (never larger than 256MB), sufficiently large key buffer.

HyperDB?
HyperDB solves DB scaling problems.

Backups
User-data backed up every hour, if something changed. Every blog backed up every 12 hours. Dedicated MySQL slaves do LVM snapshots for backups.

jQuery 1.3.2 is in WordPress 2.8, but the most exciting changes are in the automatic concatenation and compression of scripts via the script loader.

Andrew Ozz says “This feature can easily be extended to include scripts added by plugins and to use server side caching, however that would require some changes to the server settings (.htaccess on Apache).”

I have yet to figure out how to extend that feature to scripts in my plugins, but I’m working on it.

Peter at MySQL Performance Blog pointed out this sweet perl script to analyze MySQL’s slow query logs. (This is supposedly a PHP port.)

The script does a good job of aggregating similar queries (those that only differ in their query values) and displaying overall stats for them. The following two queries are showing up a lot in my WPMU installation because I also have it set to log queries that don’t use indexes. They’re not necessarily slow (MySQL < 5.1 logs execution times to the nearest second, with a one second minimum), but the fact that they’re appearing a lot probably means the value isn’t being cached in Memcached.

The first SQL line shows the prototype for the query, the second is a real example from the log.

### 90379 Queries
### Total time: 90556, Average time: 1.00195841954436
### Taking 1  to 6  seconds to complete
### Rows analyzed 744 - 587874
SELECT COUNT(ID) FROM wp_XXX_posts WHERE post_status = 'XXX' and post_type = 'XXX';
 
SELECT COUNT(ID) FROM wp_7_posts WHERE post_status = 'publish' and post_type = 'post';

### 8768 Queries
### Total time: 9033, Average time: 1.03022354014599
### Taking 1  to 4  seconds to complete
### Rows analyzed 0 - 199
SELECT option_name, option_value FROM wp_XXX_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;
 
SELECT option_name, option_value FROM wp_4_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;

I recently replaced the Mac Mini I use to host my web development with a PowerMac G4. (Story: the Mini was mine, a personal purchase I made to support my work on Scriblio and other WordPress-related projects, but recent changes in our network and firewall policy made the machine inaccessible from off-campus without using the VPN. Having a personal machine sit at my desk at work isn’t as useful if I can’t use it conveniently and for para-work activities, so I wanted to take the Mini home.) The first challenge was replacing the x86-specific apps and daemons with PPC versions, the next one was in coming to terms with how dramatically slower the dual 1GHz G4 is from the 1.66Ghz Core Duo.

John Poole’s 2006 blog post confirmed what I was feeling: G4s are way slower than I realized at the time.

Simple fact of The Google Economy: people can’t find stuff if it’s not indexed in major search engines. A slow site might not seem as bad as blocking the crawlers that search engines use to index your content, but it does seriously affect the depth and frequency of crawling they do.

The above is Google’s report of their crawling activity on a site I’ve been trying to optimize server performance on. Comparing the response time in the bottom graph to the number of pages crawled at the top offers some clear evidence of the relationship between the two. Most people seem to think server performance only matters if your site gets Dugg (old people say Slashdotted). Now even I’ll think again.

This two year old post about Rasmus Lerdorf’s PHP scaling tips (slides) is interesting in the context of what we’ve learned since then. APC now seems common, and it’s supposedly built-in to PHP6. Still, I’d be interested in seeing an update. Are MySQL prepared statements still slow?

And that’s where Rasmus’ latest presentation comes in. We don’t learn anything about MySQL prepared statements, but we do learn how to find choke points in our applications using callgrind and other tools. In his examples, he can do a little over 600 transactions per second with both static HTML and simple PHP, but various frameworks — with many inclusions and function calls — can slow that to under 50 transactions per second (I suppose they’d explain that in a TPS report).

Via an email from the Amazon Web Services group today:

…we are excited to share some early details with you about a new offering we have under development here at AWS — a content delivery service.

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

• Lets developers and businesses get started easily – there are no minimum fees and no commitments. You will only pay for what you actually use.
• Is simple and easy to use – a single, simple API call is all that is needed to get started delivering your content.
• Works seamlessly with Amazon S3 – this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
• Has a global presence – we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

You’ll start by storing the original version of your objects in Amazon S3, making sure they are publicly readable. Then, you’ll make a simple API call to register your bucket with the new content delivery service. This API call will return a new domain name for you to include in your web pages or application. When clients request an object using this domain name, they will be automatically routed to the nearest edge location for high performance delivery of your content. It’s that simple.

We’re currently working with a small group of private beta customers, and expect to have this service widely available before the end of the year.

WordPress.com VIP hosts some high-traffic sites, including Gizmodo’s live coverage of the iPhone 3g introduction. Now that the NFL has selected the service for their blogging we’ll get a chance to see how they handle the Superbowl rush.

Hearing Steve Souders at WordCamp last week got me thinking about website performance, so I went looking for more. The slides from his WordCamp talk are online, but he gave a similar talk at Google I/O which got videotaped and posted richer detail than his slides alone will ever reveal.

Also on his blog: Use the Google AJAX Libraries API when you don’t have a CDN, and a post that asks why make users wait to download all your javascript before they see the page if you’re only going to use 25% of it at first?

Google turned this up, but i have no idea how old it is: How to Monitor MySQL’s performance.

Emil Ong is the Chief Evangelist and a lead developer for Caucho Technology, the developers of the Quercus PHP to Java compiler. The idea, I guess, is to write in PHP, deploy in Java, which some people say is better supported by the “enterprise.”

Ong claims 26% performance improvement over Apache + mod_php + APC. That sounds great, I suppose, but it’s less than what Chris Lea suggests is possible if you simply replace Apache with Nginx.

Steve Souders: 10% of the problem is server performance, 90% of problem is browser activity after the main html is downloaded. He wrote the book and developed YSlow, so he should know.

JavaScripts are downloaded serially and block other activity. Most JavaScript functions aren’t used at OnLoad. We could split the JS and only load essential functions up front, and load all the rest later. How much might that help? He says 25% to 50%. This quickly gets complex, but he’s got a simple plan that considers three questions:

• Is the script URL on the same host as the main HTML?
• Should the browser indicate it’s busy, or not?
• Does script execution order mater?

And at that point things started to get too interesting to take publishable notes. I clearly need to pay more attention to this guy.

Stats he mentioned without being specific about the source:

• Google: 200ms longer download time cut revenue by 20%
• Yahoo: 100ms of latency costs … big.

Elliott C. Back points to his use of object caching, WP-Cache, and MySQL query caching among the reasons why his site “is so much faster that yours.”

Jacob Santos‘ FuncDoc notes:

The WordPress Object Cache changed in WordPress 2.5 and removed a lot of file support from the code. This means that the Object Cache in WordPress 2.5 is completely dependent on memory and will not be saved to disk for retrieval later. The constant WP_CACHE also changed its meaning.

I’ve just started using the object cache and I’m happy with how it works now, so these changes are somewhat concerning. On the other hand, I’m confident that I’ll see even more performance gains once I get my head around the new cache and figure out how to implement it.

Extra: Peter Westwood’s Cache Inspect looks useful.

The DevShed technical tour explains that MySQL can spawn new threads, each of which can execute on a different processor/core. What it doesn’t say is that a single thread can only execute on a single core, and if that thread locks a table, then no other threads that need that table can execute until the locking thread/query is complete. Short answer: MySQL works well on multi-core machines until you lock a table.

The good news is that performance is a big goal for WP 2.4, the bad news is that it’s been delayed to the end of January at the earliest.

It was part of a long thread among WordPress hackers over the summer and fall, but this post at VulgarisOverIP just reminded of it: minify promises to be an easy way to compress external CSS and JavaScript without adding extra steps to your develop/deploy process. No, really, look at the usage instructions. (To be clear, the Vulgaris and Google Code versions are different, one derived from the other and backported to PHP4 compatible. Still, the concept is the same.)

Vulgaris reports a nearly 300% decrease in time to download, definitely worth the effort.

Yes, I’m still trying to squeeze more performance out of MySQL. And since small changes to a query can make a big difference in performance…

Here are two really easy things to be aware of:

• Never do a COUNT(*) (or anything *, says Zach). Instead, replace the * with the name of the column you’re searching against (and is hopefully indexed). That way some queries can execute entirely in the keycache (while * forces MySQL to read every matching row from the table).
• When joining two large tables, but only searching against one, put the join statement at the end. Why join the two entire tables when you only have to join the matching rows?

I mention these because, well, I’ve known them forever, but upon seeing them again I realized I hadn’t really obeyed those simple rules in some of my queries.

Separately, there’s some pretty good info on what server variables affect what at mysqlperformanceblog too.

mysql, optimization, query optimization, performance, tips

Harry Shearer and Judith Owen are performing their holiday sing-a-long at the concert hall at the Society for Ethical Culture in NYC with guests TMBG and others. It’s a go on Friday, but why can’t these things happen closer to me? Actually, maybe they should all come to Warren afterwards.

Harry Shearer, Judith Owen, holiday sing-a-long, holiday sing along, show, performance, tmbg, they might be giants

I got itchy about magic quotes the other day because it’s the cause (through a fairly long cascade of errors) of some performance problems and runaways I’ve been seeing lately (pictured above).

But I deserve most of the blame for allowing a query like this to run at all:

SELECT type, data, count(*) AS hits
	FROM wpopac_WPopac_bibs_atsk
	WHERE data LIKE '%'
	AND type IN ('subjkey','author', 'title')
	GROUP BY data
	ORDER BY hits DESC
	LIMIT 7

As executed, it’s trying to select all 1.2 million records from the table, group them (a hugely memory intensive activity with text), and then report only the top seven.

It’s one of a few queries that returns the alternate searches in WPopac. I’m planning to rethink how that works, but I also should have built in a fail-safe to prevent it from trying to execute if the search variable is empty (the query above is written WHERE data LIKE '$searchterms%' or something like that). I’m amused that MySQL tries so hard to fulfill the search — so hard that it’ll consume all available RAM and swap to do so.

The sysadmin and DBA are now much happier — along with any user frustrated by poor performance — with this patched.

bug, bugs, mysql, patch, performance, queries, query, query failures, wpopac

Jonathan Schwartz made the kind of news that makes Slash Dotters happy: he announced Sun is (sort of) giving away free servers. It’s a promotion, a media play, of course, but one that might make a few lucky people very happy.

Here’s the deal: Sun is really proud of their new T2000 eight core server. Each core runs at 1.2GHz, but they’re apparently applying some distributive power of multiplication and calling it an 9.6GHz machine and touting some impressive performance benchmarks. But rather than play the game of “our experts are better than your experts” against other vendors, they’ve extended a pretty generous try and buy program to users themselves. And, as Jonathan made noise in his blog post, a few people might get lucky enough to keep their machines.

So what’s my angle? WPopac rocks, and performance is excellent with my test corpus (160,000 bib records) on even a lowly single-CPU server with only 1GB of RAM, despite the fact that each bib record is represented in both its composed form in the wp_posts table and as about 20 more rows per record in a separate table (yup, I’ve got almost 3 million rows representing every atomic detail of every bib record), and all of this is full-text indexed.

And I’m fully confident that when I put our entire catalog into WPopac, all 330,000 bib records (resulting in about 6.2 million atomic records), performance will still be up to the task. My math suggests everything should be ducky on a relatively budget server up beyond about 1 million bib records, but what happens for libraries that have more than that, say, perhaps 6 to 8 million bib records (again, 110 to 150 million atomic records; again, all full-text indexed in MySQL)?

It would seem that WPopac’s combination of, um, large MySQL databases with high-volume Apache/PHP should be an ideal fit for the T2000’s big RAM space and support for high numbers of parallel threads. Jonathan claims the T2000 costs a modest $5,000 (but the online store shows a higher price?) and outperforms everything else, but is it up to WPopac? Do they want to try?

C’mon Sun, toss a T2000 this way, let’s see what happens? (Heck, I’ll even try a T1000.)

sun, solaris, t2000, coolthreads, server, 9.6ghz, challenge, sparc, ultrasparc, 8core, free, wpopac, performance, big db, Jonathan Schwartz , try-n-buy, try and buy, free server

Steve Wynn could probably have had any show he wanted, but he chose Avenue Q, the Sesame Street and Muppets-inspired show that has to include a disclaimer denying its roots in the program and advertising. What the show’s creators don’t have to disclaim are the three Tony Awards the show won in 2004 for best score, best book, and best musical.

Sandee bought the cast recording (also at Amazon) because they’re the sort of tunes that get into your head…the sort of tunes you’ll find yourself humming days later. But that’s what you expect of a good musical — especially a hilariously funny one.

Above is Sandee with Kate Monster, the kindergarden teacher who dreams of opening a school for monsters. I believe the performer at left is the understudy, but I’ve lost the program and can’t begin to remember her name. Sorry.

vegas, las vegas, nevada, las vegas nv, wynn, fuzz, avenue q, show, musical, kate monster, kate, monster, puppet, puppet show, sandee, q, muppet, musical, theater, performance, funny