MySQL answer: utf8_unicode_ci vs. utf8_general_ci.

Collation controls sorting behavior. Unicode rationalizes the character set, but doesn’t, on it’s own, rationalize sorting behavior for all the various languages it supports. utf8_general_ci (ci = case insensitive) is apparently a bit faster, but sloppier, and only appropriate for English language data sets.

Peter at MySQL Performance Blog pointed out this sweet perl script to analyze MySQL’s slow query logs. (This is supposedly a PHP port.)

The script does a good job of aggregating similar queries (those that only differ in their query values) and displaying overall stats for them. The following two queries are showing up a lot in my WPMU installation because I also have it set to log queries that don’t use indexes. They’re not necessarily slow (MySQL < 5.1 logs execution times to the nearest second, with a one second minimum), but the fact that they’re appearing a lot probably means the value isn’t being cached in Memcached.

The first SQL line shows the prototype for the query, the second is a real example from the log.

### 90379 Queries
### Total time: 90556, Average time: 1.00195841954436
### Taking 1  to 6  seconds to complete
### Rows analyzed 744 - 587874
SELECT COUNT(ID) FROM wp_XXX_posts WHERE post_status = 'XXX' and post_type = 'XXX';
 
SELECT COUNT(ID) FROM wp_7_posts WHERE post_status = 'publish' and post_type = 'post';

### 8768 Queries
### Total time: 9033, Average time: 1.03022354014599
### Taking 1  to 4  seconds to complete
### Rows analyzed 0 - 199
SELECT option_name, option_value FROM wp_XXX_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;
 
SELECT option_name, option_value FROM wp_4_options FORCE INDEX(PRIMARY) ORDER BY option_id ASC;

The above graph is far from typical, but I love that the box (the top one in this picture) can do the job when it needs to. This activity is a result of bulk record imports, web activity results in relatively little database traffic due to my use of Memcached and Batcache.

# mysqlcheck -p -A --auto-repair --optimize
wp_1_options
info     : Found block with too small length at 17732; Skipped
info     : Wrong block with wrong total length starting at 17776
info     : Found block with too small length at 28776; Skipped
warning  : Number of rows changed from 444 to 441
status   : OK

Cleaning up the mess after a hardware failure can suck. This mysqlcheck output is from the wp_options table for this blog. Unfortunately, if the options table is unreadable, all of WordPress panics and fails to load.

MySQL 5.1 is out as a GA release, but with crashing bugs that should give likely users pause. Perhaps worse, the problems are blamed on essential breakdowns in the project management: “We have changed the release model so that instead of focusing on quality and features our release is now defined by timeliness and features. Quality is not regarded to be that important.”

Still, people are finding inspiration in OurDelta and Drizzle. Competition from those braches/forks and criticism from the community are sure to help re-align the MySQL core, or provide a reasonable alternative if Sun/MySQL can’t deliver. In the meanwhile, the High Availability MySQL blog is worth following.

This Gentoo Wiki page suggests dumping the table and using iconv to convert the characters, then insert the dump into a new table with the new charset.

Alex King solved a different problem: his apps were talking UTF8, but his tables were Latin1. His solution was to dump the tables, change the charset info in the dump file, then re-insert the contents.

Google turned this up, but i have no idea how old it is: How to Monitor MySQL’s performance.

Now that I’m the nominal MySQL DBA for PSU, it became my job to jimmy up the MySQL user privileges so that the new web server could connect. I’m not sure if this is the fastest, most efficient way to do it, but it worked quickly enough:

CREATE TABLE mysql.user_copy SELECT * FROM mysql.user;
DELETE FROM mysql.user_copy WHERE Host NOT LIKE 'OLD_HOST_NAME';
UPDATE mysql.user_copy SET Host = 'NEW_HOST_NAME';
INSERT INTO mysql.user SELECT * FROM mysql.user_copy;
DROP TABLE mysql.user_copy;
 
CREATE TABLE mysql.db_copy SELECT * FROM mysql.db;
DELETE FROM mysql.db_copy WHERE Host NOT LIKE 'OLD_HOST_NAME';
UPDATE mysql.db_copy SET Host = 'NEW_HOST_NAME';
INSERT INTO mysql.db SELECT * FROM mysql.db_copy;
DROP TABLE mysql.db_copy;
 
FLUSH PRIVILEGES;

Simply replace the OLD_HOST_NAME and NEW_HOST_NAME with the appropriate values. Most importantly, I didn’t have to know the passwords for each user to do this. This script simply copied the user info and gave them access from the new server.

When doing a bulk insert/update/change to a MySQL table you can temporarily disable index updates like this:

ALTER TABLE $tbl_name DISABLE KEYS

…do stuff…

ALTER TABLE $tbl_name ENABLE KEYS

From the docs:

ALTER TABLE ... DISABLE KEYS tells MySQL to stop updating non-unique indexes. ALTER TABLE ... ENABLE KEYS then should be used to re-create missing indexes. MySQL does this with a special algorithm that is much faster than inserting keys one by one, so disabling keys before performing bulk insert operations should give a considerable speedup. Using ALTER TABLE ... DISABLE KEYS requires the INDEX privilege in addition to the privileges mentioned earlier.
While the non-unique indexes are disabled, they are ignored for statements such as SELECT and EXPLAIN that otherwise would use them.

After an upgrade to MySQL 5.0.51b on RHEL 5 I started seeing curious results in a fairly common query. Here’s a simplified version:

SELECT ID, post_date_gmt 
FROM wp_posts
GROUP BY ID
ORDER BY post_date_gmt DESC 
LIMIT 5

What I expected was to get a handful of post ID numbers sorted in descending order by the post_date_gmt. Instead, I got a list of post IDs sorted in ascending order by the ID number. Something like this:

3	2007-05-21 00:00:00
4	2007-05-21 00:00:00
5	2007-05-21 00:00:00
6	2007-05-21 00:00:00
7	2007-05-21 00:00:00

After some fiddling I discovered that the GROUP BY clause was causing a problem. So this query works:

SELECT ID, post_date_gmt 
FROM wp_posts
ORDER BY post_date_gmt DESC 
LIMIT 5

…and outputs the results I expected:

337832	2008-06-20 15:20:03
335991	2008-06-17 13:00:42
337777	2008-06-02 12:15:46
337390	2008-05-28 00:00:00
337831	2008-05-28 00:00:00

The GROUP BY clause may be unnecessary, though it was originally written in to accommodate conditions where a JOIN (which is often added when the query is dynamically generated) causes MySQL to return multiple rows representing the same record ID.

Still, isn’t this behavior weird? It’s certainly different from previous versions.

GIS data seems to come in .shp (shape?) files, but it’s not like MySQL knows what to do with those. this MySQL forum post points to a PHP tool and Windows executable that promise to convert the .shp data into something more useful to MySQL.

Superfluo explains a little more, and there’s lots of .shp data to be had here.

MySQL offers ENVELOPE() to find the minimum bounding rectangle of a geometric object.

The result is a polygon with four segments, defined by five points. It took me a while to make sense of it, partially because the only documentation that I’ve run across so far for POLYGON() syntax is in the ENVELOPE() function mentioned above. I also had to draw a picture to think it through.

They write this: POLYGON(( MINX MINY, MAXX MINY, MAXX MAXY, MINX MAXY, MINX MINY )), I think this (in pseudocode-ish form): POLYGON(( $point_a, $point_b, $point_c, $point_d, $point_a )), with the $point_s corresponding to the diagram.

It’s MySQL spatial data week here, though I am spreading out the posts to, um, ease the pain (or boredom). Anyway, here are some commands/functions I don’t want to forget about later:

Start with an existing table called geometry, add a spatial column and index it:

ALTER TABLE geometry ADD coord POINT NOT NULL;
CREATE SPATIAL INDEX coord ON geometry (coord);

Insert some data; think in terms of POINT(X Y) or POINT(lat lon):

INSERT INTO geometry (coord) VALUES( GeomFromText( 'POINT(40 -100)' ));
INSERT INTO geometry (coord) VALUES( GeomFromText( 'POINT(1 1)' ));

Get those X,Y coordinates back from the table:

SELECT X(coord), Y(coord) FROM geometry

Get points within a bounding rectangle:

SELECT MBRContains(
	GeomFromText( 'POLYGON((0 0,0 3,3 3,3 0,0 0))' ),
	coord
)
FROM geometry

MySQL has some powerful, and perhaps underused spatial extensions, but the most interesting functions are still unimplemented: “Note: Currently, MySQL does not implement these functions…”

Among those as-yet unimplemented functions is DISTANCE(). Alternatives can be found here and here, though neither is clean or simple. I wonder if a simple MBRContains() is good enough, though…

Found in the MySQL 5.0 Reference Manual:

• Related(g1,g2,pattern_matrix)
  Returns 1 or 0 to indicate whether the spatial relationship specified by pattern_matrix exists between g1 and g2. Returns –1 if the arguments are NULL. The pattern matrix is a string. Its specification will be noted here if this function is implemented.

(emphasis mine.)

The DevShed technical tour explains that MySQL can spawn new threads, each of which can execute on a different processor/core. What it doesn’t say is that a single thread can only execute on a single core, and if that thread locks a table, then no other threads that need that table can execute until the locking thread/query is complete. Short answer: MySQL works well on multi-core machines until you lock a table.

p0ps Harlow tweeted something about trying to get an AMP environment running on his Mac. Conversation followed, and eventually I sent along an email that look sorta like this:

If you’re running 10.4 (I doubt it, but it’s worth mentioning because I’m most familiar with it), here’s how I’ve setup dozens of machines for web development and WordPress:

Install MySQL
http://dev.mysql.com/downloads/mysql/5.0.html#macosx-dmg

Install Marc Liyanage’s PHP 5 package
http://www.entropy.ch/software/macosx/php/

From there all you have to do is install WordPress.

10.5 changed lots of this (as you probably already know). The good news is that it includes current versions of the AMP suite. The bad news is that the PHP doesn’t include a number of useful components. Still, it will run WordPress and I’ve got it working on my laptop. These directions look like pretty much what I had to do:

http://stringfoo.com/2007/11/05/server_setup_on_leopard/
http://remysharp.com/2007/10/27/lamp-in-leopard-osx-105-php5-and-apache-22

It’s likely none of that will be helpful, in which case you will have discovered why I’m no longer a sysadmin.

Marc Liyanage’s builds of PHP for Mac are probably the easiest to use and they include most all the extensions a person could want, but he hadn’t released a package for Leopard at the time (he’s got a release in beta now). As it turned out, p0ps actually was running 10.4, and he got things going in a jiffy.

The WordPress team has taken up the issue of performance optimization pretty seriously, and I look forward to the fruits of their efforts, but I’m also casting a critical eye on my own code. Thanks to caching and a hugely optimized query architecture, Scriblio is now performing better than ever, and I’m now looking at the next tier of problems.

First among them is a WordPress query that runs to find which posts have pingbacks or trackbacks waiting to be processed. As it was originally written, it looked like this:

$trackbacks = $wpdb->get_results("
	 SELECT ID 
	 FROM $wpdb->posts 
	 WHERE CHAR_LENGTH(TRIM(to_ping)) > 7 
	 AND post_status = 'publish'
	");

The problem is that the query requires a full table scan, and it can’t be optimized simply by creating an index on to_ping because CHAR_LENGTH(TRIM(to_ping)) > 7 requires inspecting every row and applying the functions before the result can be tested. As written, the query could take as long as 15 minutes to execute on a busy table of 350,000 rows.

Once I read through the code and determined that it’s very unlikely that cruft would be left in to_ping, or that any that was there would get removed once processed through the ping functions, I decided to try the query below. (Hat tip also to Andy Skelton for sanity checking my reading of the code.)

$trackbacks = $wpdb->get_results("
	 SELECT ID 
	 FROM $wpdb->posts 
	 WHERE to_ping <> '' 
	 AND post_status = 'publish'
	");

A trac ticket has been created and patch added, but despite the optimized query, a table as large as I’m working with probably needs an index to keep things running smoothly. So here’s what I’ve got:

KEY ping_status ( to_ping(1), post_status )

I’ve not yet confirmed that this is the most optimal index, but queries now take a fraction of a second to complete.

After reporting weirdness last week I finally sat down with a completely clean and virgin install of WordPress 2.3.2 and traced what happens when you make a permalink request for a non-existent URL.

Here are two sets of URLs to use as examples and context:

• These are valid URLs:
  • http://site.org/archives/101
  • http://site.org/page-name
• These are _not_ valid URLs:
  • http://site.org/archivezorz/101
  • http://site.org/favicon.ico

Valid URLs get parsed, the expected MySQL queries get executed, and the results are processed and returned to the browser. Normal. The problem is that invalid URLs that get sent through WordPress still result in a query like the following being executed on the database. What?

SELECT SQL_CALC_FOUND_ROWS  test_posts.* FROM test_posts  WHERE 1=1  AND post_type = 'post' AND (post_status = 'pub
lish' OR post_status = 'private')  ORDER BY post_date DESC LIMIT 0, 10

That is, even after a URL is sent through WP->parse_request() and found to be invalid/404, WordPress marches on to WP->query_posts() and hits the database with a generic request for the X most recent posts. And because this is executed for every 404, it actually results in a lot of database activity.

In most cases MySQL has cached the result, and so it poses a minimal load on the server. And even if the cache is stale, for most sites it’s not a particularly resource intensive query.

But, if you’ve got 350,000 rows in the posts table, it’s incredibly resource intensive to order all those posts on the post_date (datetime) column. I’ve seen hundreds of them pile up and take forever to complete after writes to the table. It’s sufferable if write activity on the posts table is very low, but that’s not something I want to hope for.

So my question to the wp-hackers community is: Do we actually want to execute that query for every 404 under normal circumstances? If not, is the following (or something like it) the stupidest solution?

In wp-includes/classes.php:

	function query_posts() {
		global $wp_the_query;
		$this->build_query_string();
 
		// return if the request URI is a 404
		if( $this->did_permalink && $this->query_vars['error'] == '404' )
			return;
 
		$wp_the_query->query($this->query_vars);
	}

The above works, but there’s probably a better way to write it.

Watching the WordPress hacker list this week, a couple messages related to selecting information about users schooled me on MySQL syntax. I obviously knew the following would work, but I’d previously used the UNION syntax in similar situations and somehow hadn’t thought of writing it this way:

SELECT
	(SELECT meta_value FROM wp_usermeta WHERE meta_key = 'first_name' AND user_id = 2) AS first,
	(SELECT meta_value FROM wp_usermeta WHERE meta_key = 'last_name' AND user_id = 2) AS last,
	wp_users.*
FROM
	wp_users
WHERE
	wp_users.ID = 2

That’s much cleaner to my thinking, though I’ve no idea which is more optimal. When somebody replied asking for a solution that would work in pre-MySQL 5, this was the response:

SELECT ID, user_login, first.meta_value AS fname, last.meta_value AS lname
FROM wp_users
LEFT JOIN wp_usermeta AS first ON (wp_users.ID = first.user_id
AND first.meta_key = 'first_name')
LEFT JOIN wp_usermeta AS last ON (wp_users.ID = last.user_id AND last.meta_key = 'last_name')

And the lesson to me here is that I didn’t realize the syntax allowed us to match multiple conditions for the JOIN. Makes sense, but I just hadn’t thought of it. Thanks go to Phil Williams and Otto for tipping me to these.

Derek Sivers‘ plan, with all it’s bin2hex and regexp and back and forth between MySQL and PHP almost looks good compared to what I’m about to do. Really, why is it so difficult to go from latin1 (tables created back in MySQL 3) to utf8? Not only do you have to set the charset on the table, but also the connection, in PHP, and flipping everywhere. And then you’ve gotta deal with all this old data that’s in the wrong character set.

I got all excited when I learned of CONVERT() and CAST(), and felt completely betrayed when they turned out to be basically useless to me.

CONVERT() with USING is used to convert data between different character sets. In MySQL, transcoding names are the same as the corresponding character set names. For example, this statement converts the string 'abc' in the default character set to the corresponding string in the utf8 character set:

SELECT CONVERT('abc' USING utf8);

Thing is, I can’t figure out where it’s inheriting “the default character set,” and I can’t seem to come up with a combination of settings that will do the job. This old doc looked promising at first, but quickly let me down.

Ack. It looks like I’m going to have to read the contents of a table using PHP and latin1 connection encoding, then insert the data into a duplicate table using utf8 connection encoding. Either that, or Derek Sivers’ 28 step process.

In the interest of documenting even the things that don’t work: I had high hopes for this, but it doesn’t do anything useful.

if($tables = $wpdb->get_col('SHOW TABLES;')) {
	// For every table in the database
	foreach($tables as $table) {
		$changes = array();
 
		// create the new table, set it to UTF8
		$changes[] = “CREATE TABLE `conv_$table` LIKE `$table`”;
		$changes[] = “ALTER TABLE `conv_$table` DEFAULT CHARACTER SET=utf8”;
 
		// copy the data into the new table, transcoding to UTF8
		$fields = $wpdb->get_results(“DESCRIBE {$table};”);			
		$allfields = $selectfields = array();
		foreach($fields as $field){
			$allfields[] = $field->Field;
			$selectfields[] = (stripos($field->Type, 'char') || stripos($field->Type, 'text')) ? “CONVERT(`$field->Field` USING utf8)” : “`$field->Field`”;
		}
		$changes[] = “INSERT INTO `conv_$table` (`”. implode($allfields, '`, `') .“`) SELECT ”. implode($selectfields, ', ') .“ FROM $table”;
 
		// move the old table aside and put the new one in place
		$changes[] = “RENAME TABLE `$table` TO `preutf8_$table`”;
		$changes[] = “RENAME TABLE `conv_$table` TO `$table`”;
 
		// echo out the mysql commands
		echo “<h2 id="11972_changes-to-table-tab_1" >Changes to table <code>$table</code>:</h2>”;
		echo implode($changes, “;<br />\n”);
	}
}

mysql, character set, encoding, utf8 conversion

how to install and configure MySQL database server

yum, mysql, linux, yellow dog updater modified

Yes, I’m still trying to squeeze more performance out of MySQL. And since small changes to a query can make a big difference in performance…

Here are two really easy things to be aware of:

• Never do a COUNT(*) (or anything *, says Zach). Instead, replace the * with the name of the column you’re searching against (and is hopefully indexed). That way some queries can execute entirely in the keycache (while * forces MySQL to read every matching row from the table).
• When joining two large tables, but only searching against one, put the join statement at the end. Why join the two entire tables when you only have to join the matching rows?

I mention these because, well, I’ve known them forever, but upon seeing them again I realized I hadn’t really obeyed those simple rules in some of my queries.

Separately, there’s some pretty good info on what server variables affect what at mysqlperformanceblog too.

mysql, optimization, query optimization, performance, tips

Bam: MySQL error 28, and suddenly my queries came to a stop.

Error 28 is about disk space, usually the disk space for temp tables. The first thing to do is figure out what filesystem(s) the tables are on. SHOW VARIABLES LIKE “%dir%” will return a number of results, but the ones that matter are tmpdir and datadir.

SHOW VARIABLES LIKE “%dir%”;

basedir /
character_sets_dir /usr/share/mysql/charsets/
datadir /var/lib/mysql/
innodb_data_home_dir
innodb_log_arch_dir
innodb_log_group_home_dir ./
innodb_max_dirty_pages_pct 90
slave_load_tmpdir /tmp/
tmpdir /tmp/

As it turns out, df shows plenty of space available on all the filesystems:

# df -H
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 195G 11G 175G 6% /
/dev/sda1 98M 16M 78M 17% /boot
none 1.1G 0 1.1G 0% /dev/shm
/usr/tmpDSK 508M 12M 471M 3% /tmp
/tmp 508M 12M 471M 3% /var/tmp

Well, plenty of space if all it needs is 471MB of temp. Unfortunately, that’s not enough. The temp table is for a group/sort operation on a table that was much larger than I expected, something I realized when I looked at the table’s Data_length with SHOW TABLE STATUS.

I could change the temp directory, but unfortunately it can’t be changed dynamically, only at startup. So my choice was either to shutdown MySQL or change my query. As it turned out, it wasn’t a bad time to figure out how to write a better query.

mysql, error 28, error, disk space, temp table, table size, query optimization, mysql optimization

Peter Gulutzan, author of SQL Performance Tuning, writes in The Full-Text Stuff That We Didn’t Put In The Manual about the particulars of word boundaries, index structure, boolean searching, exact phrase searching, and stopwords, as well as offering a few articles for further reading (Ian Gilfillan’s “Using Fulltext Index in MySQL”, Sergei Golubchik’s “MySQL Fulltext Search”, Joe Stump’s “MySQL FULLTEXT Searching”). It’s one of a number of articles in the MySQL Tech Resources collection.

mysql, full text, keyword searching, full text index, mysql indexing, stopwords, Peter Gulutzan