A little more than two years after I realized how (really) bad the problem was and about 18 months after I prototyped my solution, our new library website, catalog, and knowledgebase launched last week — just in time for the fall semester opening.

It’s all built on Scriblio, includes a very simple new books list that you can narrow by subject and get via RSS. And if you search for subject areas like anthropology, economics, english writing, or any of a few dozen other topics, you’ll find our librarians’ subject guides listed at or near the top to help you out. You can also use the facets, clustered metadata shown in the right sidebar that reflect the aggregated results of that search†, to easily explore the collection or find the exact resource you need.

This started out simply, but the distance from prototype to working, um, product is difficult, dangerous, and frustrating. Still, when successful, it’s also wonderfully gratifying. And none of this would have happened without the help and support of a number of friends and colleagues both inside and outside the my library (callouts: Zach, Matt, Lichen, Jessamyn, Jon, Elaine, David, Anne, Dwight, Chris, PSU, The Andrew W. Mellon Foundation, and a lot of fellow library bloggers who’ve shared stories, spread the word, and helped make magic).

Thank you. Thank you. Thank you.

One more thing: This site isn’t perfect, but it’s a library catalog we can fix. Tell me what’s wrong, or better yet, download the software, join the list, and let’s work on it together so we can all have a better system.

† a set of facets can show you that anthropology is related to history, sociology, and ethnology, and help you narrow any of those subjects down to field work, methodology, or study and teaching.

library, libraries, lib20, library 2.0, Plymouth State University, Lamson Library, website, launch, Scriblio, WPopac

ALA Midwinter 2007, ALCTS Cataloging Norms Discussion Group presentation: Metadata and faceted searching: an implementation report based on WPopac. (slides: QuickTime & PDF.)

Faceted searching such as that made possible by WPopac (look for the new name soon) improves the usability of our systems and findability of our materials, but also puts new demands on how we catalog them.

My favorite search example is sociology of education, both because it’s a common search in our logs, but also because it demonstrates how our systems can help bridge the gap between what our users know and what our catalogs know. That is, the user doesn’t have to know that the proper term is educational sociology, the aggregated facets (or clusters) gently reveal that to the user.

Cookery may be an even better example. Not because it’s the subject term everybody loves to hate, but because a search for cookbooks has a high likelihood of returning cookery as a facet.

The problem, though, is that cookery is often cataloged as cookery, French (650 $a cookery, French) instead of cookery — French (650 $a cookery $a French).

And all those other regional divisions of cookery are there too, each one trying to claim its own place in the cluster of subject headings. The result is that, while educational sociology easily rises to the top of the results for a relevant search, cookery does not.

There are, of course, other headings. Art is divided almost the same as cookery is, giving us art, Indic, cookery, Indic, sculpture, Indic, and probably a dozen others.

If properly used, the metadata in our collections will set libraries apart from other information services in terms of usability and findability. The improvements to searching that WPopac offers rest directly on the body of good cataloging data in the collection. I look forward to even more improvements in the findability of library materials as we start to consider the new ways the metadata we’re creating can be used.

alamw2007, alcts, cataloging norms, midwinter, presentation, subject headings, wpopac

Okay, before anybody inquires if I’ve gone into boat sales or brings up the BisonBoom story again, I need to ask for your understanding. It’s not that I’ve been spending my days trying to pick out just the right shade of red for my new Corvette (really I’m not, it’s the Lotus I like), or that I’ve been moving to sunny California to take up my new job at Google (a year ago I would have been twitching with excitement, now I’m more likely to agree with this). No, my relative quiet can be partly attributed to the fact that the award has made WPopac a part of my day job, but in saying that I don’t want to suggest that I previously spent my daytime hours surfing the web and blogging =-).

Amidst the silence a few things have been happening, and I’ll soon be better able to talk about them. In the meantime, there are some clues out there, but what do they lead to?

For now I’m headed to Seattle for Midwinter. It’s a bit of a birthday for WPopac, as I introduced it in a presentation there last year. I especially like conferences for their role in spreading ideas like wind distributes pollen, which is my way of saying don’t hesitate to say hi or invite me to a drink if you spot me (I’ll be easier to spot now, I’m the one next to Lichen)

WPopac, lib20 library 2.0, libraries, quietude, silence

I’ve been following WP2.1 development, but Aaron Brazell’s post in the development blog wrapped up a lot of questions all at once.

The short story is that 2.1 is going to bring some really good changes that will allow more flexibility and better optimization of WPopac. Of the four changes Brazell names, the last two, the addition of the post_type column and a change in usage of the post_status column, are where the money is.

I’m awaiting the final release of 2.1 before building the necessary changes into WPopac, but the benefits will be worth it.

WordPress 2.1, WPopac, optimization, development

The press release:

Making Libraries Relevant in an Internet-Based Society

PSU’s Casey Bisson wins Mellon Award for innovative search software for libraries

PLYMOUTH, N.H. — You can’t trip over what’s not there. Every day millions of Internet users search online for information about millions of topics. And none of their search results include resources from the countless libraries around the world—until now.

Casey Bisson, information architect for Plymouth State University’s Lamson Library, has received the prestigious Mellon Award for Technology Collaboration for his ground-breaking software application known as WPopac. The Wpopac software will revolutionize the online search process by allowing titles and descriptions of library holdings to be found on the Internet.

The award was presented at a ceremony hosted by the Mellon Foundation on Monday, Dec. 4 at the fall meeting of the Coalition for Networked Information, in Washington, D.C. Bisson’s project was selected as one of only 10 recipients out of several hundred nominees for 2006, the first year the MATC awards have been granted. The decision was made by an all-star panel that included Tim Berners-Lee, the inventor of the World Wide Web, and Mitchell Baker, CEO of the Mozilla Foundation.

The Andrew W. Mellon Foundation supports the thoughtful application of information technology to a wide range of scholarly purposes, including developing digital technologies to enhance research, teaching, and online and distance learning, and new technical approaches to archiving text and multimedia materials.

Christopher Mackie, program officer for the Mellon Foundation’s Research in Information Technology section, was pleased with how well WPopac fits the foundation’s criteria.

“The award committee was particularly excited by the way WPopac makes library patrons more active participants in their library experience,” Mackie said. “By allowing patrons to add information to library records online, the software allows the community to work together to make their library resources more informative and more valuable. When you couple this with the reduced costs of access that WPopac permits, and the enthusiasm with which it has been received by librarians and patrons alike, the committee judged the project to have a truly revolutionary potential.”

“For years we’ve been talking about the digital divide in terms of access, and we’ve been working hard to put computers and networks into every school and library,” Bisson said. “But those same libraries, and their communities, are invisible to people online. If libraries are to be more than study halls in the Internet age, if they are to continue their role as centers of knowledge in every community, they need to be findable and available online. They need the tools to represent their collections, their services, and the unique history of their communities online. That’s what WPopac does.”

Dwight Fischer, director of information technology at PSU, called Bisson’s work an appropriate centerpiece for the university’s transformed academic library. “Over the past year, Lamson Library has implemented what is known as a Learning Commons,” Fischer explained. “This joint effort between library and IT professionals brings more technologies, online research materials, academic tutoring, writing and reading services to a central location in the library. Library faculty and staff members work side-by-side with IT professionals, forming a collaborative team that better reflects the needs of today’s students. Casey’s project will help build more bridges to more information for more people. We’re very proud of him.”

andrew w mellon foundation, award, mellon foundation, press release, wpopac, libraries, future libraries, lib20, library 2.0

MAIUG 2006 Philadelphia: Designing an OPAC for Web 2.0 (interactive QuickTime with links or static PDF)

Web 2.0 and other “2.0” monikers have become loaded terms. But as we look back at the world wide web of 1996, there can be little doubt that today’s web is better and more useful. Indeed, that seems to be the conclusion millions of Americans are making, as current estimates show over 200 million users in the US, including 87% of youth 12-17.

Web 2.0 isn’t driven by technology, it’s driven by that critical mass of users. And while social software and AJAX enabled web applications get most of our attention, people are turning to the internet for some very mundane everyday activities that were little more than science fiction in 1996. The commonality of internet banking, for example, reflects the trust users now have in the security and reliability of online services.

But the web has weathered so much hype and hyperbole that it may be difficult to recognize its arrival as a true cultural force. Computing has become so common that children often learn to type before they learn to write. And the instant, self-service access to worlds of information and services is changing industries — a fact we can see clearly in the decline of the role of travel agents, even while air travel continues to grow.

Kevin Kelly, in a Wired Magazine story described this apparent blindness:

The accretion of tiny marvels can numb us to the arrival of the stupendous. [thanks to Josh Porter for alerting me to this]

So the question of how to design a web OPAC for today is a question of how to design an information service in a world rich with information services and filled with users who make information seeking — though not necessarily at libraries — part of their everyday lives.

note: this is an update of my IUG2006 presentation.

Mid-atlantic innovative users group, iii, lib 2.0, libraries, library, library 2.0, library catalog, maiug, maiug2006, online catalog, opac, opac 2.0, presentation, web 2.0, web opac, wpopac

I got itchy about magic quotes the other day because it’s the cause (through a fairly long cascade of errors) of some performance problems and runaways I’ve been seeing lately (pictured above).

But I deserve most of the blame for allowing a query like this to run at all:

SELECT type, data, count(*) AS hits
	FROM wpopac_WPopac_bibs_atsk
	WHERE data LIKE '%'
	AND type IN ('subjkey','author', 'title')
	GROUP BY data
	ORDER BY hits DESC
	LIMIT 7

As executed, it’s trying to select all 1.2 million records from the table, group them (a hugely memory intensive activity with text), and then report only the top seven.

It’s one of a few queries that returns the alternate searches in WPopac. I’m planning to rethink how that works, but I also should have built in a fail-safe to prevent it from trying to execute if the search variable is empty (the query above is written WHERE data LIKE '$searchterms%' or something like that). I’m amused that MySQL tries so hard to fulfill the search — so hard that it’ll consume all available RAM and swap to do so.

The sysadmin and DBA are now much happier — along with any user frustrated by poor performance — with this patched.

bug, bugs, mysql, patch, performance, queries, query, query failures, wpopac

Somewhere in the WordPress code extra slashes are being added to my query terms.

I’ve turned GPC magic quotes off via a php_value magic_quotes_gpc 0 directive in the .htaccess file (we have far too much legacy code that nobody wants to touch to turn it off site-wide). And I know my code is doing one run of addslashes(), but where are the other two sets of slashes coming from?

addslashes, gpc magic quotes, php, wordpress, wpopac

I’ve re-thought the contents of the record and summary displays in WPopac. After some experimentation and a lot of listening, it became clear that people needed specific information when looking at a search result or a catalog record.

So now, when searching for Cantonese slang, for instance, the summary displays show the title, year, format, attribution, and subject keys of each result. And when viewing the record for A Dictionary Of Cantonese Slang you’ll get all of that and more.

Attribution? Yeah, the 245 subfield C that never gets much attention in our catalogs, but is so useful to actual humans who might want to read the work. Subject keys? Let’s leave that for another post.

On the downside, I let a bug sneak in that shows an empty list of 856 URLs in each record (records that do have 856s display properly). Though this may still be called an incremental improvement over my last version that didn’t show the 856s at all. Another bug is that records without publication dates are showing pub dates in (and getting filed under) year zero. Ironically, the most common source of this problem is in catalog records for websites.

Walt is among those that I think will appreciate these changes, but I’m anxious to point out, despite this gleeful post on the subject, how pedestrian these changes are. That is, anybody with a bit of skill with XHTML and PHP — say any of the thousands of people developing themes for WordPress — should be able to shape the record templates to their liking in moments.

My delay in this is a mixture of laziness, being too busy elsewhere, and simply wanting to see how things work. I’m amused, for instance, that I haven’t heard anybody ask for the ability to sort search results by date or author.

update: fixing the pubdate and 856 problems right now.

lib20, libraries, libraries 2.0, record display, search result display, wpopac

WPopac, a project I started on my nights and weekends, is now officially one of my day-job projects too.

We’ve been using our WPopac-based catalog as a prototype since February 2006, but the change not only allocates a portion of my work time specifically to the development of the project, but also reflects the library‘s decision to transition to WPopac as a our primary web OPAC.

Work to make a general release of the WPopac software available for download and use by any library (or anybody who wants to present structured data with faceted searching on the web) is in progress. And, as noted here, I’m also working with other libraries to bring WPopac-based catalogs online elsewhere.

In the meantime, I’m headed to WordCamp in San Francisco on August 5th, and the new WPopac blog has become the official source of news and information regarding the project.

lib20, libraries, library, library 2.0, library catalog, news, official, online catalog, opac, plymouth state university, wpopac

I recently submitted my proposal for the 2006 NELINET Information Technology Conference.

It’s about WPopac, of course, but the excitement now is that the presentation would be the story of the first library outside PSU to implement it.

WPopac is an open source replacement for a library’s online catalog that improves the usability, findability, and remixability of the library’s collection.

This presentation will detail the implementation of WPopac in the real world, including discussion of challenges and costs, as well as the improvements to service and increased access to library materials.

I’m keeping mum about who’s implementing just now, but the three presenters in the proposal include me, the web designer and usability engineer, and the library director.

Cross your fingers for us, and stay tuned for a public release of this mystery library’s new catalog soon.

update: I should have noted that I submitted this proposal late (it was due Friday, I submitted Monday). Siobhan, who has the enormous task of organizing the conference, wrote back to thank me for the proposal, but explained that the conference committee had already selected speakers — “I cracked the whip on this because we’re running a little behind in our planning.”

Still, she did suggest:

However, we would love to have you and your colleagues come speak at a NELINET seminar, particularly now as you have information about implementation and usage of WPopac.

it conference, lib20, libraries, library, library 2.0, library catalog, nelinet, opac, proposal, wpopac

Following Edward Tufte’s advice, I’ve been wanting to offer a presentation without slides for a long time now; I finally got my chance in Portland. The downside is that now I don’t have anything to offer as a takeaway memory aid for my talk. My speaking notes are too abstract to offer for public consumption, but below are the URLs from them along with a tiny bit of context.

Foundation

Increasing use of the web is changing our expectations of information services and places greater demands of self-service on them. If “Web 2.0” has any meaning, it’s this notion that internet services are no longer the stuff of science fiction, but a part of our every day reality.

One interesting reflection of this increasing usage and comfort with the web is the development of social software like MySpace, Facebook, blogs, wikis, social bookmarking, Flickr, and also LibraryThing.

It takes a moment, sometimes, to realize what’s changed in the ten years since the first Mosaic browser opened the web to a mass audience. Kevin Kelly tried to explain that when he noted: “The accretion of tiny marvels can numb us to the arrival of the stupendous.”

Okay, now what?

We need to understand how people now search for and interact with information. Part of that means making peace with search engines and making sense of “findability.” Peter Morville’s Ambient Findability addresses this question in terms directly relevant to libraries. To that I add the notion of The Google Economy and a set of rules for participation (and findability) in it:

• Linking must be possible
• Linking must be desirable
• Linking must be measurable

I argue that libraries are rich with the stuff people would like to link to, but the architecture of our systems often fails us on the other aspects.

I’ve been exploring this with my WPopac project, and I’ve seen some interesting results in the four months that it’s been live and available to the public. One example is that a web search for “joe monninger” returns the WPopac page as the top hit. Elsewhere, WPopac content is appearing in blogs (examples: Fuzzyfruit and Angie) and as a result some of the books in WPopac are now highly ranked in web search engines (example: A Baby Sister For Frances is now the only non-commercial result in the first page of Google results).

A rather more controversial look into how the Google Economy works can be found in Google’s explanation of their search results. Interestingly, the same factors within the Google Economy that created the controversy also made possible a solution: the top search result for “Jew” is now the Wikipedia page.

How can we track our participation in the Google Economy?

Alexa might be best described as the Nielsen ratings for the web. Tracking the daily changes in reach and rank (and looking at all the variations of the graph) can be rather addictive. Alexaholic serves that fix by offering more varied views of the same data.

It should be said, however, that the results in Alexa are the slowest to reflect changes or improvement in a service’s role in the Google Economy. A more immediate pulse of things can be taken at Technorati or within Google.

Last word

As we consider ways to improve our online services — as we look to build the online library of the near future — these words echo in my mind:

Nobody cares about you or your site. Really. What visitors care about is getting their problems solved. Most people visit a web site to solve one or more of the following three problems.

• They want/need information
• They want/need to make a purchase / donation.
• They want/need to be entertained.

Too many organizations believe that a web site is about opening a new marketing channel or getting donations or to promote a brand. No. It’s about solving your customers’ problems. Have I said that phrase enough?

In short, our libraries’ web sites are the online embodiment of our libraries. Our patrons don’t want to know how to use our library, they want to find in our online services the value that libraries offer in their in-person services. They want online services that deliver answers.

findability, future libraries, google economy, lib 2.0, libraries, library, library 2.0, online libraries, opac 2.0, presentation, usability, web, web 2.0, wpopac

IUG 2006 presentation: Designing an OPAC for Web 2.0 (also available as a PDF with space for notes)

Web 2.0 and other “2.0” monikers have become loaded terms recently. But as we look back at the world wide web of 1996, there can be little doubt that today’s web is better and more useful. Indeed, that seems to be the conclusion millions of Americans are making, as current estimates show over 200 million users in the US, including 87% of youth 12-17.

Web 2.0 isn’t driven by technology, it’s driven by that critical mass of users. And while social software and AJAX enabled web applications get most of our attention, people are turning to the internet for some very mundane everyday activities that were little more than science fiction in 1996. The commonality of internet banking, for example, reflects the trust users now have in the security and reliability of online services.

But the web has weathered so much hype and hyperbole that it may be difficult to recognize its arrival as a true cultural force. Computing has become so common that children often learn to type before they learn to write. And the instant, self-service access to worlds of information and services is changing industries — a fact we can see clearly in the decline of the role of travel agents, even while air travel continues to grow.

Kevin Kelly, in a Wired Magazine story described this apparent blindness:

The accretion of tiny marvels can numb us to the arrival of the stupendous. [thanks to Josh Porter for alerting me to this]

So the question of how to design a web OPAC for today is a question of how to design an information service in a world rich with information services and filled with users who make information seeking — though not necessarily at libraries — part of their everyday lives.

note: this is an update of my ALA Midwinter presentation.

iii, innovative users group, iug, iug2006,lib 2.0, library, library 2.0, library catalog, online catalog, opac, opac 2.0, presentation, web opac, wpopac, libraries, web 2.0

A discussion on Web4Lib last month raised the issue of Google indexing our library catalogs. My answer spoke of the huge number of searches being done in search engines every day and the way that people increasingly expect that anything worth finding can be found in Google.

There were doubts about the effectiveness of such plans, and concerns about how frustrating it might be for a searcher in California to find books (that he or she can’t access) in New Hampshire.

My answer to the first point was that once we start participating in the Google Economy, we’ll find our records well represented within it, and my answer to the second point is that we already have good solutions to that problem: ILL and OpenWorldCat. Examples: a Google search for my favorite author/friend/example returns with WPopac among the top results. And if you view one of the resulting records, you’ll see a link to “find in WorldCat Libraries.”

Thing is, it’s not just the stuff I’ve been linking to as examples that’s getting found in search engines. Listed below are the top 100 incoming search terms to WPopac from major search engines in the last week. The list is generated by bsuite, my multipurpose WordPress plugin, and the links lead to the item found with the search terms.

• What works: documenting energy conservation in buildings
• Online Recording of Pomp and Circumstance
• recording of pomp and circumstance
• harry stack sullivan
• miguel de unamuno website
• “cathedrals +england”
• athalie
• symbols in art
• Dadaism by Marcel Duchamp and Frances Picabia
• frank moake
• william luijpen
• “Man, Culture and Society”
• music scores elgar wand of youth slumber song
• Cats and Bats and Things with Wings by Conrad AIken
• paul cuffe african american
• biography of george e. mowry
• don giovanni libretto italian english
• Steroids-opposing viewpoints
• ballet plot
• ballet plots
• grice, h.p., studies in the way of words
• african american identity
• Allan Freed and the Big Beat
• The Blue Octavo Notebooks
• The Self Reliant Potter
• kartinki
• the little brown book of anecdotes
• “the fields are full”+“Armstrong Gibbs”
• forty french songs for voice and piano
• Fantasien, Op. 116
• Literary Themes
• biography ramses the great
• ramses II essays
• Rita Rapoza
• deutsche nobelpreisträger
• Erotica
• ‘henry and mudge and the green time’ website
• “Henry and Mudge in the Green Time”
• tally’s corner
• ECCENTRIC MUSCLE LOADING BASEBALL
• indian mythology 0600023699
• climacteric psychology menopause
• emilie flöge
• Brigance Comprehensive inventory of basic skills
• Brigance Diagnostic Comprehensive Inventory of Basic Skills
• Brigance Diagnostic Inventory of Basic Skills
• brigance diagnostic
• Brigance INventory of Basic Skills
• brigance inventory
• brigance testing
• Brigance
• Metropolitan readiness tests
• Bayley assessment kit
• kaja silverman “fragments of a fashionable discourse”
• palmer hayden biography african-american
• feminist theories on battered women syndrome
• otis lennon mental ability test
• otis-lennon intelligence test
• Otis-Lennon School Ability Test
• Poetry from norton anthology of by s.m.gilbert and susan gubar woman writer
• Black Frontiers
• Pioneers Of The American West.
• joycelyn elders biography
• pros and cons of tqm
• Encyclopedia of world biography
• Death Penalty:an historical& Theological survey,J.J.Megivern
• “Jewish Deliberations on Suicide: Exceptions, Toleration and Assistance” — Noam Zohar
• Samuel H. Kress biography
• biomechanical analysis leg stretching
• arguments against suicide
• assisted suicide
• assited suicide
• physician assisted suicide cartoons
• “ice age”+homophobia
• suite española Gaspar Sanz
• leprosy:king baldwin IV
• Socolow The women of colonial Latin America.
• popular music and youth culture
• mandarin revolution Keynes
• sports professionalization test
• “criminology theories, patterns, and typologies”
• xiajia
• Pangwe Africa
• “hanif kureishi+life”
• All Shook Up: How Rock ‘n“ Roll Changed America
• Fraenkel and Wallen validity and reliability
• j.k. rowling biography isbn
• tina modotti biography
• essays on Rescuing a planet under stress
• criticism beatrix potter
• harry potter literary criticism
• ”Jewish Women in the Holocaust“
• crimes and misdemeanors plato
• California and the Southwest history
• Life In the American West
• funny chemistry caricature
• di vinci ”symbols“
• di vinci symbols
• Symbols of japan
• biograph dewey

Some links will leave you scratching your head, others are clearly misdirected. But, I’m especially proud of this link, from a person who was especially happy to get a new book. Making our collections indexable also makes them linkable, and that means people can make libraries part of their lives — wherever their lives take them.

And this doesn’t just help Angie, it means faculty and students can link to library resources from class blogs or share them in AIM.

google economy, google in the catalog, lib20, libraries, library, library 20, library catalog, linking, links, loosely linked, opac, remix, search engines, wpopac

WPopac takes advantage of MySQL’s indexing and relevance-ranked searching (go ahead, try it), including boolean searching (on MySQL versions > 4.x). Here are some details and examples taken wholesale from the MySQL manual:

• +
  A leading plus sign indicates that this word must be present in each result returned.
   
• -
  A leading minus sign indicates that this word must not be present in any of the resuls that are returned.
   
• > <
  These two operators are used to change a word’s contribution to the relevance value that is assigned to a result. The > operator increases the contribution and the < operator decreases it.
   
• ( )
  Parentheses group words into subexpressions. Parenthesized groups can be nested.
   
• ~
  A leading tilde acts as a negation operator, causing the word’s contribution to the result’s relevance to be negative. This is useful for marking “noise” words. A row containing such a word is rated lower than others, but is not excluded altogether, as it would be with the – operator.
   
• *
  The asterisk serves as the truncation (or wildcard) operator. Unlike the other operators, it should be appended to the word to be affected. Words match if they begin with the word preceding the * operator.
   
• “
  A phrase that is enclosed within double quote (‘”’) characters matches only results that contain the phrase literally, as it was typed.
   

In short, it supports the quotes and plus/minus operators that people are familiar with in Google and others. The following examples demonstrate some search strings that use boolean operators:

• ‘apple banana‘
  Find records that contain at least one of the two words.
   
• ‘+apple +juice‘
  Find records that contain both words.
   
• ‘+apple macintosh‘
  Find records that contain the word “apple”, but rank records higher if they also contain “macintosh”.
   
• ‘+apple -macintosh‘
  Find records that contain the word “apple” but not “macintosh”.
   
• ‘+apple ~macintosh‘
  Find records that contain the word “apple”, but if the row also contains the word “macintosh”, rate it lower than if row does not. This is “softer” than a search for ‘+apple -macintosh’, for which the presence of “macintosh” causes the row not to be returned at all.
   
• ‘+apple +(>turnover ‘
  Find records that contain the words “apple” and “turnover”, or “apple” and “strudel” (in any order), but rank “apple turnover” higher than “apple strudel”.
   
• ‘apple*‘
  Find records that contain words such as “apple”, “apples”, “applesauce”, or “applet”.
   
• ‘“some words”‘
  Find records that contain the exact phrase “some words” (for example, rows that contain “some words of wisdom” but not “some noise words”).
   

Now I really need to configure my own version of MySQL without the over-reaching stopword list.

libraries, library, library catalog, online catalog, opac, wpopac

I’ve whittled things down to the point where the only baseline change from WordPress 2.0.2 is in the next_posts_link function of the wp-includes/template-functions-links.php file. The change is necessary because WPopac rewrites the SQL search queries in a way that’s incompatible with a piece of this function, but necessary for performance reasons.

Here’s how my version reads:

function next_posts_link($label='Next Page »', $max_page=0) {
	global $paged, $result, $request, $posts_per_page, $wpdb, $max_num_pages;
	if ( !$max_page ) {
			if ( isset($max_num_pages) ) {
				$max_page = $max_num_pages;
			} else {
				preg_match('#FROM\s(.*)\sGROUP BY#siU', $request, $matches);

				// added April 5 2006 by Casey Bisson to support WPopac
				// necessary because the preg_match above fails with some queries
				if(!$fromwhere)
					$fromwhere = $wpdb->posts;

				// changed April 5 2006 by Casey Bisson to speed the query by eliminating
				// the slow DISTINCT clause
				//$numposts = $wpdb->get_var(“SELECT COUNT(DISTINCT ID) FROM $fromwhere”);
				$numposts = $wpdb->get_var(“SELECT COUNT(*) FROM $fromwhere”);
				$max_page = $max_num_pages = ceil($numposts / $posts_per_page);
			}
	}
	if ( !$paged )
		$paged = 1;
	$nextpage = intval($paged) + 1;
	if ( (! is_single()) && (empty($paged) || $nextpage < = $max_page) ) {
		echo '<a href=“';
		next_posts($max_page);
		echo '”>'. preg_replace('/&([^#])(?![a-z]{1,8};)/', '&$1', $label) .'</a>';
	}
}

baseline modification, hack, mysql query optimization, wordpress, wordpress hacking, wpopac

Between WordPress 1.x and 2.x there was a big change to the way rewrite rules are handled.

In the old days, everything got written out to a .htaccess file. Every condition, every form of permalink could be found there, and I had some comfort knowing I could see and mess with it all. I was a bit surprised to find that with 2.0.2, WP writes out a sparse file that has only one significant rule. Something that looks like this:

RewriteRule . /blog/index.php [L]

That one line takes everything in your request URL and passes it through to index.php for processing. The beauty of this is that WP doesn’t need to write to the file system with every change to the permalink structure (which used to include every new “page” added or renamed).

The only downside to this is that you can no longer expect a $_REQUEST array full of all the query terms your plugin might use. Instead you’ll have to use the $wp_query object. All of this was probably mentioned in the API docs before I built a bunch of dependancies on $_REQUEST, but it was easy enough to fix.

$wp_query, $_REQUEST, .htaccess, apache, apache rewrite, rewrite rules, wordpress, wpopac

Ross Singer gets the prize for submitting the first reader contributed tag, the latest feature in bsuite.

There are arguments about whether user-contributed tags are useful or even valid, or whether they should be stored in my site or aggregated at places like del.ici.ous. But who’s to worry about such questions? Who’s to worry when you can put together the work already done to support author’s tags with WordPress’s pretty good comment system and get user contributed tag support with just a few extra lines of code? Who’s to worry when we can try it and see what comes of it?

It’s all managed using the same tools we use to approve, moderate, and edit comments, which also means the spam filtering that works for comments works for contributed tags too. And because bsuite is already part of WPopac, that means it gains the new tagging features too (well, it will soon).

bsuite, collabulary, folksonomy, tag, tagging, tags, user contributed, wisdom of the crowd, wpopac

Jonathan Schwartz made the kind of news that makes Slash Dotters happy: he announced Sun is (sort of) giving away free servers. It’s a promotion, a media play, of course, but one that might make a few lucky people very happy.

Here’s the deal: Sun is really proud of their new T2000 eight core server. Each core runs at 1.2GHz, but they’re apparently applying some distributive power of multiplication and calling it an 9.6GHz machine and touting some impressive performance benchmarks. But rather than play the game of “our experts are better than your experts” against other vendors, they’ve extended a pretty generous try and buy program to users themselves. And, as Jonathan made noise in his blog post, a few people might get lucky enough to keep their machines.

So what’s my angle? WPopac rocks, and performance is excellent with my test corpus (160,000 bib records) on even a lowly single-CPU server with only 1GB of RAM, despite the fact that each bib record is represented in both its composed form in the wp_posts table and as about 20 more rows per record in a separate table (yup, I’ve got almost 3 million rows representing every atomic detail of every bib record), and all of this is full-text indexed.

And I’m fully confident that when I put our entire catalog into WPopac, all 330,000 bib records (resulting in about 6.2 million atomic records), performance will still be up to the task. My math suggests everything should be ducky on a relatively budget server up beyond about 1 million bib records, but what happens for libraries that have more than that, say, perhaps 6 to 8 million bib records (again, 110 to 150 million atomic records; again, all full-text indexed in MySQL)?

It would seem that WPopac’s combination of, um, large MySQL databases with high-volume Apache/PHP should be an ideal fit for the T2000′s big RAM space and support for high numbers of parallel threads. Jonathan claims the T2000 costs a modest $5,000 (but the online store shows a higher price?) and outperforms everything else, but is it up to WPopac? Do they want to try?

C’mon Sun, toss a T2000 this way, let’s see what happens? (Heck, I’ll even try a T1000.)

sun, solaris, t2000, coolthreads, server, 9.6ghz, challenge, sparc, ultrasparc, 8core, free, wpopac, performance, big db, Jonathan Schwartz , try-n-buy, try and buy, free server

As with all my other presentations, the my slides tell less than half the story, but I’ve posted them anyway. I’m told the audio was recorded, and there’s a chance that will help explain all this, but until then you’ll have to piece this all together from my previous writings, what little I’m about to offer here, and the slides (which, again, without the spoken component, probably do more to misdirect interested readers than answer questions).

My code4lib 2006 presentation included discussion not only of (WPopac, my open source OPAC based on WordPress, but also a plea for us within the libraries to look outside our community for practices and standards that are in use and supported by larger populations than we can ever hope for on our own. WPopac is one attempt at that, using an application that can already claim “hundreds of thousands” of current users and many thousands of developers. Amazon offers another example, boasting 140,000 registered developers of its API, making it the defacto standard for the exchange of bibliographic information online. Meanwhile, our community of programmers within libraries, which is far smaller than 1% of Amazon’s registered API users, must contend with dozens of metadata standards (MARC, MODS, METS, DC, etc., etc., etc.) and communication interfaces (SRU/SRW, z39.50, and more) to do substantially similar work.

I’ve talked about this before (here, here, here, and here, among others), and I’ll be talking about it more yet. Most exciting for me, I wasn’t alone in my plea, as Art Rhyno made some great points about how our acquisitions and accounting processes are substantially similar to what’s called ERP in the outside world.

library, libraries, standards, wpopac, code4lib, presentation, great wall of standards, population density, sustainable development, sustainability, programmers, coders, developers, isolation, future libraries, library 2.0

First things first, this thing probably needs a better name, but I’m not up to the task. Got ideas? Post in the comments. For the rest of this, let’s just pretend it’s an interview.

What is WPopac? It’s an OPAC — a library catalog, for my readers outside libraries — inside the framework of WordPress, the hugely popular blog management application.

Why misuse WordPress that way? WordPress has a a few things we care about built-in: permalinks, comments, and trackbacks (and a good comment spam filter), just to start. But it also offers something we’ve never seen in a library application before: access to a community of knowledge, programmers, and designers outside libraries. Because the core of WPopac is WordPress, and because it preserves WordPress’s rich plugin API and themes structure, it already has more users, designers, developers, and administrators than all the ILS vendors combined.

So, down with the ILS? Well, no. There are some brave souls working on full-fledged open-source ILSs, but that’s not my goal here. The ILS does a lot of stuff I don’t want to be responsible for, like the acquisitions workflow and financial, inventory, and circulation management. When you peak inside your ILS, you realize there’s a lot there you don’t want to have fix.

So, we have to have both an ILS and WPopac? Well, you don’t have to have anything, but if you want it, at least WPopac is free, extensible, and open-source. Less flippant answer: yes, it does assume there’s an ILS in the background somewhere, but more than a few people see potential for projects like this to serve underfunded libraries that may lack automation. That could be interesting.

But blog posts are unstructured and library data is full of structure. What gives? The standard WordPress content database is buttressed with extra tables to represent all the bibliographic information in its atomic detail. But even the ‘unstructured’ data takes some clues from the microformats camp, putting everything in XML parsable XHTML.

How’s that work again? Well, let me be careful here. I’m not proposing WPopac as a solution, rather as a framework for building a solution. That said, you can get a pretty good idea of how the first draft of this concept works by looking at a real record (be sure to view the source, as there are some hidden divs in there). But if you don’t like that, you can change the look by fiddling with the stylesheet or switching themes, and you can change the content with the WordPress API or by changing the way it’s loaded in the first place.

Further, because all the bibliographic data is there in its atomic detail, plugins can use and display that data anywhere on the page. Try a search to see how I’m using that data in the right column to improve findability, as in my clustered search results prototype from last fall.

So, does that mean I can do XYZ that I’ve wanted to do? Maybe. Anybody who knows how to write a WordPress plugin can take a stab at playing with all that data. The “refine search” content in the right column, and the “alternate searches” content at the bottom is generated that way. Try this one: I’ve finally got the Wikipedia results I’ve always wanted in the catalog, just look in the right column. Or take a look at the “add to del.icio.us” link in the record display, that’s generated by a regular wordpress plugin written by Arne Brachhold, who wasn’t thinking of libraries or OPACs when he wrote it. And down at the bottom of the page you’ll see the a list of related works that’s built by my own bsuite plugin. Want COinS-PMH/unAPI? The interface and all the data are there to make it happen, and here’s a good plugin to start from.

So no guarantees, but hey, give it a try. And if you run into trouble you’ll be among hundreds of thousands of WordPress users and supported by a huge community of plugin and theme authors.

What about RSS, XML, OpenSearch? WordPress solves the RSS feed for us (look at this URL to see). A feature-complete XML API, is a bit further off, but maybe somebody wants to pitch in to help solve that one? And full OpenSearch support, taking advantage of the suggested and alternate search features, is my next big project (here’s where I’m going with that).

This is awesome, can I run it at my library? Well, Jenny called dibs… But, really, this project started with my attempts to find a way to make my work sharable, so, yes. Call me a dreamer, but I find the notion of a community of libraries sharing plugins and code changes really exciting. But right now, there are three major components — the data importer, the plugin, and some modifications to the WordPress baseline code — and all of them need a little more work to make them distributable. Stay tuned.

This sucks, it doesn’t do X, and your plan for Y is all wrong. You’re probably right. This is my first stab at a really big problem, and there’s a lot that isn’t done and certainly a few things I didn’t think of. The plan here is to build a framework that let’s us ask questions, build possible solutions, and share them easily. The only thing I’m certain of is our need to find ways to make our systems easier to use, easier to extend, and integrated into the larger stream of progress that’s shaping the internet that over 200 million Americans are making an essential part of their lives. Take this as an invitation to get involved, there’s lots to do.

library, libraries, library 2.0, lib20, opac 2.0, opac, library catalog, library services, wordpress, wpopac, future libraries