OpenSearch is a standard way of querying a database for content and returning the results.

The official docs note simply: “Any website that has a search feature can make their results available in OpenSearch format,” then adds: “Publishing your search results in OpenSearch™ format will draw more people to your content, by exposing it to a much wider audience through aggregators such as A9.com.”

It’s a lot easier to understand OpenSearch once you’ve used it, so take a look at A9.com and do a search. A9 isn’t the only OpenSearch aggregator out there, but it’s a great example. You can query a number of OpenSearch targets by clicking the buttons to add columns (also try resizing the columns), or you can add any of the 422 public search targets listed at A9.

Now, if you’ve got the beta of IE 7, you can see how it’s extending beyond server-side aggregators and into client software. Even better, you can see how this is becoming automigical via autodiscovery.

One of the most exciting features of OpenSearch is its support for complex queries as well as simple keyword searches, and the ability to return intelligent responses to a search, such as alternate search suggestions (think spelling corrections) and facets (hey, any librarians attending this?)

Now, the question for libraries is when are we going to demand OpenSearch interfaces from our information providers? The inclusion of OpenSearch in IE7 more than gives it critical mass, but so far it seems to be just something a few progressive library-types are experimenting with. In the short term, imagine how improved our metasearch tools would be if based on fully-implemented OpenSearch feeds (with the facets and suggestions). In the long term, I can’t imagine any aspect of a library’s online services not touched by this technology.

a9, API, lib20, libraries, library, library 2.0, OpenSearch, search, search aggregator, search api

I prefaced my point about how the standards we choose in libraries isolate us from the larger stream of progress driving development outside libraries with the note that I was sure to get hanged for it.

It’s true.

I commented that there were over 140,00 registered Amazon API developers and 365 public OpenSearch targets (hey look, there’s another one already), but that SRW/SRU would always play to a smaller audience. Basing arguments on the popularity of the subjects is dangerous, especially so within the library community, and touching on such inflammatory arguments during a 20 minute presentation is certain to leave people feisty.

It’s also especially dangerous to use an apparently sacred cow as the object of what I wanted to be a general example. My overall argument was (and remains) that we should look for opportunities to break down the barriers that isolate our work and find means to expand our community. Still, I believe a specific argument about SRW/SRU has merit, and I’m willing to carry the flag on this side.

So let’s start with what I believe we can agree on: SRW/SRU, OpenSearch, and Amazon Web Services all serve substantially similar interests: the ability to issue a query, get a list of results, get a detailed record for each result (not possible with OpenSearch). From here, many people seem to argue that XSLT can be used to mutate the results of one schema to the other, or directly to browser-displayable content with ease. On the face of it, this seems to solve many of the incompatibilities while preserving the unique features of each.

Sadly, those XSLT arguments ignore one problem while creating another.

XSLT (and similar techniques) can change the representation of the data in a record, but they can’t change the type or nature of the data and such techniques certainly can’t address differences in the way applications interact with the API. As an example, consider that an XSLT could likely be written to translate Flickr’s schema for a single image into something that looks like Amazon’s schema for a single title, but no XSLT can make an application that interacts with one API properly interact with the other.

The problem that XSLT solutions ignore is that if all these schemas can be translated between eachother (either cleanly or not), and if catalogers working with one metadata standard must be aware of the limitations of other standards to which their work might get XSLT’d to, then what’s the value of their differences? Why invest the duplicated time and effort in each?

The rest of this argument assumes that XSLT solves neither the needs of the programmer who must still learn to navigate different APIs nor the cataloger who must either use lowest-common-denominator cataloging standards or write metadata that can’t be cleanly translated to other schemas.

With XSLT out of the picture, it becomes clear that SRU/SRW is indeed among the wall of standards that make it impossible for us within the library to share executable code with anybody outside our community. And because of our low numbers and natural variations in chosen environments (preferred language & database among them), we often find it difficult to share executable code among others within our community.

It’s also worth considering the differences in features between SRW/SRU, OpenSearch, and Amazon Web Services: Both OpenSearch and AWS offer ways to include suggested alternate searches within the search response set (OpenSearch does this especially well). Nothing I’ve seen in SRW/SRU does this (please correct me if I’m wrong), yet considering how much interest there is in developing more human search interfaces and those that allow faceted searching, these are clearly essential components of any useful standard.

Further, AWS supports all aspects of the usage of materials, not just the search and retrieval of them. Are AWS’s shopping cart and checkout features not similar to our circ checkout procedures? Could AWS’s list management features not be used to show patrons what they have checked out now or throughout their history (if we or they wanted that), as well as allowing them to maintain the reading wishlists or personal bibliographies?

And AWS’s support for returning related and recommended items for each record, as well as comments and reviews is outside the scope of SRW/SRU, but required for many of the features we want to add to our applications.

The point here is that while there are substantial differences in the details between SRW/SRU and OpenSearch or AWS, it is not easy to conclude that SRW/SRU is substantially better for the applications we seem to most want to build.

And this is when we have to take note of the recent University of California libraries report and the quote that puts us all in our places: “for the past ten years online searching has become simpler and more effective everywhere, except in library catalogs” (and the same could be said of our online databases).

The problem isn’t that we’ve been bad coders, and we certainly haven’t intentionally built systems that were difficult to use. The problem is that our community has been isolated and unable to leverage advances made elsewhere. Again, my argument is that we need to change this, that we need to find more ways to collaborate not only with those within our community, but with those outside our community.

There’s much we might be able to offer coders outside libraries, but the arguments defending SRW/SRU seem to ignore the lessons we might learn from them.

Final example: it’s pretty obvious to all of us now that chat reference should be done using common and freely available IM tools, but that didn’t stop us from investing huge sums of money in building and buying custom, library specific chat reference tools. Where else will history show we’ve made similar mistakes?

a9, amazon api, amazon web services, argument, AWS, cage match, code4lib, code4lib 2006, future libraries, information retrieval, lib20, libraries, library, library 2.0, library standards, opensearch, search, search and retrieval, search retrieval, sru/srw, srw/sru, web services

I just received this email from the A9 OpenSearch team:

We have just released OpenSearch 1.1 Draft 2. We hope to declare it the final version shortly, and it is already supported by A9.com. Uprading from a previous version should only take a few minutes…

OpenSearch 1.1 allows you to specify search results in HTML, Atom, or any other format (or multiple formats) in addition to just RSS. In addition, OpenSearch 1.1 will be supported by Internet Explorer 7, among other software, so we strongly recommend that you upgrade. Also new is the ability to specify suggested searches, such as spelling suggestions and related queries. (link and emphasis addded)

Woot! I’ll be doing something with this soon.

a9, opensearch, open search, amazon, search, libraries, library, opac, library catalog, library catalogs, a9.com, metasearch, aggregated search, search, federated search

A9, the search engine from Amazon.com, does some pretty interesting things that libraries should be aware of. First, any library considering a metasearch product should look at what can be done for free, and second, libraries should take a look at the OpenSearch technology that drives it.

So now, when searching for Harry Potter, you’ll also find relevant results from Plymouth State University’s Lamson Library. We’re not the first library — I think Seattle Public was — and my work mostly follows the cookbook written up by Ryan Eby, of Michigan State University Libraries. Thanks also go to our university IT sysadmins who installed the XSLT extension for PHP5 earlier this week.

tags: a9, lamson library, metasearch, michigan state university, opensearch, plymouth state university, ryan eby, search engine, search engines, search technology, seattle public library, university, web search, websearch, xslt

I’ve been looking seriously at metasearch/federated search products for libraries recently. After a lot of reading and a few demos I’ve got some complaints.

I’m surprised how vendors, even now, devote so much time demonstrating patron features that are neither used nor appreciated by any patrons without an MLS. Recent lessons (one, two, three) should have made it clear that libraries need to conform to patron expectations of how online resources should work. Our own search statistics show that only 0.0067% (YES, less than a hundredth of a percent!) of the searches on our OPAC get “limited” to specific languages, locations, dates, or material types. What our patrons expect is that a natural language search will yield relevant results in the first page of hits. “Googlization” isn’t about dumbing things down, it’s about making the technology smarter.

And that’s the problem with these vendor’s metasearch products. They don’t do much to improve the quality of the results retrieved from any database. Shovelware, products that pile up junk in an attempt to generate value based on quantity, is poor solution for libraries or researchers. Still, that’s how these products work, and it’s how they’ll continue to work until libraries and their database providers adopt some of the advances in search technology now used on the web (it’s not just Google, but Yahoo, Teoma, Clusty, and others).

At the same time these metasearch products are doing little to improve the results we get, they’re also making the search process slower. Why do they all make us wait while slowly updating a table that shows only the number of hits retrieved from each database? A9 can teach all these vendors quite a few lessons on that point. A9 reports results in resizable columns, and fills in the details from various databases as they become available. The biggest lesson A9 can teach these vendors, however, is that metasearch should be free. They’re pushing OpenSearch as a public standard based on RSS/XML and already they’ve got access to 236 databases. That’s not bad compared to z39.50 (which we all still respect as the elder parent of current search standards), but remember that the standard was only announced in March 2005.

We need to pressure database vendors to improve their search engines and give better results. Maybe database providers need to rank journal articles by the number of citations they receive? Maybe libraries need to buy Google Search Appliances and do their own indexing of database content. That way, links from university faculty would increase the rank of articles they link to, making search results especially relevant.

Then, we need to ask where our money is going when we buy software like this. We need to demand standards-based products with outstanding ease of use. Go try out A9 and compare it to anything in your library. Yeah, don’t you wish you could offer that to your patrons?

tags: a9, clusty, federated search, google, libraries, library, library systems, metasearch, natural language search, opac, patron, patrons, search technology, teoma, yahoo