Plugin Development

Will Norris‘ talk at WordCamp PDX introduces WordPress coding standards, common functions, and constants to would be plugin developers (and smacks those who’ve already done it wrong). Also notable: functions, classes, variables, and constants in the WordPress trunk.

Custom Installations

Just as WordPress has a number of hooks and filters that plugins can use to modify and extend behavior, it also has a cool way to customize the installation process.

Extending The WYSIWYG Editor

TinyMCE, the WYSIWYG editor in WordPress has a rich API to allow adding buttons and stuff, but the docs are hard to get into. We can get a jump on that by looking at how it’s implemented in other WP plugins. This code creates the buttons, while the function that responds to the button click and does the work is defined within the plugin. The TinyMCE plugins in core are also informative.

The one way relationship between our user accounts and the MIS database also makes it difficult to engage with new users online. If you can’t get an account until you become a student, how do you allow potential students to apply online if all your systems are integrated with single sign-on? And if you can’t authenticate the online identity of your users, how do you set initial passwords into your system? Or allow them to reset a forgotten password online?

Internet companies never struggled with this issue, as their customers could only approach them online, but most universities built systems around paper applications and have fond (and relatively recent) memories of offering their students their first internet experience. It’s still not unusual for universities to offer their students their campus computing account with a default password based on supposedly secret data shared between the user and the school. But your SSN, birth date, and mother’s name are no longer secret. A proposed change in FERPA policy (see the the top of page 15586 in the NPRM) would have barred the use of “a common form user name (e.g., last name and first name initial) with date of birth or SSN, or a portion of the SSN, as an initial password to be changed upon first use of the system” in systems that store academic data. The final rule excluded that provision, much to the relief of those schools with more lobbying clout than brains.

Platform Choices

Rather than wait to see how the ruling played out last year, we went to work trying to improve security while easing access to our systems (no, that is not self-contradictory). Our challenges were thus:

• Fix initial password assignment
• Fix password resets
• Allow users with a loose or undefined relationship to the institution to create limited accounts for the purpose of interacting with the institution or its members

We considered a number of paths to a solution, including hacking of our university portal (which hosts the CAS single sign-on in our environment), expansion of a limited home-built solution, and a review of commercial and open source products and frameworks. We simplified the problem by confirming that the FERPA rule did not require us to authenticate the “real life” identity of a person; rather, we had only to validate the online identity of a person (saving us from needing to do things like send confirmation PINs by postal mail to a person’s home address).

In the end, we chose WordPress MU. Significant factors were our experience with the software (all the MIS developers use it personally), the extensibility of it as an application platform, the development focus on user experience (especially in recent versions), and our interest in using it as a framework for other user-facing services (especially BuddyPress).

Our Needs vs. WordPress

• The system must serve as the front end to our single sign-on environment, using our AD and LDAP password stores to authenticate users who have accounts in those systems.
• External email addresses, once verified with some challenge/response, can be used to reset a password.
• Users who are presently affiliated with the school have a school-provided email address, but no external address with which to reset their lost passwords.
• Users who are not presently affiliated with the school have no school-provided email address, and must verify their external email address before their account is activated. They can then set their own password once they verify their email address.
• The ability to send password reset codes via SMS would be nice (especially considering the number of long-time employees of the university who do not have personal email accounts), though that also requires the verification of the user’s cell phone number.

After reviewing what we wanted to do, we surveyed WordPress’ code to develop an implementation plan. And, because a number of aspects of our application process were changing, we decided to focus on allowing current users to self-reset their password and postpone development of account self-creation features for new users. Still, a few issues quickly emerged:

• WordPress requires a username be assigned to each user, rather than relying on email address (this is likely to change in WP 2.9) Creating a new username for our users is unacceptable, but adding a large number of new users to our existing username space will quickly deplete the “good” usernames. And changing a user’s username as their affiliation with the institution is unacceptable.
• The core user authentication function can be replaced with our own function. (And in 2.8 it became filterable)
• WordPress MU will validate email addresses, but the system isn’t built to be extensible.
• WordPress only stores one email address per user, but the user meta system can be used to store a second one. Unfortunately (and in a manner inconsistent with post meta), only one value per meta key per user is allowed, making it difficult to allow users to have an arbitrary number of email addresses associated with their account.
• The function that identifies a user by a given email address can be replaced with a function that also checks the secondary address.
• WordPress user profiles have no phone field, but the user meta system can be used to store one. A function to identify a user by a given phone number must also be created.
• Unlike some settings pages, the fields on the user profile editor cannot be changed simply by modifying the $wp_settings_fields array.
• Upon doing a password reset, the user is sent a temporary password, rather than being allowed to set a new password. This contradicts University policy about how passwords are used and communicated and could train users that sending passwords by mail is acceptable.
• The various functions in wp-login.php cannot be replaced, and in WP 2.7 the code had no way to add or replace various login actions (WP 2.8 changed that).
• WPMU-specific functions don’t always follow WP coding standards or models.

(Note that we began our work and deployed the system under WPMU 2.7. WPMU 2.8 included a few changes that made the process easier. I’m proud to say that some of those changes were a result of code we offered back to WP during our development.)

What We Did

• We decided that email addresses (both PSU addresses and external addresses), as well as PSU usernames would be acceptable identifiers for an account, and that a person should be able to log in to our web services using any of those identifiers. So…
• We replaced wp_autenticate() with our own function that accepts either email address or university username, checks to see if the user exists locally, checks to see if they exist in AD or LDAP, confirms their password, provisions their WordPress account (for university users who’ve not logged in via this method yet), establishes a session with our university portal and redirects them there (unless $redirect is set to something more specific that the dashboard).
• We decided to replace WordPress’ usernames with a random string matching a pattern we established. This became the WPID. Doing this required us to hide references to username (easy if you set a preferred display name)
• To store phone numbers and secondary email addresses, and allow users to edit those within their profile, I created the Alternate Contact Info plugin (browse source). This requires more use of output buffering than I’d like, but it gets the job done.
• To confirm email addresses and phone numbers via a challenge/response message (and support other interactions), I created the WordPress Ticket Framework plugin (my introduction, browse source).
• To send messages via SMS, we used my wpSMS plugin (in the plugin directory, browse source).
• Matther Batchelder re-skinned the login screen via a plugin that inserts our custom CSS.
• After determining that our university portal could not be made to authenticate via CAS, I gave up work on my wpCAS Server plugin and developed another method to initiate the portal session (which then establishes a CAS session using the portal’s CAS server).
• We replaced most of the functionality of the wp-login.php page (by hacking core at first, then taking advantage of the action hook in 2.8). In doing so we were able to change the password reset behavior to allow users to immediately change their password after entering their reset code (which was sent to their email address or phone via SMS).

Over time we extended the system to host multiple domains and replace our CMS. Soon we’ll consolidate our  public blogging instance into it, and we’re building an invite system that we can use to invite people to join our community.

What It Looks Like

The re-skinned WordPress login

Entering an email address or username to get a password reset code

SMS text with password reset code

Enter the password reset code from the SMS text message here, or follow the link from the email

Extended contact information in the WordPress profile

And that’s how we replaced our authentication system with WordPress, gained self-service password resets, and built the foundation to invite new users into our system.

Moving data by homing pigeon requires some planning, and pigeons

It was a tech story so apparently humorous that the popular media felt compelled to cover it: carrier pigeons delivered 4GBs of data faster than an ADSL line. The BBC story’s subtitle read “broadband promised to unite the world with super-fast data delivery – but in South Africa it seems the web is still no faster than a humble pigeon,” and that’s how most stories played it. Unfortunately, they all got it wrong.

The race was run by The Unlimited Group, but the clearest telling of it comes from Wikipedia:

Inspired by RFC 2549, on 9 September 2009 the marketing team of The Unlimited, a regional company in South Africa, decided to host a tongue-in-cheek “Pigeon Race” between their pet pigeon “Winston” and local telecom company Telkom SA. The race is to send 4 gigabytes of data from Howick to Hillcrest, approximately 60 km apart. The pigeon carrying a microSD card (an avian variant of a sneakernet), versus a Telkom ADSL line. Winston beat the data transfer over Telkom’s ADSL line, with a total time of two hours, six minutes and 57 seconds from uploading data on the microSD card to completion of download from card. At the time of Winston’s victory, the 4GB ADSL transfer was just under 4% complete.

The critical numbers here are 4GB vs. 164MB (4% of 4GB) in 2 hours. Also, because the challenge was peer to peer transfer, rather than a simple download, the bandwidth is limited by both the upload and download speed of the technology. ADSL, like most consumer broadband technologies, offers slower upload speeds than download speeds.

Speedtest.net's summary of worldwide test results

My home ADSL line is claimed to be 3Mbps/768Kbps, but manages a little less than that in practice according to Speedtest.net (and their sweet iPhone app). That’s a little below average for the US (be sure to view upload speeds), but not dramatically so. And even the fastest region, Asia, can only boast 1.72 Mbps.

In the race between pigeon and internet, the 164MB network transfer over 2 hours is about 187 Kb/s. Significantly slower than my home connection is rated for. But even the fastest consumer upload speeds would have trouble beating the pigeon’s transfer speed. It would take sustained uploads of about 5Gb/s to transfer 4GB of data in the two hours of the pigeon stunt, a rate much faster than possible with the company’s 1Mb/s ADSL connection. But it isn’t just the South African internet that’s slow: The best performing “ISP” in New Hampshire, Dartmouth College, gets a 10 Mb/s rating in Speedtest.net, though very few others can manage over 4Mb/s (Comcast advertises upload speeds up to 10Mb/s, though that’s with “PowerBoost”; sustained speeds are lower, the company reportedly throttles FTP and other uploads, and Speedtest.net rates their business-class uploads at 3.47 Mb/s). If the story makes South African networks laughable, it makes the whole internet a comedy act.

microSD card sizes compared to other SD types

And the pigeon could have carried even more data and made a bigger win over broadband. microSD cards such as the one carried by the pigeon weigh less than half a gram and are now available in capacities up to 16GB (with a theoretical capacity of about 128GB). Assuming you can attached two such cards (one to each leg), you’d get an upload rate of about 40Mb/s in the South African test. Perhaps you could even attach four such cards (totaling under 2 grams of cargo and yielding 80Mb/s) or send more pigeons.

Consumer download speeds are typically much faster, and popular web services go to great lengths to ensure they fill those pipes to speedily deliver rich media. Apple was an early investor in Akamai, one of the first content delivery networks that speed downloads by distributing servers around the world so that your iTunes music and movie downloads are delivered from the closest servers and through the fastest pipes to your location. Netflix’s on demand service leverages the broad availability of these services to deliver TV shows and movies faster than the US Postal Service. But comparing internet speeds to postal service delivery of DVDs (a 14 year old standard) gives no real suggestion of attainable postal service bandwidth (especially when the network-delivered content is far more compressed than on DVD).

You could probably stuff about 30 MicroSD cards (which have almost five times the storage density of regular SD cards — 32GB in 1612.8mm³ vs 16GB in 165mm³) in a First Class letter ($.44 for 6.125 inch x 11.5 inch and 1 ounce) that gets delivered almost anywhere in the US in three days, yielding about 480GB in 72 hours or 15 Mb/s. A pair of 512GB SSDs, on the other hand, can be delivered over night, yielding 1TB in 24 hours or 97 Mb/s. Given that DVDs are 1.2mm thick and 120 mm in diameter, Netflix could easily fit as many as 80 microSD cards — 1280 GB (41 Mb/s over three days) — in their current delivery envelope, much more capacity than even the 50 GB (1.6 MB/s over three days) capacity of the largest Blue-ray discs. It’s for that reason that Amazon offers bulk import/export services of physical media for their web services.

Point: storage capacity has increased dramatically over the past few years, while internet speeds have remained relatively stable. The pigeon gimmick didn’t show us a laughably slow internet connection, it showed us how unimaginably vast our storage has grown (and yet we still find ways to fill it).

Photos: Winston, Speedtest.net, Wikipedia. History worth remembering: avian photographers (via Modern Mechanix).

Ben Fisherman’s JSNES runs entirely in the browser using nothing more intrusive than JavaScript. It apparently manages real-time performance within Chrome, but it works (if not playably) on an iPhone.

I wish the screen was resizable and that it supported iPhone compatible controls, but both of those assume that browser performance will improve enough to make it playable. Interestingly, though not surprisingly, the Safari JS engine is limited to consuming a single CPU (which it quickly does while playing JSNES).

Will Norris has discovered that iTunes 9’s interactions with the Store are more web-happy. I’ve been asking where the iTunes Store API was for some time, now I think I’ve got what I need to build one.

I don’t know if it’s just the Mother’s day effect, but the top 10 online retailers for May 2009 were dominated by flower shops. The top shop is converting almost 40% of their visitors to buyers, though the average is just over 5%. Tim, meanwhile, claims he’s lowered his bounce rate to just 10%.

The AP’s diagram of their Protect, Point, Pay “news DRM” scheme looked like a joke, then I saw the parody.

Despite all the smoke and hype, Ed Felton explains that it’s underwhelming, at most. Still, hNews might be an interesting format for some blogs to adopt. Most of what the AP is rattling their saber about is in the rights (containing ccREL declarations). Felton thinks the dependence on ccREL may extend derivative usage rights, rather than limit them. ccREL, after all, “states unequivocally that it does not limit users’ rights already granted by copyright and can only convey further rights to the user.”

Okay, so hNews mightn’t be so bad, but what’s good about it? It brings together a number of pieces that we all expect in a news story (and many other stories). It makes it easy to identify the dateline and geocoding of a particular story, as well as the publisher and its principles.

Oddly, the format doesn’t appear to address media within the content, but perhaps they expect us to leverage Media RSS and rel=image_src links.

All Facebook is happy to share the ten laws of Facebook advertising, but will those rules lead to better results than the .02% CTR Bob Gilbreath got a year ago?

Howard Weaver wants newspapers to play offense against Google and others, but Chris Tolles, CEO of news aggregator Topix.com says he’s been trying Weaver’s plan for a while, and there’s no bucket of gold to be found in it.

The problem, it would appear, is that newspapers don’t sell news. They sell advertising space and pair it with news as a loss leader to keep the eyeballs. And while that worked in print, it doesn’t work on the web. Web users don’t click on ads on news sites (perhaps for the same reasons they don’t click ads on social network sites).

Yes. The news in a newspaper was a loss leader, though it’ll probably make you feel better to think of the sports and entertainment that way. Television overtook newspapers as the primary news source in the 1970s. And in 2008, newspapers fell behind internet sources. The economics of it now are that news printed on bad paper are less of a loss leader and more simply just a loss. (It’s hard to know if the poor performance of newspaper websites is a result of this or if it reflects other failings.)

Those worried about what will happen to the public sphere after the death of newspapers should take a look at this 2004 study comparing television vs. newspaper front pages: newspapers had twice as many lifestyle and crime stories as the evening TV news. The public sphere has long needed better nourishment.

Mashable’s Josh Catone thinks NPR is the future of mainstream media. Their audience is way up, and it’s a well proven example of the crowdfunding model. My local affiliate provides some of the best local news around.

I just thought to follow up on this 2007 story about DayJet, a high-flying air taxi service that planned to operate tiny, three-passenger Eclipse 500 jets. The story doesn’t deviate from economic trends: DayJet ceased operations in September 2008, and the aircraft manufacturer entered Chapter 7 in February 2009.

The Air Taxi Association says their operators save big money over scheduled airline service, but finding the price of that service can be hard.

Aside from DayJet’s inventory of planes, the company has a lot of transportation research and service algorithms that may stand as assets. The research includes a “Sim City on steroids” that models “the entire U.S. transportation system. They’ve mapped travel patterns into 10-square-mile blocks, complete with income levels, demographics, historical driving patterns, airport drive times, and airline schedules and fares.” Further, they’ve developed an algorithm that supposedly could manage the resource allocation issues and estimate the cost for passengers trying to hail such a taxi.

It can be difficult to get the Zimbra iSync connector, as the company doesn’t offer a simple download from their site. Fortunately, the license allows us to freely redistribute their software.

Download the Zimbra iSync Connector here.

What is the current status of web servers…Is Apache 2.x “fast enough?”
Automattic uses Lightspeed (for PHP), nginx (for static content), and Apache (for media uploads). For WordPress-generated content, all server options are approximately the same speed.

What about APC?
Automattic uses beta versions of APC, and provides a 3-5x performance increase. It’s tied closely to the PHP version, so Automattic recently switched from PHP 4 to PHP 5.

Databases?
MySQL scales well and is easy enough to use that there’s little reason to consider other DBs for WordPress content. Other applications may have different needs. Note: FriendFeed uses MySQL to store schema-less data. Single-table key lookups in MySQL are faster than getting the data from Memcached.

Caching?
Automattic uses Batcache for full-page caching (.002 to .003 second), Memcached persistent object cache, very limited MySQL query cache (never larger than 256MB), sufficiently large key buffer.

HyperDB?
HyperDB solves DB scaling problems.

Backups
User-data backed up every hour, if something changed. Every blog backed up every 12 hours. Dedicated MySQL slaves do LVM snapshots for backups.

Why BuddyPress? “Build passionate users around a specific niche.”

Do you have to become a social network? “No, look at GigaOM Pro,” a recently launched subscription research site based on BuddyPress.

But, yo do get “BYOTOS: bring your own terms of service.” That is, you get to control content and interactions. And your service won’t be subject to the whims of a larger network like FaceBook (or vagaries of their service — think Ma.gnolia)

It’s pretty easy, Andy says, to create a custom BuddyPress component, and there are already a number at the BuddyPressDEV Community.

jQuery 1.3.2 is in WordPress 2.8, but the most exciting changes are in the automatic concatenation and compression of scripts via the script loader.

Andrew Ozz says “This feature can easily be extended to include scripts added by plugins and to use server side caching, however that would require some changes to the server settings (.htaccess on Apache).”

I have yet to figure out how to extend that feature to scripts in my plugins, but I’m working on it.

90% of WordPress blogs he sees are spam. But for those who aren’t spammers and want to do better in Google….

“WordPress automatically solves a ton of SEO issues…WordPress takes care of 80-90% of SEO.”

Still, he recommends a few extra plugins:

• Akismet — reduce spam comments
• Cookies for Comments — reduce spam comments
• FeedBurner FeedSmith
• WP Super Cache — improve performance

“We crawl roughly in order of PageRank…higher ranked sites get crawled faster and deeper.”

“What is PageRank? The number and importance of links pointing to you.” But “avoid BO (backlink obsession). You want to be relevant and reputable.”

Relevant is what you say on your page/site.

Reputable is what others say (link) about you.

Be relevant: Blog about what you love. Blog about what you’re really good at doing (or, I suppose, what you want to be really good at). Blog in your own voice. Write often, write every day.

Think about the keywords that users will type. Include them naturally in your posts

Avoid jargon mismatch. Be sure to include language that non-expert users may use to find information. Include relevant information for beginners on the front page. Try Google Keyword Tool to understand what people are searching for.

Recommends /%postname%/ permalinks. And use slightly different terms in the permalink and title. Other URL tips:

• Use categories that are also good keywords
• keywords in URL paths
  • dashes best
  • next best is underscores
  • no spaces is worst
• Should I change old URLs? No.

Ferris’s law: don’t do it if it’s not fun.

Gaining Reputation?

• Be interesting
• Update often
• Apply Katamari Philosophy — start small, build up, don’t over reach. Start in a niche, then “ambigining” that niche.

Build an audience:

• Provide a useful service
• Do original research or reporting
• Give great information
• Creative niche
• Write some code
• Live blogging
• Make lists
• Create controversy
• Meet folks on Twitter, Facebook, etc
• Make a video

Consider using:

• Google website optimizer (a/b testing)
• <!– google_ad_section_start –> and <!– google_ad_section_end –>

In your content:

• Identify and leverage “evergreen” content
• Show related content
• Avoid shortcuts and scams
• Avoid paid posts

Keep your WordPress updated! Don’t let spammers hack your site.

LifeHacker: productivity porn, read about it more than you do it

Those unwilling to open of their iPhone to adjust the camera focus might take a look at Griffin’s Clarifi, a case with a built-in close-up lens that can slide in our out of place as needed.

Flickr user Meine Ideenecke, meanwhile, has figured out the iPhone camera specifications. He says it’s about 37MM (35MM equivalent), though this source says it’s 27MM.

What happens when one of Google’s street view camera vehicles encounters a low bridge or a muddy Australian road?

The above are the result of PanoLab, Hugin, Calico, and a single shot with a very wide angle lens (Canon’s 10-22mm, effectively 16mm on my Rebel XTi). The first three originated on my iPhone and the PanoLab shot was stitched and originally uploaded to Flickr on my iPhone (though I have since done some color enhancement and reuploaded the photo from my MacBook Pro). Hugin is GPL, the other solutions are less free (in both senses).

MySQL answer: utf8_unicode_ci vs. utf8_general_ci.

Collation controls sorting behavior. Unicode rationalizes the character set, but doesn’t, on it’s own, rationalize sorting behavior for all the various languages it supports. utf8_general_ci (ci = case insensitive) is apparently a bit faster, but sloppier, and only appropriate for English language data sets.

Doesn’t everybody need a PocketTorch? It’s a “safe, practical tool,” they say. More amusingly, the list of suggested uses includes: melting your cache of gold, scaring grandma, lighting illegal fireworks, dental/lab work, and making friends jealous.

Who doesn’t want to be an anarchist librarian? Or a bibliophian?

Compare the retouching portfolio here against the regular posts at Photoshop Disasters.

Jessamyn pointed out the dust up over the dissapearing of PaperOfRecord.com, a historical newspaper archive.

One of the many odd questions this prank job application asks is: What Are You Willing To Wear At Work? (Check All That Apply)

• Paper Hat
• Tie
• Hairnet
• Spandex
• Singing Omelet Costume

This plugin is the next step after my proposal for a common invite API. Here’s how I described it when requesting hosting at the plugin directory:

A common framework for registering tickets that will be acted upon later. Use it to manage challenge/response interactions to confirm email addresses, phone numbers, IM screen names, Twitter accounts, etc. Build an invite system around it, or use it as the foundation of a short URL system. It’s an extensible framework that takes cues from WordPress’ cron and admin Ajax functions.

Tickets are unique 1-32 character strings associated with actions and some stored data. Upon receiving a ticket, the matching action is executed with the stored data as an argument. After receiving the action, a plugin can destroy the ticket (as for challenge/response actions), or or leave the ticket in place for repeated use (like redirecting to longer post permalinks for a short URL resolver).

Here’s how it works:

Registering a ticket requires three things:

• The unique string that identifies the ticket
• A string representing the action that will be executed when the ticket is called
• Some data that will be passed to that action when the ticket is called

And here it is in action:

$ticket = $wptix->register_ticket( 'action_name_string', $wptix->generate_md5(), array( 'data' => 'val', 'more_data' => 'another_val' ));

To actually do anything with that ticket you’ll have to also register a function associated with the action name you gave when you registered it:

add_action( 'action_name_string', 'my_function_name' );

WP Ticket Framework registers a URL base in the form of http://site.net/do/any_ticket_string, so you can call a ticket simply by visiting that URL, or you can call tickets within your code. The following demonstrates one method of handling ticket inputs as part of a larger form submission.

if( ( $ticket = $wptix->is_ticket( $_POST['phone_confirmation'] )) && $ticket->arg['user_id'] == $current_user->ID ){
 $wptix->do_ticket( $_POST['phone_confirmation'] );
}else{
 $errors->add( 'user_phone', __( "<strong>ERROR</strong>: The confirmation code is invalid." ), array( 'form-field' => 'phone_confirmation' ) );
 return;
}

Caveats:

• Tickets don’t have any built in expiration mechanism. You can, however, include an expiration time in data saved in the ticket and act on that accordingly.
• Tickets don’t have any built in security mechanism. If only a certain user should be allowed to call a ticket, your code needs to be aware of that and enforce it.
• Tickets are deleted after being called, though you can suppress that by calling $wptix->clean_up_after( FALSE );.
• Tickets can only be queried by ticket string. If your application needs to keep track of open tickets for each user (as an example) it should also make a user meta entry about it.