XML.com: XML From the Inside Out
oreilly.comSafari Bookshelf.Conferences.


Hacking eBay: Turning Email Alerts into Atom

November 23, 2005

From our geeky perspective, Atom and RSS seem to be sweeping through the internet, changing the way people and systems get notified about events. From a broader perspective, though, they've got a long way to go — we all have plenty of computer-literate friends who've never heard of either.

This means that plenty of opportunities remain to improve systems and applications using RSS or Atom. (Because Atom is the latest and greatest in the history of RSS formats, with endorsements from key representatives of the earlier formats, I'm going to focus on using Atom, but the basic ideas here would work for any flavor of RSS.) I see two basic categories of such opportunities: as demos to show how a given system can benefit from an Atom delivery option, and as personal utilities to make your own life easier. A given application can fall into both categories; I wrote something to convert eBay saved search notifications into an Atom feed to make my own life easier, but if I were an eBay employee I'd be showing it to my boss and saying "Hey! This is easy to implement and a great new way to spread the use of our product!"

An Atom version of an email notification system makes a great demo of Atom's power for several reasons. First, a popular incentive for Atom use is that it reduces our email in a controlled fashion, reducing not only the overall email load but also the chance of spam-checker false positives. Second, the difficult part has probably already been done for you — many online products and systems already have elaborate infrastructures in place to track who wants email notification of what. These systems may let recipients specify what they want to be notified about, the email address to send the notification, the frequency of the emails, the format of the emails, and other details.

For example, when you perform a search such as "Elvis black velvet" on eBay, the results screen includes an Add to Favorite Searches link that lets you name the search and add it to a list accessible from your My eBay screen. In addition to naming the search, you can tell the eBay system to send you an email when relevant new items appear.

black velvet Elvis portrait
Figure 1. Black velvet Elvis portrait

Elvis Black Velvet

My wife and I have a room in the basement with a lot of Elvis stuff. The room has its own bathroom with more Elvis stuff, so we refer to the combination as the "Presley Suite." We're not fanatical Elvis fans — I don't think we have more than a dozen CDs and vinyl Presley albums — but he was responsible for a lot of great rock-and-roll and a lot of silly kitsch that's nearly as entertaining.

Our Presley Suite lacked any painted black velvet Elvis pictures, a serious omission in a collection of Elvis kitsch, so I created an eBay saved search on the phrase "Elvis black velvet." Like a lot of eBay searches, you find some great deals that aren't as great when you consider shipping costs, and you see the same people selling the same things over and over. I eventually bought one of the recurring ones, shown above: a somewhat Native American-looking Elvis and an even less Native American-looking woman. (Elvis fans will recognize it as an allusion to Flaming Star, a 1960 western in which Presley played the son of a Native American woman and a white man.)

Occasionally some real one-of-a-kind things showed up in the search results, and the serious collectors quickly drove the price out of my reach. My favorite was an old black velvet Elvis paint-by-numbers kit that hadn't been used yet — imagine white lines dividing up his black velvet face into tiny numbered sections. That would have been a highlight of our collection, right up there with D.J. Fontana's autograph and the Thai bamboo curtain of Warhol's gunslinger Elvis portrait that we found on London's Portobello Road.

Several libraries and services are available to convert email to RSS or Atom. Evaluating each of these, picking one, installing it, and configuring it on my host provider's system seemed like more trouble than just creating something myself. Doing it myself also lets me customize the input and output as much as I like. For example, the Mailbucket service, which lets you create an email address on their server and then creates an RSS feed of mail sent to that address, won't work for an eBay favorite search because eBay sends the notification emails to the email address they have on file for you, not to any address that you want specify when you create the search. Services like Mailbucket also convert each email into a single RSS or Atom entry, but I wanted to see a single eBay email about six hits for the search on "Elvis black velvet" converted to six Atom entries.

To convert the formats, I knew that tools like Perl and XSLT make it easy to convert between plain text, HTML, and XML, but I needed a way to have the arrival of the mail trigger the conversion. This turned out to be relatively easy once I found out about procmail.


procmail is a venerable Unix utility for automating the processing of mail. (I'm not running my own mail server, but my host provider lets me create procmail configurations.) It first became popular for automated sorting of mail into different folders, but now most popular mail clients include built-in features to do that. procmail then became popular for spam detection, but mail client programs usually do that now as well. As a general-purpose tool for redirecting emails meeting certain criteria to specific locations or, better yet, for routing them to be processed by particular programs, procmail is still a great tool for adapting email workflows to newer technologies such as Atom.

Before procmail can do anything with your mail, you have to route your mail through it. This may mean creating a .forward file; in my case, I had to fill out a mail account configuration screen on my host provider's mail administration page. Mail then gets piped through the procmail program, which checks a .procmailrc file to see if anything special should be done to the email. If not, the email is passed along to your inbox untouched.

For the syntax of this .procmailrc file, a tutorial at the Ohio State University math department provides a good start, and the procmail quickstart at Infinite Ink is far more detailed than its name implies.

A typical .procmailrc file begins with the setting of some environment variables and then has a series of rules that each specify which mail they apply to and what to do to those emails. Most rules start with :0: on its own line. The following rule tells procmail to route mail that has the string sales in its subject line to the sales folder:

* ^Subject:.*sales

A regular expression on the rule's second line describes the mail to look for, usually by specifying a line in the mail header to check, and the third line indicates where to send the email message. Beginning the third line with a pipe symbol tells procmail to send the email message to be used as input to a program, like in this procmail rule:

* ^From:.*savedsearches@ebay.com
| /usr/www/users/bobd/rss/bin/ebaymail2atom.sh

This message tells procmail that when the "From" line includes the string savedsearches@ebay.com, the contents of that message should be piped to the ebaymail2atom.sh script in the specified directory.

Pages: 1, 2

Next Pagearrow