Catching Up with the Atom Publishing Protocol
December 7, 2005
The Atom Syndication Format is now also known as RFC 4287. Atom is an internet standard in the same way that little things like SMTP and HTTP -- also known as "email" and "the Web," respectively -- are internet standards. (If you're just now coming to terms with the Atom Syndication Format, check out Uche Ogbuji's Agile Web XML.com column this week for some practical advice and running code.)
Progress on the Atom Publishing Protocol (APP) has lagged a bit, in part because it relies so heavily on the format specification. Now that the format is done, work has progressed rapidly on the protocol.
Here is the general outline of how the protocol works; in particular, I'll point out design alternatives that the Working Group (WG) has considered and rejected and some of the reasoning behind those decisions. I'll also point out the lingering questions and open problems. Please be advised that this is just one person's viewpoint and that everything and anything can change before the WG puts its stamp of approval on a final version of the APP.
Here is an Atom Entry:
<entry xmlns="http://www.w3.org/2005/Atom"> <title>Atom-Powered Robots Run Amok</title> <link href="http://example.org/2003/12/13/atom03"/> <author> <name>John Doe</name> </author> <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id> <updated>2003-12-13T18:30:02Z</updated> <summary>Some text.</summary> <content type="xhtml"> <div xmlns="http://www.w3.org/1999/xhtml"> <p><i>[Update: The Atom draft is finished.]</i></p> </div> </content> </entry>
There are two things to notice about this Atom Entry. The first is that it is a completely
valid Atom document. Yes, that's right, an Atom Entry is a valid Atom document in
right. The format actually defines two kinds of documents: one with a document element
atom:feed, the other with a document element of
The second thing to notice about this document is that it contains almost all of the information you would need to describe a weblog entry. You've got a title, author, summary, and the content. An Atom Entry is a good representation of a weblog entry, which shouldn't be terribly surprising, but it helps out a great deal when we go to design a RESTful web service for weblog editing. If you go back and read How to Create a REST Protocol, you see that we want to answer the four questions:
- What are the resources?
- What are the representations?
- What are the methods supported?
- What status codes can be returned?
The two major resources in this problem space are members and collections. There is a single member resource for each entry on your weblog. That member resource has an Atom Entry as its representation, and changing that Atom Entry representation is how the corresponding weblog entry is changed. The collection resource is a list of URIs of member resources. Put in our standard tabular form we get:
|Member||GET||Atom Entry||Retrieve the Atom representation of the entry.|
|Member||PUT||Atom Entry||Update the member resource with the Atom entry representation.|
|Member||DELETE||Atom Entry||Delete the member resource.|
|Collection||GET||Atom Feed||Retrieve a list of the members in the collection. May be a subset.|
|Collection||POST||Atom Entry||Create a new member resource with the given Atom Entry.|
Back to the Future
This isn't very different from the protocol described in draft-gregorio-09,
which is what the WG started with almost two years ago. The language of gregorio-09
is a bit
rough, and certainly not proper specification text, but the basic ideas are there.
then called the EntryURI is now called a member resource. In the intervening years
the FeedURI and PostURI have merged into a single collection resource. The basic
editing model hasn't changed: we use the four basic methods of HTTP (
DELETE) to shuffle Atom Entries
One of the big differences from gregorio-09 is the absence of SOAP. A SOAP binding may appear some day, but it was considered too much to have that in the core protocol. There were also some proposals to use WebDAV. While those proposals were not accepted, they did have some influence and the WG adopted the term collection from the WebDAV spec. While the APP is not based on WebDAV, there seems to be general agreement that we should peacefully coexist and that a WebDAV collection can also be an APP collection.
You've Got Issues
Now it may look pretty simple but there are quite a few issues that have to be dealt with.
Who Controls the Elements in the Entry?
When creating or editing an entry using an Atom Entry, there is the basic question
side controls which element in the Atom Entry. For example, the client should probably
control of the
atom:title. On the other hand, two elements cause some concern
when considered in the context of the protocol. The first is
atom:id. As the
atom:idelement conveys a permanent, universally unique identifier for an entry or feed.
Right. So what does that mean when creating an entry? Does that mean the client
has to create a globally unique
atom:id just to
POST a new entry?
Should the server overwrite that
atom:id just to guarantee that the
atom:id is unique? Remember the use case for this restriction: entries that
were duplicated in multiple feeds wouldn't be displayed more than once in your aggregator.
How do I transfer my entries from one blogging system to another and keep the
atom:id the same?
Consider the following scenarios:
We have collection A and collection B. What should happen if I copy an entry "a" from
collection A to collection B? Should that entry be given a new universally unique
atom:id or should it keep the old one? It would be nice if we kept the same
atom:id because that would avoid users seeing duplicate entries in their
If copying "a" from A to B was done because we we're migrating content from one system
another, or adding content into a different category, then keeping the
is the desired behavior.
On the other hand, what if I start editing entry "a" in collection B while keeping
original in collection A unchanged? What if I changed it significantly? Then I would
two entries with the same
atom:id but diverging content. That's not very
consonant with the idea of a "permanent, universally unique identifier for an entry
There are also issues with
atom:updatedelement is a Date construct indicating the most recent instant in time when an entry or feed was modified in a way the publisher considers significant.
Let's repeat that last bit for emphasis -- "was modified in a way the publisher considers significant."
We're designing a protocol here, what the heck are we supposed to do with that? Is the client or server considered "the publisher"? And how exactly do they determine "significant"?
Remember I said at the beginning that collections contain members that have representations
as Atom Entries? I lied. There are actually two types of collections, one that contains
Entries and the other called a media collection that contains any type of media:
images, PDFs, etc. The mechanics of dealing with media collections are the same as
POST a representation to the collection to create a new member
DELETE on the member
resources to edit. The only difference is that the representation of a member doesn't
to be an Atom Entry.
A weblog is more than just a single list of entries. There may be a media collection, or you could have a link blog, a book list, and a music list. How does your blogging client discover all the collections associated with your blog? And what if you have multiple blogs associated with a single account, like on some third-party weblog hosting services?
Here we have the battle of the formats. There have been proposals to use all of the following:
- Custom Format
Actually many different custom XML formats have been proposed. Here is a sample of the format defined in the current draft.
<?xml version="1.0" encoding='utf-8'?> <service xmlns="http://purl.org/atom/app#"> <workspace title="Main Site" > <collection title="My Blog Entries" href="http://example.org/reilly/main" > <member-type>entry</member-type> ... </collection> <collection title="Pictures" href="http://example.org/reilly/pic" > <member-type>media</member-type> ... </collection> </workspace> <workspace title="Side Bar Blog"> ... </workspace> </service>
One of the older microformats, XOXO has been proposed as a format for listing the collections and workspaces. Here is a fragment from the XOXO microformat draft specification:
<ol class='xoxo'> <li>item 1 <dl> <dt>description</dt> <dd>This item represents the main point we're trying to make.</dd> </dl> <ol> <li>subpoint a</li> <li>subpoint b</li> </ol> </li>
You can see how it would be easy to describe groupings of collections and workspaces using such a format.
- Atom Syndication Format
Atom feeds have been proposed in several forms, including one that uses one entry per collection, and others that use link elements to point to collections, workspaces, and other Atom feeds.
In the table above I said that doing a
GET on a collection returns an Atom
feed with what might be a subset of all the entries in the collection. So how do I
listing the rest of the entries? Good question; apparently a painful question too.
have been proposals that have spanned almost every possible combination and permutation.
link/@rel="next". Since we are using an Atom feed to enumerate the entries
in a collection we can use the
link element with a
"next" to point to the next n entries. Do that recursively so
that you end up with a chain of Atom feeds all linked together going back in time.
several advantages; the server can always give you a reasonable amount of data, the
ETags on the first feed in the chain to see if there have been any
updates. Of all the solutions, this is the only one that can be served statically.
this is how enumerating entries worked in
gregorio-09. Currently it appears
that this method of enumerating members will end up in the core.
The disadvantages are that it gives the client very little control and the client
to do many
GETs following the chain of feeds if it wants to enumerate
every entry in a collection.
The remaining proposals used either indexes or dates or some combination. The index approaches, where the collection is treated like a giant array and you pass in values to select a subset, have some drawbacks. The first is that you really need the total size of the collection, but realize that that number may change between the time you read it and the time you actually use it. The second drawback is that since the number of entries returned is determined by the client, it is possible for the client to accidentally, or maliciously, request a huge number of entries. Not really a problem for a weblog which would have its APP service protected by authentication, but think about an APP-enabled wiki which may not be protected by authentication.
URI Templating. URI templating is part of the current draft but I believe it will be departing soon.
This is a system where a string is given with brace-delimited keywords embedded in
braces and keywords get replaced with values and that constructs a URI. For example,
index and it accepted a range of index values separated by a dash,
then this URI template:
could be filled in like so:
to request the first 15 entries in the collection. This is a mechanism I described and provided code for in a previous column, Constructing or Traversing URIs.
A variety of proposals used different sets of keywords that did index- or date-based queries. All of these have the disadvantages I stated previously for index- and data-based queries. In addition this approach has the disadvantage that the client has to get the URI template from somewhere before it starts working with a collection.
SEARCH. Yes, someone actually proposed
extending HTTP with a new method for searching. Again, all the disadvantages of a
date- or index-based approach, with the additional obstacles presented by trying to
Range. HTTP already supports the idea
of a partial
GET where you use a
Range header, but that is only
specified in terms of bytes. There was a proposal for doing this using a new unit
Range that was in
updated. This was actually in the
specification for a while back in draft-ietf-atompub-protocol-03.
Implementing the Atom Publishing Protocol
httplib2: HTTP Persistence and Authentication
Doing HTTP Caching Right: Introducing httplib2
Dreaming of an Atom Store: A Database for the Web
This has all the disadvantages of a date-based query with the added problem of its
In order to find out if a collection supports the new
Range unit you have to do
GET on the collection and look in the headers for an
Accept-Ranges header with the right value.
"Just" Use Query Parameters. There was even a proposal to "just" use query parameters. While simple to implement and specify, this has a pretty major flaw in that it prevents the server from using query parameters for anything else. For example, if you had multiple collections that were all accessed through the same CGI application, and you passed in the collection you wanted via query parameter, that would have been impossible with this approach. The more complex URI templates were invented in part to solve this problem but their complexity was in part responsible for them being rejected.
While the WG has certainly worked with a great many proposals and alternatives I believe the lessons are pretty clear: when in doubt, go for the simplest thing that could possibly work and trust in the utility of hypertext. A simple format sprinkled with links contains a surprising amount of power.