Making Web Services Work at Amazon
December 9, 2003
Jeff Barr, Amazon's web services evangelist, explained to XML 2003 attendees the decisions facing Amazon in opening up their systems for public use via web services. Barr's case study, delivered to a full room, formed part of the product presentations track on the first day of the conference.
Barr set the scene by outlining the various groups that Amazon's customers fall into: buyers, sellers (merchants who sell on Amazon's platform), web site owners (associates), and developers (people who use Amazon's web services.) Amazon's associates scheme has been very successful: founded in 1996, it now has over a million registered associates. This success augured well for the uptake of Amazon web services.
As Amazon's systems developed, they developed in the direction of interoperating feature components inside the firewall; e.g., the catalog, shopping cart, and personalization engine. Through their web services platform, Amazon is beginning to open these features up to public use, and Barr said they have ambitious plans to expose much more functionality.
So how did Amazon arrive at the decision to provide web services? One of the main drivers was that its partners needed better data access -- some major ones had XML data feeds, others simply scraped Amazon's web pages -- so the process of collaboration was both expensive and brittle. A move to defined and reusable web services was thus a logical solution.
Amazon's aims in providing web services were, according to Barr, to support industry standards, provide remote access to data and functionality, decouple their data from its presentation, create a software development platform, unlock creativity from collaborators, and realize more benefit from their internal development investment. Particularly key for Barr, previously employed in Microsoft's developer tools division, is the aspect of creating a development platform, in much the same way as traditional software vendors do.
Considerations for Providing Web Services
Barr then outlined the various issues Amazon considered in the web service deployment, as well as their resolution. The first of these was revenue -- should Amazon pursue web services as an experiment or with the intention of generating revenue? Building on the success of their associates and sellers, Amazon decided to pursue revenue, building on the relationships with these two groups in the first instance. It seems that the choice to do this was key, as it provided a real-world customer base driving their design decisions.
The second issue to consider was that of licensing. In order to create conditions for success, the terms needed to protect the rights of both the developers and of Amazon itself. While providing a degree of openness in order to foster creativity and adoption, the license needed also to sustain Amazon's business model. Practical issues include ensuring data freshness and preventing excessive server load. These needs were met with licensing constraints including one API call per second, a ban on reselling data, storing non-price relevant data for 24 hours maximum and pricing data for 1 hour maximum, and a mandatory link to amazon.com.
The next issue was that of protocols. Should they support SOAP or XML over HTTP (that is, REST)? In the end Amazon provided both and let developers make the choice. Despite it being the "standard", only about 15% of Amazon web services calls are made with SOAP, the remainder with REST.
The fourth issue was to consider how to create a software platform for developers. This was addressed by borrowing best practices from the regular software world: document the APIs, commit to API stability, and provide backward compatibility across API revisions.
The final consideration for Amazon in deploying its web services was developer support -- how to help developers to succeed with the API. Amazon use a combination of a discussion board, weekly developer chats, a regular newsletter, frequent releases of the software development kit, and an online FAQ. Being responsive and open to their developer community has worked well for Amazon.
Barr's talk provided many good pointers for large businesses considering opening themselves to greater programmatic interaction with developers. Amazon's decisions certainly seem to have set them on a course for success. Perhaps the best mark of this success and future promise is that Amazon is increasing the size of its internal team by five times for 2004.