XML.com: XML From the Inside Out
oreilly.comSafari Bookshelf.Conferences.

advertisement

How eBay Uses Metadata to Enhance Its Web Services

September 28, 2005

With a simple web service like a stock price lookup service, you can pass in a string representing the stock symbol and get back a decimal indicating its current price. Web services like these are easy to document and even with slightly more complex services, like an address validation service where you pass in multiple pieces of data, it is still relatively easy to document what data gets passed in and is returned by the service. However, for companies like eBay that have released sophisticated web services, the task of documenting those services is increasingly complex. This article describes how eBay tackled the problem of documenting its web services using structured metadata. In the end, our problems were solved by using the little-known XML Schema standard appinfo element. We'd like to propose this as a standard for other web services to use. We will open up the spec under a creative commons license and we have a Java tool we have open sourced.

eBay Web Services

The eBay Web Services schema defines the services that are exposed and the data types that are used when interacting with those services. Each service is referred to as a call, and there are calls that support operations such as listing an item, searching the eBay database, viewing My eBay data, and much more. One recent release of the eBay Web Services schema contained the following:

  • 95 calls
  • 170 simple types
  • 432 complex types
  • 1,874 elements
  • 1,907 enumeration items

Each call, type, element, and enumeration item must be documented to allow developers to code applications that work seamlessly with eBay Web Services.

The initial release of the web services in 2001 provided an XML over HTTPS interface. eBay Web Services grew organically over a few years, with new calls and new functionality added piece-by-piece with each incremental release. During that time the documentation team wrote standard HTML-based overview and reference guides for the Web Services. There was no published schema or DTD for the Web Services, so the only source of information on how to use them was the documentation.

eBay recently made all its Web Services through a SOAP interface. The SOAP web services offer the same functionality as the original XML/HTTPS web services, but they format the data using a new, more robust schema. Early in 2005, eBay applied the new schema to the existing XML over HTTPS interface. Now, no matter how you use eBay Web Services, the data is formatted using the same underlying schema.

XML Schema

The "new schema" web services use the XML Schema standard to specify the data format. The data for each call is specified by a pair of complex types, a request type and a response type. Here is the request type for the GetUser call, which is used to retrieve information about an eBay user:

<xs:complexType name="GetUserRequestType">
  <xs:complexContent>
    <xs:extension base="ns:AbstractRequestType">
      <xs:sequence>
        <xs:element name="ItemID" type="ns:ItemIDType" minOccurs="0">
        </xs:element>
        <xs:element name="UserID" type="xs:string" minOccurs="0">
        </xs:element>
      </xs:sequence>
    </xs:extension>
  </xs:complexContent>
</xs:complexType>

The schema contains information about how the data should be structured. From this snippet, we can see that the GetUser call request takes two elements, ItemID (of type ItemIDType), and UserID (a string). Both elements are optional, as you can tell from the minOccurs="0" attribute. The XML Schema standard also allows you to add documentation to the schema through the documentation tag. Here is part of the GetUserRequestType with documentation added:

<xs:element name="UserID" type="xs:string" minOccurs="0">
  <xs:annotation>
    <xs:documentation>
      Specifies the user whose data will returned by the call. UserID is
      optional. If not specified, the user data returned is that for
      the requesting user.
    </xs:documentation>
  </xs:annotation>
</xs:element>

With these tags a schema can specify the data format for a web service, as well as provide the additional documentation that an end user needs to understand the data elements.

Deficiencies in XML Schema

One of the main benefits of adopting XML Schema is that it allows the definition of complex types. In eBay Web Services, the "Item" is a piece of data that is accepted or returned by many of the calls, including AddItem, GetItem, and GetSearchResults. The old eBay Web Service calls would, at times, format an Item differently. By adopting the XML Schema standard there is now a definition for an Item: a complex type called ItemType. Here is one of the element definitions in ItemType:

<xs:element name="Currency" type="ns:CurrencyCodeType" minOccurs="0">
  <xs:annotation>
    <xs:documentation>
      Currency associated with the item's price information. 3-letter ISO 4217
      currency code that corresponds to the site specified in the item-listing
      request.
    </xs:documentation>
  </xs:annotation>
</xs:element>

Currency is an element of an Item. It is a required input for AddItem, it is always returned in GetItem, and it is sometimes returned in GetSearchResults. However, if you notice the definition in the schema, the element's minOccurs attribute is set to 0. The reason for this is that since the element is not always included in at least one instance of ItemType, the element must specify that it is optional.

eBay recognized this problem early on, and the solution that the documentation team tried first was to include additional documentation for each element that indicated when an element was required or optional, and when it was returned by a call. This solution wasn't enough, because end users found the schema too hard to navigate, and the extra documentation could not be read by tools that understand XML Schema.

The Solution

Based on feedback from the users of eBay Web Services, the documentation team decided to try another approach. Users were saying consistently that they wanted the old HTML-formatted reference documentation. So, the documentation team decided to see what it would take to generate HTML-based documentation from the schema, thus providing the benefit of easy-to-use documentation without the drawback of having to maintain it all by hand.

To do this, the team identified the need to add additional metadata to the schema that specifies the circumstances under which any particular element is required, optional, always returned, or sometimes returned. The XML Schema specification sets aside the appinfo tag for using additional tags that are not explicitly specified in the spec. The tag structure the team came up with is as follows:

<xs:element name="Currency" type="ns:CurrencyCodeType" minOccurs="0">
  <xs:annotation>
    <xs:documentation>
      Currency associated with the item's price information. 3-letter ISO 4217
      currency code that corresponds to the site specified in the item-listing
      request.
    </xs:documentation>
    <xs:appinfo>
      <CallInfo>
        <CallName>AddItem</CallName>
        <RequiredInput>Yes</RequiredInput>
      </CallInfo>
      <CallInfo>
        <CallName>GetItem</CallName>
        <Returned>Always</Returned>
      </CallInfo>
      <CallInfo>
        <CallName>GetSearchResults</CallName>
        <CallName>GetMyeBay</CallName>
       <Returned>Conditionally</Returned>
      </CallInfo>
    </xs:appinfo>
  </xs:annotation>
</xs:element>

The CallInfo tag is used for each piece of metadata. Each CallInfo tag contains either a RequiredInput or Returned tag, depending on whether the element is accepted as input or returned as output. The allowed values for RequiredInput are:

  • Yes
  • No
  • Conditionally

The allowed values for Returned are:

  • Always
  • Conditionally

Each CallInfo tag also contains one or more CallName tags that specifies what call, or calls, the metadata applies. In the example above, the third set of CallInfo metadata indicates that the Currency element within the ItemType is returned conditionally in the GetSearchResults and GetMyeBay calls.

There are certain special cases that are handled using additional metadata tags, but the basic idea is that by adding metadata to the schema we are able to make up for some of the deficiencies of the XML Schema specification.

One decision I faced was whether to keep the metadata in the schema file or in a new external file. One benefit of having the metadata in the schema is that it is easier for those who maintain the schema to edit the metadata and keep it in sync. We have found this to be an especially important consideration, because eBay releases new versions of its Web Services every two weeks. Because of this, I chose to include the metadata in the schema file, even though this would increase the overall file size.

A Tool to Generate the Documentation

Having great metadata to work with doesn't matter unless you have a tool that understands how to use it. To produce the documentation that we wanted, I wrote a tool in Java that processes the schema and the additional metadata. The tool builds an XML structure for each call request and response that contains every possible element that can be accepted or returned by the call. The body of each element contains a pipe-delimited String that contains information about the element, such as its type and the documentation that was added in the schema. Following is part of this XML structure for the Item element in the AddItem call:

<Currency>
  CurrencyCodeType|Required|Currency associated with the item's price information.
  3-letter ISO 4217 currency code that corresponds to the site specified in the
  item-listing request.|
</Currency>
<Description>
  string|Required|Description of the item.|
</Description>
<FinanceOfferID>
  string|Optional|Specifies a promotional offer that allow the buyer to purchase
  items on credit. You can call GetFinanceOffers to retrieve the current set of
  valid offers that can be specified for items.|
</FinanceOfferID>

With this XML structure, the tool then creates an HTML representation:

Figure 1
Figure 1. Click image for full-size screen shot.

The tool created for this purpose has been released as open source under the MIT license in the eBay Community Codebase. You can download it from the apireferencedocs project page. We are actively seeking out any additional ideas and contributions.