How eBay Uses Metadata to Enhance Its Web Services
With a simple web service like a stock price lookup service, you can pass in a string representing the stock symbol and get back a decimal indicating its current price. Web services like these are easy to document and even with slightly more complex services, like an address validation service where you pass in multiple pieces of data, it is still relatively easy to document what data gets passed in and is returned by the service. However, for companies like eBay that have released sophisticated web services, the task of documenting those services is increasingly complex. This article describes how eBay tackled the problem of documenting its web services using structured metadata. In the end, our problems were solved by using the little-known XML Schema standard appinfo element. We'd like to propose this as a standard for other web services to use. We will open up the spec under a creative commons license and we have a Java tool we have open sourced.
eBay Web Services
The eBay Web Services schema defines the services that are exposed and the data types that are used when interacting with those services. Each service is referred to as a call, and there are calls that support operations such as listing an item, searching the eBay database, viewing My eBay data, and much more. One recent release of the eBay Web Services schema contained the following:
- 95 calls
- 170 simple types
- 432 complex types
- 1,874 elements
- 1,907 enumeration items
Each call, type, element, and enumeration item must be documented to allow developers to code applications that work seamlessly with eBay Web Services.
The initial release of the web services in 2001 provided an XML over HTTPS interface. eBay Web Services grew organically over a few years, with new calls and new functionality added piece-by-piece with each incremental release. During that time the documentation team wrote standard HTML-based overview and reference guides for the Web Services. There was no published schema or DTD for the Web Services, so the only source of information on how to use them was the documentation.
eBay recently made all its Web Services through a SOAP interface. The SOAP web services offer the same functionality as the original XML/HTTPS web services, but they format the data using a new, more robust schema. Early in 2005, eBay applied the new schema to the existing XML over HTTPS interface. Now, no matter how you use eBay Web Services, the data is formatted using the same underlying schema.
The "new schema" web services use the XML Schema standard to specify the data format. The data for each call is specified by a pair of complex types, a request type and a response type. Here is the request type for the
GetUser call, which is used to retrieve information about an eBay user:
<xs:complexType name="GetUserRequestType"> <xs:complexContent> <xs:extension base="ns:AbstractRequestType"> <xs:sequence> <xs:element name="ItemID" type="ns:ItemIDType" minOccurs="0"> </xs:element> <xs:element name="UserID" type="xs:string" minOccurs="0"> </xs:element> </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType>
The schema contains information about how the data should be structured. From this snippet, we can see that the
GetUser call request takes two elements,
ItemID (of type
UserID (a string). Both elements are optional, as you can tell from the minOccurs="0" attribute. The XML Schema standard also allows you to add documentation to the schema through the documentation tag. Here is part of the
GetUserRequestType with documentation added:
<xs:element name="UserID" type="xs:string" minOccurs="0"> <xs:annotation> <xs:documentation> Specifies the user whose data will returned by the call. UserID is optional. If not specified, the user data returned is that for the requesting user. </xs:documentation> </xs:annotation> </xs:element>
With these tags a schema can specify the data format for a web service, as well as provide the additional documentation that an end user needs to understand the data elements.
Deficiencies in XML Schema
One of the main benefits of adopting XML Schema is that it allows the definition of complex types. In eBay Web Services, the "Item" is a piece of data that is accepted or returned by many of the calls, including
GetSearchResults. The old eBay Web Service calls would, at times, format an Item differently. By adopting the XML Schema standard there is now a definition for an Item: a complex type called
ItemType. Here is one of the element definitions in
<xs:element name="Currency" type="ns:CurrencyCodeType" minOccurs="0"> <xs:annotation> <xs:documentation> Currency associated with the item's price information. 3-letter ISO 4217 currency code that corresponds to the site specified in the item-listing request. </xs:documentation> </xs:annotation> </xs:element>
Currency is an element of an Item. It is a required input for
AddItem, it is always returned in
GetItem, and it is sometimes returned in
GetSearchResults. However, if you notice the definition in the schema, the element's minOccurs attribute is set to 0. The reason for this is that since the element is not always included in at least one instance of
ItemType, the element must specify that it is optional.
eBay recognized this problem early on, and the solution that the documentation team tried first was to include additional documentation for each element that indicated when an element was required or optional, and when it was returned by a call. This solution wasn't enough, because end users found the schema too hard to navigate, and the extra documentation could not be read by tools that understand XML Schema.
Based on feedback from the users of eBay Web Services, the documentation team decided to try another approach. Users were saying consistently that they wanted the old HTML-formatted reference documentation. So, the documentation team decided to see what it would take to generate HTML-based documentation from the schema, thus providing the benefit of easy-to-use documentation without the drawback of having to maintain it all by hand.
To do this, the team identified the need to add additional metadata to the schema that specifies the circumstances under which any particular element is required, optional, always returned, or sometimes returned. The XML Schema specification sets aside the appinfo tag for using additional tags that are not explicitly specified in the spec. The tag structure the team came up with is as follows:
<xs:element name="Currency" type="ns:CurrencyCodeType" minOccurs="0"> <xs:annotation> <xs:documentation> Currency associated with the item's price information. 3-letter ISO 4217 currency code that corresponds to the site specified in the item-listing request. </xs:documentation> <xs:appinfo> <CallInfo> <CallName>AddItem</CallName> <RequiredInput>Yes</RequiredInput> </CallInfo> <CallInfo> <CallName>GetItem</CallName> <Returned>Always</Returned> </CallInfo> <CallInfo> <CallName>GetSearchResults</CallName> <CallName>GetMyeBay</CallName> <Returned>Conditionally</Returned> </CallInfo> </xs:appinfo> </xs:annotation> </xs:element>
CallInfo tag is used for each piece of metadata. Each
CallInfo tag contains either a
Returned tag, depending on whether the element is accepted as input or returned as output. The allowed values for
The allowed values for Returned are:
CallInfo tag also contains one or more
CallName tags that specifies what call, or calls, the metadata applies. In the example above, the third set of
CallInfo metadata indicates that the
Currency element within the
ItemType is returned conditionally in the
There are certain special cases that are handled using additional metadata tags, but the basic idea is that by adding metadata to the schema we are able to make up for some of the deficiencies of the XML Schema specification.
One decision I faced was whether to keep the metadata in the schema file or in a new external file. One benefit of having the metadata in the schema is that it is easier for those who maintain the schema to edit the metadata and keep it in sync. We have found this to be an especially important consideration, because eBay releases new versions of its Web Services every two weeks. Because of this, I chose to include the metadata in the schema file, even though this would increase the overall file size.
A Tool to Generate the Documentation
Having great metadata to work with doesn't matter unless you have a tool that understands how to use it. To produce the documentation that we wanted, I wrote a tool in Java that processes the schema and the additional metadata. The tool builds an XML structure for each call request and response that contains every possible element that can be accepted or returned by the call. The body of each element contains a pipe-delimited
String that contains information about the element, such as its type and the documentation that was added in the schema. Following is part of this XML structure for the Item element in the
<Currency> CurrencyCodeType|Required|Currency associated with the item's price information. 3-letter ISO 4217 currency code that corresponds to the site specified in the item-listing request.| </Currency> <Description> string|Required|Description of the item.| </Description> <FinanceOfferID> string|Optional|Specifies a promotional offer that allow the buyer to purchase items on credit. You can call GetFinanceOffers to retrieve the current set of valid offers that can be specified for items.| </FinanceOfferID>
With this XML structure, the tool then creates an HTML representation:
The tool created for this purpose has been released as open source under the MIT license in the eBay Community Codebase. You can download it from the apireferencedocs project page. We are actively seeking out any additional ideas and contributions.