Designing Extensible, Versionable XML Formats
An XML vocabulary should be designed in such a way that the applications that process it do not break when it is inevitably changed. One of the primary benefits of using XML for building data interchange formats is that the APIs and technologies for processing XML are quite resilient when faced with additions to vocabularies.
If I write an application that loads RSS feeds looking for
item elements, then processes their
title elements using any one of the various technologies and APIs for processing XML -- SAX, the DOM,
or XSLT -- it is quite straightforward to build the
application so that it is unaffected by RSS specification
changes as long as the link and title elements always appear
in a feed.
However, this gives the false impression that there are no versioning issues to consider when designing XML formats, since you can always add elements and attributes to the format without causing harm. Experience has shown that this assumption is false. In fact, many of the same problems that face developers when versioning non-XML data formats affect XML-based formats as well.
This article explores some of the points to consider when versioning XML formats as well as some approaches to designing extensible XML formats in a manner compatible with existing XML technologies.
The Difference Between Versioning and Extensibility
Enabling extension of the format by programmers other than the designers of the format -- thus decentralizing the evolution of the format -- is a desirable feature and one that is often touted as an XML benefit. An example of the enabling nature of extensible XML formats is RSS 2.0 and the various modules that extend the functionality provided in the base specification. Designing a format so that it supports third-party extensions is the extensibility problem.
An XML format should be both backward and forward compatible. It should be backward compatible in that new versions of the format should also be valid instances of older versions of the format and thus not break any consumers of the original format.
It should be forward compatible in that older versions of the format should also be valid instances of newer versions of the format; so that old producers can work with consumers of the new format.
There are a four broad classes of changes that could occur in the process of transitioning from one version of a format to another.
1. New concepts are added (e.g., new elements or attributes added to format or new values for enumerations).
2. Existing concepts are changed (e.g., existing elements and attributes should be interpreted differently, added elements or attributes alter semantics of their parent/owning element).
3. Existing concepts are deprecated (e.g., existing elements and attributes should now issue a warning when consumed by an application).
4. Existing concepts are removed (e.g., existing elements and attributes should no longer work when consumed by an application).
Designing XML formats so that the above changes can occur between versions of a format, yet the format remains backward and forward compatible is, obviously, a big versioning problem.
Versioning differs from extensibility in two broad ways. The first major difference is that versioning mechanisms must support change in a linear fashion, while extensibility mechanisms must support change in a concurrent fashion. A versioning mechanism must offer a way to create a version 1, a backward-compatible version 2, a backward-compatible version 3, and so on.
On the other hand, rather than defining an evolutionary process, an extensibility mechanism provides a way to allow new data to show up side by side (or concurrently) with data for a given format. Secondly, changing an XML format by creating subsequent versions is usually done by the entity that controls the format, while extensions are typically added by third parties. In practice, this tends to mean that versioning is done by the owner of the namespace of the XML format.
An ideal XML vocabulary is both extensible and versionable.
Guidelines for Designing Extensible XML Formats
In the article Versioning XML Vocabularies, David Orchard provided some guidelines for designing extensible XML formats. The following guidelines are slightly modified from some of the guidelines in Orchard's article.
- XML formats should be designed to be extensible.
- Extensions must not use the namespace of the XML format.
- All XML elements in the format should allow any extension attributes, and elements with complex content should allow for extension elements as children.
- Formats that support extensibility must specify a processing model for dealing with extensions.
The guidelines above differ from those in David's article in a few key ways. The major difference is that the guideline about specifying a processing model for extensions is upgraded from a should to a must. The reason I changed this rule is that without explicit and consistent rules for consumers of the format when dealing with extensions, then interoperability across implementations will suffer.
Another difference is that the guideline about allowing extension elements specified that elements with simple content (i.e. text content) shouldn't be allowed to have extension elements. The following discussions explore each of the above guidelines in more detail.
Why XML formats should be designed to be extensible.
The primary benefit of allowing extensibility in a format is that it enables a format to evolve without requiring central control of the format. A secondary benefit is that it allows the format to stay focused and simple by pushing specialized or niche-use cases and complex solutions into optionally supported extensions.
An example of the benefit of designing a format to be extensible is RSS 2.0 and the various RSS modules. The core RSS specification defines how to provide basic information such as the title, description, and publication date of one or more entries in a syndication feed.
Various extensions have been designed that enable much richer functionality, such as providing information about the number of comments posted in response to an entry and mechanisms for retrieving the comments to an entry as a separate RSS feed. These extensions have proliferated without the need to modify the RSS specification yet are not mandatory -- so web sites and news aggregators do not have to bear the cost of implementing complex features if they just need simple content syndication.
Another showcase of the benefit of extensibility in XML formats is
the W3C XML Schema recommendation. The XML Schema recommendation allows one to
place extensions to the specification either as namespace-qualified attributes on elements in the XML schema namespace or as
element children of the
Extensions to XML schema have been used to augment its functionality in a number of ways. One example is the ability to embed Schematron assertions in an XML schema to check constraints that are beyond the capability of W3C XML Schema. Another example, is the XML schema annotations used by Microsoft's SQLXML 3.0, which are used for mapping between XML and relational schemas.
Annotated schemas used by SQLXML 3.0 can still be used for validating XML documents but have the added benefit of also being used for shredding XML data into relational tables and vice-versa. These extensions increase the utility of XML Schema without increasing the complexity for all consumers of XML Schema documents since they are primarily beneficial in specialized scenarios.
It should be noted that extensibility is a double-edged sword. The fact that evolution of an extensible format is decentralized may harm interoperability in certain cases since not all clients will support the same extensions.
Why extensions must not use the namespace of the XML format.
There are two main reasons extensions must use their own namespace name. The first is that it is important that each family of extensions be distinguishable from the core components of the XML format and other extensions. Without providing such identification there is potential for naming conflicts between different extensions or between extensions and future additions to the core specification.
Secondly, there should be a straightforward way to go from identifying an extension to learning more about it. If the namespace name of the extensions is an HTTP URI that points to human- and machine-readable information about the extensions then it allows consumers of the format the chance to learn about the extensions they encounter.
Why all XML elements in the format should allow any extension attributes and extension elements.
Once the decision has been made to make a format extensible the next question is where one should allow extensibility. One could restrict the elements whose content model can be extended or annotated using attributes, but this may end up unnecessarily restricting the usefulness of extensions. Consider the following XML fragment:
<books xmlns='http://www.example.com/books'> <book publisher="Addison Wesley"> <title>Mythical Man Month</title> <author>Frederick Brooks</author> <publication-date>1995-06-30</publication-date> </book> <book publisher="Apress"> <title>Programmer's Introduction to C#</title> <author>Eric Gunnerson</author> <publication-date>2001-06-30</publication-date> </book> </books
In the above document there are multiple areas where one could augment
the information provided. The
could be extended with an
ext:edition attribute that
indicates what edition of the book is being described. An
ext:price element could be added to the content model
book elements describing the price of the book.
author element could be augmented with an
ext:is-editor, which is used on collected works such
as anthologies to indicate that the specified author was actually
an editor. And so on.
The point is that in an XML format it is likely that a lot of the structured data (i.e. elements) in the document can be extended or annotated in a way that adds more value to the data being transmitted. Given this situation, it is likely that designers of XML formats may not be able to anticipate the various ways the data in a format may be augmented or annotated.
Thus, if the author(s) of an XML format restrict which elements can be augmented by extensions there is the possibility that useful extensions may be prohibited by such restrictions. Allowing all elements in the XML format to be extended ensures that no useful extensions are prohibited.
When authoring a schema for an XML format using W3C XML Schema,
xs:anyAttribute to allow extension
attributes to appear on an element and
allow extension elements to appear as children of an element.
Why formats that support extensibility must specify a processing model for dealing with extensions.
Another key decision that must be made once an XML format is deemed extensible is what the processing model should be for handling extensions in consumers of the format. As pointed out by David Orchard in his article, Designing XML Vocabularies, the most popular processing model for dealing with extensions has been the use of Must Ignore rules in combination with mustUnderstand constructs.
With Must Ignore rules in place consumers of the format are expected to ignore extensions they do not understand. In the case of extension elements this could take one of two forms. Presentation formats such as XHTML only ignore the unknown start and end tags for the extension elements but still process their contents. David calls this the Must Ignore Container rule. In most other situations, XML formats apply what David called the Must Ignore All rule where the extension element and its children are ignored by the consumer if they are not understood. This implies that ignoring extensions is not fatal to the application and the data they contain is not significant.
In certain cases, an extension could be introduced to an XML format that contains data that is significant to the application and should not be ignored by consumers of the format. This situation is especially likely in the case of XML formats that mainly act as containers or envelopes for more specialized data such as SOAP.
scenarios are to be supported then designers of the XML
format should consider adding mustUnderstand
constructs to the vocabulary. An example of a
mustUnderstand construct is the
SOAP mustUnderstand attribute. The rules for the
mustUnderstand attribute in SOAP are given below.
mustUnderstand global attribute can be used to indicate
whether a header entry is mandatory or optional for the recipient
to process. The recipient of a header entry is defined by the SOAP
actor attribute (see section 4.2.2). The value of the
mustUnderstand attribute is either "1" or "0".
The absence of the SOAP
mustUnderstand attribute is semantically
equivalent to its presence with the value 0. If a header element
is tagged with a SOAP
mustUnderstand attribute with a value of 1, the recipient of that header entry either MUST obey the
semantics (as conveyed by the fully qualified name of the element)
and process correctly to those semantics, or MUST fail processing
XML formats that intend to allow extensions that need to be understood by consumers should use a
mustUnderstandconstruct such as a namespace attribute, which must appear on the extension element.
Other processing models for extensions are possible. One approach
could be making the
mustUnderstand rule the default for the format, meaning that a consumer must always fail if it
does not understand an extension. Another approach could be
restricting the structure of extensions so that they can be
provided to the end user in a consistent manner. For example, restricting extensions to key-value pairs
in XML configuration files.