Versioning XML Vocabularies
December 3, 2003
Editor's Note: An update to this article has been posted here on 10/27/04.
XML is designed for the creation of languages based upon self-describing markup. The inevitable evolution of these languages is called versioning. Versioning means adding, deleting, or changing parts of the language. Making versioning work in practice is one of the most difficult problems in computing, with a long history of failed attempts. Arguably one reason why the Web rose dramatically in popularity is because evolution and versioning were built into HTML and HTTP headers, each of which provides explict extensibility points and rules for understanding extensions that enabled their decentralized extension and versioning.
XML Namespaces provide an ideal mechanism for identifying versions of languages, and all XML schema languages -- such as W3C XML Schema -- provide for controlled extensibility.
This article describes techniques to achieve more effective loose coupling between systems by increasing the possibility for backwards- and forwards-compatible changes to occur when related systems evolve. These techniques are designed for compatible changes with or without schema propagation. A number of rules are described for versioning XML vocabularies, making use of XML Namespaces and XML Schema constructs. It includes rules for working with languages that provide an extensible container model, notably SOAP. The collective set of rules is called the "Must Ignore" pattern of extensibility. Strangely, the "Must Ignore" pattern for HTML tags and HTTP headers that significantly helped their adoption has not been widely adopted by XML practitioners. This article aims to rectify that situation for current Schema validation environments. Follow-on material will explore innovative relaxed schema validation environments.
FOLDOC  provides definitions of backwards and forwards compatibility. This article will reprise those definitions and focus on the actions of messages, specifically senders and receivers instead of clients and servers. Backwards compatibility means that a new version of a receiver can be rolled out so it does not break existing senders. This means that a sender can send an old version of a message to a receiver that understands the new version and still have the message successfully processed. Forwards compatibility means that an older version of a receiver can consume newer messages and not break. Of course the older version will not implement any new behavior, but a sender can send a newer version of a message and still have the message successfully processed.
In other words, backwards compatibility means that existing senders can use services that have been updated, and forwards compatibility means that newer senders can continue to use existing services.
Forwards-compatible changes typically involve adding optional element(s) and/or attribute(s). The costs associated with introducing changes that are not backwards- or forwards-compatible are often very high, typically requiring deployed software to be updated to accommodate the newer version. The rules described below optimize for the case of making backwards- and forwards-compatible changes. These article argues that this means not changing the namespace name or changing element names.
Compatibility is defined for the sender and receiver of an individual message. However, most web service specifications provide definitions of inputs and outputs. In these definitions of compatibility, a web service that updates its output schema is considered a newer sender. This simply reverses the sender/receiver terminology of input messages when applying compatibility definitions to output messages. If a receiver updates the schema of the output message, then it is "sending" a newer version of the message, hence it is considered a "sender".
Identifying and Extending Languages
Designing extensibility into languages has typically resulted in systems that are more loosely coupled. Extensibility allows senders to change instances without going through a centralized authority. Thus, the first rule relating to extensibility is
A fundamental requirement for extensibility is to be able to determine the language of elements and attributes. XML Namespaces  provide a mechanism for associating a URI with an XML element or attribute name, thus specifying the language of the name. This also serves to prevent name collisions.
W3C XML Schema 
provides a mechanism called a wildcard,
<xs:any>, for controlling where
elements from certain namespaces are allowed. The wildcard indicates that elements
specified namespaces are allowed in instance documents where the wildcard occurs.
allows for later extension of a schema in a well-defined manner. Receivers of extended
documents can identify and, depending upon the processing model for the extensions,
ignore the extensions they don't understand.
<xs:any> uses the
namespace attribute to control what
namespaces extension elements can come from. The main values for this attribute are
##any, which means one can extend the schema using an element from any
##other, which only allows extension elements from
namespaces other than the current one; and
##targetnamespace, which only allows
extension elements from the current namespace.
processContents attribute to control how a
XML parser validates extended elements. Permissible methods include "lax", allowing
validation to occur, "strict", requiring validation, and "skip", to skip validation.
article uses "lax" validation, as it is the most flexible and is the typical choice
The main goal of the "Must Ignore" pattern of extensibility is to allow backwards-
forwards-compatible changes to documents. At a minimum, this means neither changing
adding namespace names, nor changing element names. Adding element names in the target
namespace can only be done with the
##any namespace or a combination of
##other namespace and the target namespace of the vocabulary.
A number of examples illustrate the application of these rules. Imagine that a Purchase Order is being sent from one machine to another. The processing of the purchase order results in a "shipped" message. But that message could be sent quite some time after the Purchase order is received. It would be undesirable for the sending software to have to wait for a potentially arbitrary amount of time for the response (synchronous messaging). A preferable model is for the receiver to be able to send the "shipped" message under its own control, without the sender waiting. The receiver "calls back" to the initial sender, hence the term "callback". The sender provides an address to the receiver in a Callback address. This indicates the address that the receiver should use for sending any subsequent messages to the sender. In web services, this Callback would typically be sent as a SOAP Header block.
Our preference would be to have an extensible style of
##any. A callback type
that uses this model is:
<s:complexType name="CallbackType"> <s:sequence> <s:element name="callbackLocation" type="s:anyURI" minOccurs="1" maxOccurs="1"/> <s:any processContents="lax" minOccurs="0" maxOccurs="unbounded" namespace="##any"/> </s:sequence> <s:anyAttribute/> </s:complexType>
Example 1 -- A schema using ##any for extensibility
However, the determinism constraint of W3C XML Schema, described in more detail later, prevents this model from working. The problem arises when a subsequent version of the callback adds an optional element. A timeout is a good example of this. The timeout of a callback is an informative piece of information for a receiver. Receivers can continue processing if they don't understand the timeout. The following schema is roughly what is desired using wildcards, but it is illegal because of the determinism constraint:
<s:complexType name="CallbackType"> <s:sequence> <s:element name="callbackLocation" type="s:anyURI" minOccurs="1" maxOccurs="1"/> <s:element name="expires" type="s:dateTime" minOccurs="0" maxOccurs="1"/> <s:any processContents="lax" minOccurs="0" maxOccurs="unbounded" namespace="##any"/> </s:sequence> <s:anyAttribute/> </s:complexType>
Example 2 -- An illegal schema
Since this pattern does not work, we need to create a design pattern that enables roughly the equivalent in order to achieve the original goals. For allowing new extensions in the same namespace, the author must create an extension type that allows extensions in the same namespace. The extension type should be used only for future compatible extensions in the same namespace. We need two more rules to allow proper versioning of XML language definitions. First the rule for namespaces:
The rule for allowing extensibility:
An example of a Callback type that follows these rules:
<s:complexType name="CallbackType"> <s:sequence> <s:element name="callbackLocation" type="s:anyURI" minOccurs="1" maxOccurs="1"/> <s:element name="Extension" type="wscb:ExtensionType" minOccurs="0" maxOccurs="1"/> <s:any processContents="lax" minOccurs="0" maxOccurs="unbounded" namespace="##other"/> </s:sequence> <s:anyAttribute/> </s:complexType> <s:complexType name="ExtensionType"> <s:sequence> <s:any processContents="lax" minOccurs="1" maxOccurs="unbounded" namespace="##targetnamespace"/> </s:sequence> <s:anyAttribute/> </s:complexType>
Example 3 -- A Callback type with extensibility
Because each extension in the targetnamespace is inside an Extension element, each subsequent target namespace extension will increase nesting by another layer. While this layer of nesting per extension is not desirable, it is what can be accomplished today when applying strict W3C XML Schema validation. It seems that having multiple nested elements is worthwhile if multiple compatible revisions can be made to a language. This technique allows validation of extensions in the targetnamespace, while retaining validation of the targetnamespace itself.
In general an extension can be defined by a new specification that makes a normative reference to the earlier specification and then defines the new element. No permission should be needed from the authors of the specification to make such an extension. In fact the major design point of XML namespaces is to allow decentralized extensions. The corollary is that permission is required for extensions in the same namespace. A namespace has an owner; non-owners changing the meaning of something can be harmful.
Senders should ideally be able to extend existing XML documents with new elements without receivers having to change existing implementations. Extensibility is one step toward this goal, but achieving compatibility also requires a processing model for the extensions. The behavior of software when it encounters an extension should be clear. Thus, we introduce the next rule:
The simplest processing model that enables compatible changes is to ignore content that is not understood. This rule is
This rule does not require that the elements be physically removed, only ignored for processing purposes. There is a great deal of historic usage of the Must Ignore rule. HTML 1.1, 2.0, and 3.2 follow the Must Ignore rule; they specify that any unknown start tags or end tags are mapped to nothing during tokenization. HTTP 1.1  specifies that a receiver should ignore any headers it doesn't understand: "Unrecognized header fields SHOULD be ignored by the recipient and MUST be forwarded by transparent proxies." The Must Ignore rule for XML was first introduced in 1997 by the WebDAV working group , standardized in the WebDAV specification RFC 2518  section 14 and later separately published as the Flexible XML Processing Profile .
There are two broad types of vocabularies relating to dealing with extensions. The types are data oriented and presentation (or document) applications. For data oriented applications, such as web services, the rule is
For example, if a message is received with unrecognized elements in a SOAP header block, they must be ignored unless marked as "mustUnderstand" (see Rule 10 below), but it is reasonable to expect that the unrecognized elements might be written to a log file.
Document oriented vocabularies may need a different rule as the application will typically want to present the content of an unknown element. The rule for document-oriented applications is
This retains the element descendants, such as for display purposes.
Instead of ignoring unrecognized components, a language may provide a different model for handling extensions. One model is that a receiver will generate a fault if it finds a component it doesn't understand. An example might be a security specification where a receiver must understand any extension. This suffers from the significant drawback that it does not allow compatible changes to occur in the language; changes can't be ignored. Another model is a fallback model, where alternate elements are provided if the receiver does not understand the extension. XSLT 2.0 provides such a model.
When a new version of a language is required, and it is backwards compatible with the older language, then the author must make a decision about the namespace name for names in the new language. There are two choices: create a new namespace name or reuse the existing namespace name. We argue that reusing is more efficient, and we will explore the problems with option #1 in the "new namespace" section. The reusing namespace rule is
An important conclusion is that a new namespace name is only required when an incompatible change is made.
Non-backwards compatible changes typically occur in two ways: a required information item is added or the semantics of an existing information item are changed.
The reuse namespace names rule requires the previous Must Ignore and Any Namespace rules be followed. If these rules are not followed, then a language designer is precluded from making compatible changes and reusing the namespace name.
We've articulated that reusing namespace names for compatible extensions are good
The counter position is that the namespace owner could use a new namespace for the
compatible changes by providing extensibility points allowing other namespaces --
<xs:any namespace="##other">. This technique suffers from the problem
that an extension in a different namespace means that the combined schema cannot be
validated. Specifically, there is no way to create a new schema that constrains the
wildcard. For example, imagine that ns1 contains foo and bar. It is not possible to
SOAP schema -- an example of a schema with a wildcard -- and require that ns1:foo
must be a child of the header element and ns1:bar must not be a child of the header
using just W3C XML Schema constructs. Indeed, the need for this functionality spawned
of the WSDL functionality. The new namespace name approach results in specifications
namespaces that are inappropriately factored, as related constructs will be in separate
namespaces. Further, the reuse of the same namespace has better tooling support. Many
applications use a single schema to create the equivalent programming constructs.
tools often work best with single namespace support for the "generated" constructs.
reuse of the namespace name allows at least the namespace author to make changes to
namespace and perform validation of the extensions.
Default processing model over-ride
Given adoption of the Must Ignore rule, it is often the case that the creator of an extension wants to require that the receiver understand the extension, overriding the Must Ignore rule.
This rule and the Must Ignore rule work together to provide a stable and flexible
processing model for extensions. Arguably the simplest and most flexible override
mustUnderstand flag that indicates whether the item must be understood.
The SOAP , WSDL , and WS-Policy  attributes and values for
specifying understand are respectively:
wsp:Usage="wsp:Required". SOAP is probably
the most common case of a container that provides a
mustUnderstand model. The
default value is
0, which is effectively the Must Ignore rule.
A mustUnderstand flag allows the sender to insert extensions into the container and use the mustUnderstand attribute to override the must Ignore rule. This allows senders to extend messages without changing the extension element's parent's namespace, retaining backwards compatibility. Obviously the receiver must be extended to handle new extensions, but there is now a loose coupling between the language's processing model and the extension's processing model.
There are other techniques possible, such as providing an element that indicates which extension namespaces must be understood.
In some cases a language does not provide a mustUnderstand mechanism. In the absence of a mustUnderstand model, there is no way to force receivers to reject a message if they don't understand the extension namespace.
XML DTDs and W3C XML Schema have a rule that requires schemas to have deterministic content models. From the XML 1.0 specification,
For example, the content model
((b, c) | (b, d))is non-deterministic, because given an initial
bthe XML processor cannot know which
bin the model is being matched without looking ahead to see which element follows the
The use of
##any means there are some schemas that we might like to express,
but that aren't allowed.
- Wildcards with
minOccursdoes not equal
maxOccurs, are not allowed before an element declaration. An instance of the element would be valid for the
##anyor the element.
##othercould be used.
- The element before a wildcard with ##any must have cardinality of
minOccurs. If these were different, say
maxOccurs="2", then the optional occurrences could match either the element definition or the ##
any. As a result of this rule, the
minOccursmust be greater than zero.
- Derived types that add element definitions after a wildcard with ##
anymust be avoided. A derived type might add an element definition after the wildcard, then an instance of the added element definition could match either the wildcard or the derived element definition.
As shown earlier, a common design pattern is to provide an extensibility point --
element -- allowing any namespace at the end of a type. This is typically done with
Determinism makes this unworkable as a complete solution in many cases. Firstly, the
extensibility point can only occur after required elements in the original schema,
the scope of extensibility in the original schema. Secondly, backwards compatible
require that the added element is optional, which means a
Determinism prevents us from placing a
minOccurs="0" before an extensibility
point of ##any. Thus, when adding an element at an extensibility point, the author
the element optional and lose the extensibility point, or the author can make the
required and lose backwards compatibility.
Why is this hard?
We've shown that using XML and W3C XML Schema to achieve loose coupling via compatible changes that fully utilize yet do not require new schema definitions is hard. Following these extensibility rules leads to W3C XML Schema documents that are more cumbersome and at the same time less expressive than one might like. The structural limitations introduced by W3C XML Schema's handling of extensibility are a consequence of W3C XML Schema's design and are not an inherent limitation of schema-based structures.
With respect to W3C XML Schema, it would useful to be able to add elements into arbitrary
places, such as before other elements, but the determinism constraint constrains this.
less restrictive type of deterministic model could be employed, such as the "greedy"
algorithm defined in the URI specification . This would allow optional
elements before wildcards and removing the need for the Extension type we introduced.
still does not allow wildcards before elements, as the wildcard would match the elements
instead. Further, this still does not allow wildcards and type extension of the type
coexist. A "priority" wildcard model, where an element that could be matched by a
or an element would match with an element if possible would allow wildcards before
element declarations. Additionally, a wildcard that only allowed elements that had
defined -- effectively other namespaces plus anything not defined in the target namespace
is another useful model. These changes would also allow cleaner mixing of inheritance
wildcards. But that still means that the author has to sprinkle wildcards throughout
types. A type-level
any element combined with the aforementioned wildcard
changes is needed. One potential solution is that the sequence declaration could have
attribute specifying that extensions be allowed in any place, then a commensurate
specifying namespaces, elements, and validation rules.
The problem with this last approach is that with a specific schema it is sometimes necessary to apply the same schema in a strict or relaxed fashion in different parts of a system. A long-standing rule for the Internet is the Robustness Principle, articulated in the Internet Protocol , as "In general, an implementation must be conservative in its sending behavior, and liberal in its receiving behavior". In schema validation terms, a sender can apply a schema in a strict way while a receiver can apply a schema in a relaxed way. In this case, the degree of strictness is not an attribute of the schema, but of how it is used. A solution that appears to solve these problems is to define a form of schema validation that permits an open content model that is used when schemas are versioned. We call this model validation "by projection", and it works by ignoring, rather than rejecting, component names that appear in a message that are not explicitly defined by the schema. We plan to explore this relaxed validation model in the future.
A final comment on W3C XML Schema extensibility is that there is still the unmet need to define schemas that validate known extensions while retaining extensibility. An author will want to create a schema based upon an extensible schema but mix in other known schemas in particular wildcards while retaining the wildcard extensibility. We encounter this difficulty in areas like describing SOAP header blocks. The topic of composing schemas from many schemas is difficult yet pressing.
Leaving the topic of wildcard extensibility, the use of type extension over the Web
be more palatable if the instance document could express a base type if the receiver
not understand the extension type, as in
xsi:basetype="". The receiver could
then fallback to using the basetype if it did not understand the base type's extension.
Another area for architectural improvement is that XML -- or even W3C XML Schema --
have provided a
mustUnderstand model. As things stand, each vocabulary that
mustUnderstand model reinvents the mU wheel. XML could have provided
xml:mustUnderstand attribute and model that each language could use. Tim
Berners-Lee articulated the need for this in XML in his design note on mandatory extensions
in Feb 2000, but
neither XML 1.0 nor 1.1 included this model.
Finally, there is ambiguity in compliance testing for W3C XML Schema implementations.
W3C XML Schema test collection  does not test
some of the more common cases that have been precluded here. For example, the wildcard
cover a different style, which is
xs:any inside a complex type. These do not
cover some of the non-deterministic cases, typically achieved by combining
maxOccurs variations with
combining inheritance with
##any. Thus, some implementations do not correctly
test for non-determinism, which may yield non-interoperable documents.
One common concern is about implementation support for these features and combinations. These samples have been tried in many different schema parsers and toolkits, such as XML Beans, SQC, and JAX-RPC. While it's impossible to know whether all implementations support these rules, there seems to be good support for what was tested. The author is certainly interested in hearing about toolkits that don't support these rules.
The W3C TAG decided that the topic of versioning and extensibility is important enough to web architecture to work on a finding  and to include material into the Web Architecture document . While this article provided a starting point for the TAG material, that material will cover a broader scope and progress in a more interactive and iterative fashion than an article can. Readers can follow the TAG material for an ongoing treatment of the area of extensibility and versioning.
This article describes a number of rules for using XML, W3C XML Schema, and XML Namespaces in language construction and extension. The main goal of the set of rules is to allow language designers to make backwards- and forwards-compatible changes to their languages in order to achieve loose coupling between systems.
To a certain degree, the technique described herein is a combination of the
##other designs with well-known rules to produce a
design that achieves the goals of compatible extensibility and versioning with validation
using W3C XML Schema. The namespace name owner can add backwards- and forwards-compatible
changes into the extensibility element while retaining the ability to validate all
components, and other authors can add their changes at the
- Free Online Dictionary of Computing
- Flexible XML Processing Profile
- IETF RFC 791
- IETF RFC 2396
- IETF RFC 2518
- IETF RFC 2616
- SOAP 1.1
- WSDL 1.1
- WS-Policy Framework
- Xfront's Schema Best Practices
- W3C Note, Web Architecture: Extensible Languages
- W3C XML 1.0
- W3C XML Namespaces
- W3C XML Schema Part 1
- W3C XML Schema Working Group's Test collection for Any
- XML.com: W3C XML Schema design Patterns, by Dare Obasanjo
- Tim Berners-Lee's writings on evolution, extensibility and must Understand:
- W3C TAG Finding on extensibility and versioning
- W3C TAG Web Architecture document section on extensibility and versioning
The author thanks the many reviewers that have contributed to the article, particularly David Bau, William Cox, Edd Dumbill, Chris Ferris, Yaron Goland, Hal Lockhart, Mark Nottingham, Jeffrey Schlimmer, Cliff Schmidt, and Norman Walsh. This article borrows, with permission of the authors, examples and some text from WS-Callback .