Data Encoding or Data 'n Coding?

November 21, 2001

Martin Gudgin and Timothy Ewald

One key to designing a web service is determining what information a client and server will pass back and forth on the wire. This process has two stages. First, you need to develop an abstract notion of what data needs to be exchanged, e.g., an Invoice, an Itinerary, etc. Then you need to design a concrete XML representation of that abstract data; should a customer's name be an element or an attribute. There are two very different ways to accomplish the second step. One is to focus on the structure of the XML itself and to write a schema definition by hand. The other is to infer the structure of the XML from an existing programmatic data type, like a structure or a class.

A Tale of Two Type Systems

The advantage of the XML schema approach is that it gives total control over how transmitted data will be represented on the wire. The programmatic data type approach eschews this control in order to hide the details of XML from programmers who want to work with the data types they are already familiar with. In short, the XML schema approach sets the XML type system above all, while the programmatic data type approach sets more traditional type systems on a pedestal.

The essential difference between these two approaches is their starting point. If there were an isomorphic mapping between XML types and programmatic data types, this wouldn't be a problem because anything that could be expressed in one type system could also be expressed in the other. Unfortunately, no such mapping exists. While there are some areas of overlap between XML and programmatic types, there are also areas of disjunction. It is possible to define XML types using XSD that can be, depending on the expressiveness of a given type system, extremely difficult to map to programmatic types (e.g., types derived by restriction). Likewise, it is possible to define programmatic data types, especially those that involve levels of indirection, that are difficult to map directly to XML types (e.g., cyclic object graphs).

When SOAP 0.9 was being designed there was no XML type system; the XML Schema specification was still an early working draft. Given that the "O" in "SOAP" then stood for "Object", it seemed reasonable to start with programmatic data types and map to XML from there. Thus, the SOAP Encoding rules -- often referred to as SOAP Section 5 -- were born. These rules relegated attributes to the role of pure metadata and mandated that all "real" data be transmitted as elements. Rules for naming elements were included along with mechanisms for encoding data structures such as linked-lists, circular queues, and arrays.

The major advantage of SOAP Section 5 is that the encoding rules make it relatively easy to build a standard marshaller that uses reflection to serialize and deserialize object instances. The major disadvantage is that there is not just one, but many programmatic type systems to map to and from. The tale of two type systems is really the tale of n type systems. As a result, many inferred XML data representations favor one language at the expense of others.

For instance, consider the XML elements that encode the fields of a structured type like an object. Should they be ordered or unordered? Ordered elements are easy to serialize and deserialize if you can use reflection to walk through an object's fields in the order in which they were defined. But in Perl, where structured data is stored in hash tables, maintaining order is quite difficult. So which approach should be taken? Either way, someone's marshaling code just got a lot more complex.

The times they are a-changin'

Since SOAP's debut the world of XML has moved on. We now have a native XML type system in the XML Schema Recommendation. We also know a lot more about XML in general than we did in mid-1999. Some of the framers of the original SOAP specification have been heard to say of SOAP Section 5, "If I were you, I wouldn't start from there". The reason is simple. XML Schema is portable and interoperable and is integrated with other core XML technologies. The results of applying SOAP Section 5 to infer an XML representation for a programmatic type are difficult or impossible to describe accurately in XML Schema. Whether or not these representations are interoperable with each other depends on how a toolkit chooses to map from a given programming language's type system to XML. They won't be fully interoperable with XSD.

More from Rich Salz

SOA Made Real

SOA Made Simple

The xml:id Conundrum

Freeze the Core

WSDL 2: Just Say No

It comes down to this -- trying to start from n programmatic type systems to apply a set of rules to define XML representations for data which every toolkit can understand is not a good idea. SOAP Section 5 was a necessary stop-gap measure when the schema language was in development. But now that XSD is a ratified Recommendation, the SOAP Encoding rules are more trouble than they are worth. Web service implementors should favor XSD, gaining instant interoperability, instead.

"But what about all the types I'm used to using in my programming language?", you may wonder. Admittedly, XML Schema is far from perfect. Some features are missing, notably support for typed references and arrays. In most programming languages such types are synthetic, that is, they are created as needed by the interpreter or compiler without the need for an explicit type definition. Such a notion is interesting but ultimately not needed in the XML type system. The schema language can be used to define a set of types representing references and arrays. They will not be synthetic, that is, they have to be explicitly defined, but this can actually be an advantage: declaring them explicitly allows the designer to retain some level of control over their XML representation.

Using XML Schema to describe SOAP message formats is as important to the long-term success of web services as the use of XML itself. If some web service clients and servers map the XML messages they send and receive back to programmatic data types in their native languages, that's fine, as long as the definition of the mapping is derived from XSD and interoperability is guaranteed. If other web service clients and servers decide to work with the XML messages they send and receive directly, that's fine too. The shift to an XSD-centric approach is an easy one. All that's required is a departure from the current programming language-centric view of web services that many developers hold.