Notes and XQueries

October 20, 2004

Hero of the XML world this week, and appropriately named for the role, is Mike Champion. His contributions towards debate on XML standards and their creation have long been among the most enlightening and tolerable. Both of my summaries from the XML-DEV mailing list this week are taken from conversations inspired by Champion.

How Long Must We Wait?

It's been widely rumored, eagerly expected by some, feared by others. It's been a long time in the making, the subject of hushed discussions at chance meetings, and reputedly better than all that's gone before. No, it's not the latest prequel to Star Wars, but XQuery, the W3C's XML Query language.

On the heels of an extended thread that had something to do with Microsoft, and it's hard not to have an extended conversation when that company is mentioned, Mike Champion picked up the essential question arising, Why is XQuery taking so long?

This is a bit of a permathread, but I would like to solicit opinions from current or former XQuery participants or outside observers. The XQuery activity kicked off with a workshop in 1998 and the working group was chartered in 1999. Five years later, no Recommendation in sight. Why did this happen?

A reasonable question to ask, I think, even if the answer isn't quite as vital as it might once have been. XQuery is generally held to be the last great project of XML standardization, and any lessons to be drawn probably won't be applied in the field of XML itself.

Jonathan Robie was quick to remind Champion that it was hardly the case that no Recommendation was in sight, merely that it would be around in "late 2005." Robie added, "I'm not sure that we knew ahead of time what we now know in retrospect."

So, what of the reasons for the delay? Champion himself provided this thoughts to start things off, each of which drew response from others.

XQuery's requirements were far too ambitious and beyond the state of the art. It became an exercise in design by committee rather than standardizing actual experience.

Robie responded that yes, this was true at the beginning. But reality and implementation have been playing catch-up while XQuery waits, so now they are finally engaged upon standardization. Robie also points out that W3C's position lies somewhere between the extremes of design and standardization.

There's another permathread philosophical question again: should we just let individual companies create things, watch them take hold in the marketplace, and then standardize them? The standard W3C answer is that we don't want the big players to dictate terms to everyone else, and that for many crucial technologies, we should work together to create solutions.

Champion's second suggestion as to the delay with XQuery concerns the degree to which it is interlinked with other specifications, some of which themselves undergone long development periods.

XQuery has become rather tightly coupled with several other W3C specs, especially Schema, XPath, and XSLT. As is usually the case, this creates a bit of a hairball -- changing anything requires untangling everything.

While he will find many who agree this situation is regrettable, and many who think it's essential, Champion is unlikely to find any dissenters on the entailed disadvantages. Robie again:

Absolutely. And that has been really difficult, frustrating, and time consuming. But I do think the wider community wants compatibility among standards.

The compatibility claim is prima facie true, of course, but is a convenient blanket for spiraling complexity and any other number of sins that arise from each of the specifications so linked.

Champion's final suggestion is the more conventional one that was largely the reason behind the lumbering development of W3C XML Schema.

There are a lot of conflicting intellectual and corporate agendas interacting, and coming to a mutually acceptable consensus is challenging at best.

In the end, nobody disagreed with such explanations. But, despite these, need XQuery have taken so long? And indeed, Robie asks how long should a standard take to develop?. Len Bullard replied with the pithy advice "Long enough to produce applause. Real results are readily recognized." Indeed they are. The risk though, says Bullard, is that some other act will steal your thunder while you're still getting ready.

But understand the audience and don't play Bach if they came to boogie. Could XQuery have come out in successive versions with simpler accompaniment? ... companies do have to pick the right time to implement and face up to the costs of re-implementing. Customers have to accept that the costs of global markets may be occasional hiccups in supply. No one I know is using the original MS version of XSD.

Robie replied by sharing some of the hindsight he alluded to earlier. If not boogie, XQuery might have been able to cut more of a jig than it has.

We now know that we could have done most of the type work later, releasing a dynamically typed language and adding the static typing in a subsequent release. The awful fact is, we couldn't know that for sure until after the static typing was done.

And I think we could have been done 1 to 2 years earlier if we had learned the lessons we now know about process on a complex spec with lots of players and lots of industry interest. The awful fact is, we had to flounder a while to figure some of this out.

And the sad fact is that there probably won't be another chance, in XML at least, to apply that knowledge.

Toward an XML-I?

A later conversation centered around the fact that the makers of the XML Spy editor had deliberately left part of the W3C XML Schema specification unimplemented because they materially disagreed with it. Unthinkable variance from a standard or rightful difference of opinion with a recommendation?

Leaving the rights and wrongs of that particular issue behind, Mike Champion wondered whether W3C specifications should be treated as edict or not. Some years ago W3C specs, known as Recommendations, were just that. Since that point, though, there's been a drift to see them as standards of similar significance to those issued by ISO. We do not yet sit, however, at a point where a W3C Recommendation truly is a standard.

Champion presented the question of how to proceed, saying there were two paths we could take.

Moving XML recommendations toward true standards status, presumably clarifying and refactoring them to reflect actual experience over the last few years, and developing rigorous conformance tests.

Accepting that they are indeed recommendations from joint design efforts; creative interpretation and ad-hoc profiling are part of the process necessary to build the base of experience upon which real standardization can build, someday.

Liam Quin, W3C XML Activity Lead, responded, pointing out first that adherence to ISO standards wasn't actually quite as wonderful and ideal as Champion intimated.

It's actually not uncommon for people to interpret ISO (and other) specs as they see fit, and try to get away with things. On the other hand, this is why national standards bodies such as the BSI have conformance tests and logos such as the "BSI Kitemark" seal of approval that manufacturers can place on approved devices.

This does not ensure complete conformance, as a device at one end of tolerance might not connect well with one at the other end, given the further tolerances of the manufacturing process. But it's about as good as we (they) can get.

I'd like to see some form of conformance testing for XML software.

I don't think it's something W3C has the resources to do right now, and I'm reluctant to consider making it something expensive, since that could rule out many open source initiatives and also penalize small companies.

Conformance testing has been on and off the agenda ever since we had XML 1.0, and has been a major weapon from time to time in encouraging certain interests to embrace XML standards more thoroughly than they might otherwise have done. Indeed, XML.com itself has been pleased to play a role in such testing. In that context, we can certainly testify that it is both useful and difficult to do.

The W3C is in an impossible position regarding conformance testing, as such tests would potentially give life or death to the products of the very corporations that fund it. From any perspective, the W3C's position would be unsound should it attempt to do the testing itself.

Instead though, its working groups can at least produce conformance tests to help implementers and third parties test implementations. Champion, in reply, questioned the usefulness of conformance testing if specifications were merely recommendations, not standards.

That gets to the nub of my issue: conformance testing for speculative/loosely specified recommendations seems a bit pointless to me. To stick my neck out, I'd argue that it would not be a good investment of time for the XQuery WG to write rigorous conformance tests until it is much more clear which parts of the spec can be effectively implemented in which types of software, which will be widely supported in ways where real people will expect interoperability, which are actually put to use in mission-critical infrastructure and applications, etc.

Given the significant investment from major brains in XQuery, this utterance was certainly inviting rebuke. XQuery implementer Michael Kay said of course XQuery is a small language and 100% implementable. Not so, said Elliotte Rusty Harold, if for no other reason than its dependency on other specifications. Jonathan Robie pointed out that these days you can't become a W3C Recommendation without implementation anyway, and what better way to prove implementation than conformance tests?

I'll leave that one there, and come back to another of Champion's ideas. Given that W3C XML Schema and XQuery might well be hairy and not 100% agreeable, who's to say what can be left out?

As much as I am sure it would pain the W3C :-) maybe what's needed is an XML Interoperability Organization modeled on WS-I to do the profiling and conformance test building.

So now we know, Champion was actually constructing an elaborate joke expressly for the purposes of causing readers to snort coffee up their noses. Or perhaps he's serious? Modeled on WS-I?

The last word on that can go to Paul Sumner Downey, who wrote:

Sadly under its current structure, real practical testing is not something the WS-I is seemingly able to undertake, let alone conformance testing.

Births, Deaths, and Marriages

The latest announcements from the XML-DEV mailing list.

TagSoup 1.0rc1 released: John Cowan's TagSoup is a SAX parser written in Java that accepts ugly, nasty HTML and produces well-formed SAX events or XML output.
XSLT Standard Library v1.2.1: A collection of commonly-used templates written purely in XSLT. This release includes new SVG and comparison modules and new templates in the string, date-time, and math modules.
XML Binary Characterization Properties: W3C's XML Binary Characterization Working Group announces the publication of the first draft of its Properties document, which describes properties that have been identified as potentially desirable for any serialization of an XML data model.
Rx4RDF and Rhizome 0.4.1: Rx4RDF is application stack for building RDF-based applications and web sites. Rhizome is a wiki-like content management and delivery system built on Rx4RDF that brings the wiki metaphor to building dynamic web sites.
<oXygen/> XML Editor version 5.0: Major new release of commercial XML editor adds XSLT 2.0, XPath 2.0 evaluator, an XQuery Editor, WSDL Editor , and SOAP Analyzer, SVG Viewer.
XML 2004 -- Make Hotel Room Reservations Today! and Atom Hackathon to be featured at XML 2004: Latest announcements from the organizers of the main XML event of the year in the U.S.

Scrapings

I've never quite understood the appeal of talking about XML when you could be out dining, but at least at XML 2004 we're being offered the chance to pick the subject ... an invitation somewhat similar in nature was surprisingly not accepted by anybody ... 121 messages to XML-DEV last week, Len rating 15% (talkin' standards) ... please no, not again ... patent redux ... disturbing realization that XQuery has been in development longer than it has taken the present U.S. administration to do all it has in the last four years.