Web Services: It's So Crazy, It Just Might Not Work

October 3, 2001

Clay Shirky

That high-pitched sound you hear is the Web Services hype machine revving up, as words like "revolution' and "paradigm" begin making their regularly scheduled appearance in the press and white papers, where we are promised a Shiny New World of on-the-fly software creation.

The hype is happening just as practical applications for XML-structured data beginning to appear. Web Services can reduce the effort and quicken the process of creating standards between developers or businesses which want to work together, an important if somewhat modest improvement in the Internet's plumbing.

Unfortunately, though, Web Services are being sold not only as improved plumbing but also as a way to create fantastic new software, seamlessly and automatically connecting any two business processes or applications anywhere on the network as if by magic.

A bit too much as-if-by-magic, in fact, because against the backdrop of inflated claims for automatic interoperability is the usual set of mundane but intractable problems that plague any grand vision. While these problems are complex and various, their essence can be summed up fairly simply:

Two old men were walking down the street one day, when the first remarked to the second, "Windy, ain't it?"

"No," the second man replied, "It's Thursday."

"Come to think of it," the first man replied, "I am too. Lets get a coke."

If two parties are out of sync, communication can seem to be happening, without much really getting communicated. To see in more detail why the idea of automatic interoperability is (much) harder than it first appears, it helps to look at the examples of public Web Services being offered as proof of concept.

Hello, WRLD

Automated stock price retrieval is the the standard Web Service -- it is simple, obvious, and valuable, and almost every description of Web Services uses it as an example. It is not, however, representative of any real-world challenges.

Stock prices exist in an extremely constrained world: there is a canonical list of company-to-ticker-symbol mappings, the types of data that can be derived from these ticker symbols are both narrow and well understood, and it is illegal to tamper with the data.

The stock quote example, in other words, illustrates Web Service technology without solving a single problem of shared context, because all the coordination has been established in advance. The connections between company names, ticker symbols, and stock prices were worked out not just prior to Web Services, but prior to the Web, the Internet, and even computers.

Financial data is a lousy proof of concept; networking examples using financial data are almost always successful, even on networks with otherwise poor technological characteristics, e.g. WAP.

Without the shill of financial data, the Web Service examples start to look a little thin. A scan of XMethods.com, which lists existing public Web Services, reveals that most of the entries either rely on services where the terms are already well understood by all parties -- lease calculators, flight information, converting Fahrenheit to Celsius or inches to millimeters -- or places where the data is already publicly available on the Web but the interface can be automated -- address lookups, eBay auctions, Amazon book ranks.

If these are proof-of-concept implementations, what concepts are they proving? Take the inches-to-millimeters example; who would willingly incur the overhead and latency of a remote HTTP call when they could simply multiply inches by 25.4 to get millimeters? These sample services illustrate that it's difficult to create a web service that doesn't rely on both the client and the server understanding the terms of the transaction in advance.

No Bootstrapping Coordination

The problem these examples sidestep is shared context. How can you create interoperability among two parties who've never interacted with one another before? Without defining in advance what language they will use to communicate?

And the answer of course is "you can't," unless you're willing to limit yourself to applications like inch-to-millimeter conversions, where the units involved are so trivial and so universally defined as to present no real problems of interoperability in the first place. Any coordination problem worth solving does not have a solution that can be completely specified in advance.

The vision of Web Services as a framework for automated interactions suffers from these coordination issues at several different points.

• XML only eases interoperability, it doesn't create it
• The current Web Services stack simply moves the problem to higher and higher layers
• As long as there are humans involved, there will be context errors, both intentional and unintentional

Parseable != Interoperable

XML has been oversold as a kind of semantic pixie dust for creating interoperability. Sun says, in its ONE (Open Net Environment) white paper, that the

industry is clearly focusing in on the eXtensible Markup Language (XML) as the lingua franca to enable Web services...XML-based Web services communicate over standard Web protocols using XML interfaces and XML messages, which any application can interpret. Hence, all XML-based Web services can interoperate -- at least in theory.

Even with the "in theory" caveat, this goes too far. XML is not only not a lingua franca, it isn't even a lingua.

As an analogy, take English and Danish. They have almost identical alphabets but are nevertheless different languages. An alphabet is a limited set of characters that can represent an unlimited number of words through recombination. XML is an alphabet, not a language. It provides the primitives for describing larger concepts, and it works by allowing an unlimited number of semantic concepts to be encoded using those primitives. Any XML parser should be able to declare any given XML document structurally valid -- analogously to the way native speakers can tell if a word is or isn't part of their native tongue -- but that says nothing about whether the contents of that document will be comprehensible to the recipient.

At best, XML makes it possible for businesses or developer groups to share data, provided they agree on the semantics of that data in advance. This is not to say XML is not an enormous advance. It plainly is. However, its advance lies in aiding data interoperability where shared semantics can be assumed. It does nothing at all to create semantic interoperability.

WSDL and UDDI: It's Turtles All the Way Up

To answer XML's inability to produce shared semantics between unknown parties, the Web Services world has begun to define a stack built on top of XML, in an attempt to bootstrap coordination. The first layer to take on the problem of automated interoperability is the Web Services Description Language, which provides a way to describe the simple aspects of a web service such as data types, operations supported by the service, bindings, and so on.

WSDL lets a provider describe a service in XML. But even that doesn't solve the problem, since a provider can't describe critical business aspects of their services, such as price or contractual requirements. Furthermore, to get a particular provider's WSDL document, you must know where to find them. Enter another layer in the stack, Universal Description, Discovery, and Integration (UDDI), which is meant to aggregate WSDL documents. But UDDI does nothing more than register existing capabilities; it doesn't actually create shared semantics either. As the UDDI technical white paper says,

The ability to locate parties that can provide a specific product or service at a given price or within a specific geographic boundary in a given timeframe is not directly covered by the UDDI specifications. These kinds of advanced discovery features require further collaboration and design work between buyer and sellers.

Even after using XML to create a description in WSDL and registering it with UDDI, two arbitrary entities still can't achieve automatic interoperability because there is no guarantee that an entity looking for a Web Service will be able to specify its needs clearly enough that its inquiry will match the descriptions in the UDDI database.

Even the UDDI layer does not ensure that the two parties are in sync. Shared context has to come from somewhere, it can't simply be defined into existence.

Its Windy, Ain't It? No, Its Thursday.

UDDI has proposed solution, yet another set of layers, optimistically entitled the Universal Service Interop Protocols, though, as they dryly note on their Web Services stack diagram, "these layers are not defined yet."

This attempt to define the problem at successively higher layers is doomed to fail because it's turtles all the way up: there will always be another layer above whatever can be described, a layer which contains the ambiguity of two-party communication that can never be entirely defined away.

No matter how carefully a language is described, the range of askable questions and offerable answers make it impossible to create an ontology that's at once rich enough to express even a large subset of possible interests while also being restricted enough to ensure interoperability between any two arbitrary parties.

The sad fact is that communicating anything more complicated than inches-to-millimeters, in a data space less fixed than stock quotes, will require AI of the sort that's been 10 years away for the past 50 years.

Without AI, the description and discovery of Web Services is going to require a great deal of good old fashioned human intelligence. And whenever humans get involved, things get sticky.

Human Factors: Advertising, Branding, and Context Errors

WSDL and UDDI are earnest attempts to provide a framework for rigorous business descriptions, but there is no third party to judge the accuracy of the contents of the documents. If WSDL and UDDI are a way to advertise (note the word) the availability of services in order to get customers, then there will be permanent pressure on the creators of those documents to design their descriptions to get the maximum number of subscribers, rather than to create maximum accuracy.

Inasmuch as there is room for open-ended description of services, we'll see the familiar cat-and-mouse game between advertisers' claims and potential customers' interest, thus restoring the need for human judgment on the customers' part -- caveat emptor. And if WSDL and UDDI only work for services that can be unambiguously described, then they'll be limited to that tiny subset of business problems that require no human approval -- calculating shipping costs, looking up Zip Codes, reordering toner. Hardly the stuff of revolution.

Even in the world of completely commodified services, the pressure to advertise will create barriers to seamless interoperability. It has long been possible to upgrade operating systems with new drivers in the background, or provide bookmarkable URLs for complex searches, to take but two examples. Yet when a user installs a new printer, they are still frequently walked through an install routine, because the printer manufacturer wants to brand their service and offer the user other products and services. Likewise, many sites with search engines use POST instead of GET for their forms, so that the users can't bookmark the results, but must go back to the search page, with its attendant ad impression mechanisms.

While commodity services are good for customers, businesses dislike them; even in the cases where Web Services can be used to offer interchangeable services in the background, businesses are unlikely to embrace those functions if it means they lose the ability to brand their products and services.

Finally, even without the pressure to advertise, human factors create subtler difficulties because our minds are so given to intuition and induction, at the expense of predictability and consistency. The real world abounds with examples of these sorts of context errors, where two parties to a conversation operate with different assumptions even when they are trying to communicate clearly.

As an example, the interface to the NYC subway card vending machines offers two flavors of card: Regular Metrocard and Unlimited Ride Metrocard. The Regular metrocard, though, is a misnomer. It allows the user a fixed number of rides, and it was thus analogous in the system designers' minds to the old token system, but for the users, there is nothing more regular about one kind of card than another, since both kinds of cards were introduced at the same time. The change from the users' point of view was from tokens to cards; the distinction that made sense to the system designers has nothing to do with the users' frame of reference.

There's no guarantee that even when the people offering a new service are trying to describe the service unambiguously that they will be able to find a language that the recipients understand, a problem that is well documented by usability theorists like Mark Hurst, Don Norman, and Jakob Nielsen. Just because Web Services dispense with the world of icons and pull-down menus does not make them any less driven by user interfaces or any less susceptible to mismatches between the expectations of sender and recipient.

Web Services can't create a framework in which any two arbitrary applications can interact because XML doesn't provide shared languages, merely shared alphabets. The Web Services stack pushes this shared semantics problem into higher and higher layers without solving it. Humans often cannot create perfectly transparent descriptions even when they are trying to, and they simply won't try when there's an economic incentive to stretch the truth.

Web Services Are Not the Web

The Big Picture notion of Web Services as a way to automate remote transactions is based on a false comparison with the Web. The idea is that Web Services can do for applications what the Web did for publishing if Web Services adopt the Web model (roughly: loosely coupled passing of structured documents) with two minor changes: the documents will not be limited to a predefined set of methods and tags, and the recipients of the data will be other applications, rather than human beings.

The problem is that these are not the minor changes they seemed to be at first. A big part of the reason the Web worked as well as it did is because it had limited methods and tags and because the recipients of the data were human beings.

By using a limited and pre-defined set of methods (GET, POST, and so on) and tags (basic HTML), the Web achieved interoperability between arbitrary clients and servers the same way the stock quote examples do: by defining the data set in advance and by ensuring that all parties adhered to it. (There's even a name for this architecture, REpresenational State Transfer, (REST), and its proponents are seeking to define it well enough to apply it to other kinds of problems.) The very extensibility of XML means that this sort of coordination can never exist in the Web Services world as a whole, and it can only exist in smaller domains where all parties agree in advance to a common set of semantics.

Furthermore, humans are good at pulling information from noise. We are able to extract data we need from badly written, badly formatted web pages. There is no machine in the world that comes close to our ability to comprehend a wide range of inputs. Replacing the human recipient of the data with a computer application involves a huge loss of decoding power on the client side. Humans can be counted on to read between the lines -- applications can't even read the lines themselves. Applications must operate in a domain with narrower semantics than the Web had, not broader ones.

A real comparison with the Web suggests that the Web model is best extended into the application domain not by making it infinitely extensible, but by implementing it in areas where broad agreement on what and how to communicate already exists. Like the Web, in other words, Web Services are likely to work best in areas with predefined methods and vocabularies, where the recipient can be counted on to be able to decode the sender's message.

Private Conversations and Contractual Data

In contrast to the presentation of public and universally interoperable services, Web Services are actually best at automating private, previously negotiated conversations. The idea of unknown but perfectly described capabilities existing out there in the cloud -- at once unfamiliar enough to need to be discovered, tailored and reliable enough to build a business on, and not so critical that they need to be hosted locally -- describes a small and fairly trivial set of possible services. The ASP business ran aground on this issue, and there is no sign that the Web Services solution will work any better.

Where Web Services work beautifully, however, is where the parties involved already agree about interesting things and already have a framework for cooperating or collaborating. This is not the sexy stuff of a consumer revolution. Instead it is a combination of RPC++ and EDI++: automating things like remote database lookups, lowering costs in supply chain management, or providing developer groups easier ways to link applications.

An interesting example of what a group of like-minded developers can do with Web Services is the Blogger, Manila, and Jabber collaboration, which is using XML-RPC. But in this instance, as in all working Web Services examples, there is an existing relationship, and the semantics are already widely shared. Web Services can't replace developers but may let them get through the gritty details faster, once they've worked out the high level agreements.

Web Services is a background technology, a plumbing technology, a way to lower current thresholds, not Yet Another Paradigm Shift. And that in itself is enough to be a big deal. Lowering the effort and costs associated with interoperability is a good goal. Let's just hope the real advantages of sending data between small or private groups doesn't get swamped by the hype of perfect but unachievable automation.