Clay Shirky: What Web Services Got Right ... and Wrong

April 23, 2002

Richard Koman

Web Services has gone from a vague idea to the major trend in Internet development, with Microsoft and Sun both recasting their entire companies to deliver on the promise of Web Services. To understand whether Web Services are real or hype, we talked to Clay Shirky, author of O'Reilly's Planning for Web Services report, a keynoter at the upcoming Emerging Technology conference, and a longtime observer of peer-to-peer networking.

Richard Koman: What's so great about Web Services, anyway? Why is this idea taking hold now?

Clay Shirky:I'm most interested in Web Services within the larger theme of decentralization. Once you "data-fy" the functions that used to be done by middleware, you suddenly get to this place where Web Services start to look like the way to implement peer-to-peer. In Web Services, unlike the Web itself, you don't have any client/server asymmetry built in.

Koman: One of the points Rick Rashid (director of Microsoft Research) made in my interview with him was that peer-to-peer is the norm and client/server is a special case.

Shirky: Right. My formulation is that client/server should be the description of a transaction and not a topology. Which is to say client/server is a normal relationship between any object that has something and any object that wants something. But there's no reason to define "this is a having object" and "this is a wanting object" for all transactions.

What we have now is Web servers as having objects and Web browsers as wanting objects. There's a lot of places where that just doesn't make any sense. You'd like to be able to treat any node as either a client or a server, depending on who's doing the having and who's doing the wanting.

Koman: And the fact that client/server has been a topology is historically related to the fact that individuals have had wimpy machines and servers have been big iron. But now users have gigahertz processors, hundreds of gigs of disk space, hundreds of megs of RAM ....

Shirky: Right. We have server-class boxes under our desks for $1,000. And the other enforcement wasn't just the iron but also the quality of the connection. I just saw this statistic saying there's as many broadband hours of use as there are dial-up hours of use. Plainly this isn't because the industry's done a good job of rolling out broadband; they've done a dismal job of rolling out broadband. What it says is, once someone gets a broadband connection they leave it on all the time.

Koman: Another interesting twist is people redistributing their connection outward via wireless transmitters.

Shirky: Yeah, that is so odd. I mean the economic logic of it makes perfect sense on some level, which is that the majority of the value is captured by the user, not by the owner or the operator of a network. So there's an economic argument that says the people who are capturing the value are the people who should be investing in the physical plant. And so in a way all of these daisy-chained 802.11b hubs are better able to extend bandwidth to the people who need it, [instead of] leaving the telcos as sort of limiting points of control.

Koman: In a world where the Baby Bells are so bad at deploying -- "We can give you DSL if you live within two miles of the station" -- it sort of makes sense that people are saying, "We can do better."

Shirky: Well, here's a data point for you. We're building a wireless network at NYU, and you can't currently walk into J&R Computer World and buy 802.11b equipment, because people ask for it as it comes off the truck. When they get a shipment then they sell out that day and then they have to get another shipment. We've crossed some unbelievable tipping point in wireless where it's easy enough to set up at home that people are just buying the kit down at J&R and doing it on their own.

One of the interesting things is that this thing has benefitted so much -- like the original growth in voiceover-IP -- from the fact that the bandwidth providers have not understood that unlicensed spectrum could in any way be a threat. The big issue now is whether or not they attempt to hijack the part of the FCC that actually allocates unlicensed spectrum, in order to damage or even roll back 802.

Koman: That all sounds very bureaucratic and sort of depressing.

Shirky: Yeah, it is weird, but one of the amazing things to me has been in the last two years, everyone's been so focused on the recession, of course, and the attack and so forth. But one of the big changes that was going on even as the recession was starting is that Washington now figures into people's plans.

It used to be that they had nothing to do with anything. But given things like the SSSCA and the DMCA and so forth, Washington has now become a control point that people are fighting for. And there's no way any longer for the entrepreneurs to simply say, "Oh, what goes on in Washington doesn't affect us." Because, of course, having seen that the Internet potentially allows new entrants to challenge the entrenched ... And of course the entrenched incumbents like Michael Eisner and Jack Valenti are down in Washington, so the entrepreneurs have to be, too.

It's a big change and it was masked a little bit by the enormity of some of the other changes, both in the industry and the country as a whole. But as the economy starts picking up and as we're used to being at war, some of the things that change in the meantime stand out. In my mind the importance of regulation, and unfortunately bureaucracy is one of those big changes. You see at the conference with the CIA and the FBI being present because they also have become aware that this technology's critical to them. So there's a kind of two-way conversation going on that's going to matter a lot more than it has in the past.

Koman: Let's move back to Web Services.

Shirky: Well, the thing to me about Web Services is it's obviously a good idea technically, and the low-level stuff is actually astonishingly far along. I think people don't realize how far along it is. But the general agreement that SOAP is a pretty good way to package and move data back and forth between applications is fairly widespread. And even though there are different implementations of SOAP, and even though there's still SOAP 1.2 yet to come, so many people are interested in having it work that it seems to me like it's going to work.

So the technological case for Web Services is really quite strong. The business case for Web Services is more limited right now, in part because they run under the same difficulties that the application service providers and B-to-B marketplaces did. Just because it's possible to adopt the same data standard as your competition, and just because that's better for the market as a whole, doesn't mean it's better for you.

So what I think we're going to see in Web Services is rather than adoption by the most fragmented industries, in fact the most consolidated industries will adopt it first because those industries can best impose standards by fiat and lower coordination costs among all players.

So if I'm IBM, the easiest place for me to get value out of Web Services is to Web-service my own internal stuff first. Because Web Services is more about cost savings, lowering the difficulty of doing something, more than it is about creating fabulous new sources of revenue.

In particular I think this notion of a cloud of anonymized services that you can discover in real-time and bind together at run-time to create a new application from scratch is just an absurdity. There's no business case for selling anonymized services. Anyone who's playing that market will want to differentiate themselves.

And any service which is so commodifiable that you could sell it that way is probably best done in-house. As the software industry has learned to its chagrin over and over and over again, people strongly prefer buying a product outright to paying an annuity for services. So Web Services is more of a technological infrastructure than it is a business model right now.

That having been said, one of the most important things about the technological infrastructure is, unlike the Web, call and response are in the same language. With the Web you have an HTTP request out and HTML back. The thing that knows how to make a HTTP request doesn't know how to serve an HTTP request. And the thing that knows how to send HTML doesn't know how to read HTML.

So this is what I was talking about earlier: client/server on the Web is defined in advance, based on what kind of object you are. You are an object that makes requests or you are an object that fills requests. With Web Services, if you can read SOAP you can write SOAP. If you can write SOAP you can read SOAP. So modulo only the firewall issue, everybody's a peer.

And although the big companies are selling this vision of a mega-server in the sky -- we're going to run a giant passport service, we're gonna sell lots of big iron -- in fact, once you have a client that can read and write SOAP, that plus an address and you're in business.

So given that the peer-to-peer world has been historically bad at standards, for a variety of historical reasons, it looks to me like the standards for peer communication are actually going to come out of Web Services. And they're going to be adapted to a world where client/server is the description of a transaction but not a topology.