Google's Gaffe

April 24, 2002

Paul Prescod

Google's release of an API has been heralded as a bright moment for web services. It is an exciting development, but at the same time there is a subset of the programmer community that is disappointed. Google had a similar XML-based API a year ago, but neither documented nor publicized it. Merely by changing "/search" to "/xml" in any Google query, you could get back an XML representation of the query result. It became a pay-only service last fall, which, while unfortunate, was understandable as the service was a straight money-loser.

Imagine the surprise at Google reviving the service, adding new features, documenting and promoting it. But Google also moved it to an inferior technical platform, SOAP RPC running over HTTP POST. On many mailing lists, weblogs and discussion groups the reaction was mixed. It felt like one step forward and two steps back.

Google seems to be caught up in the hype of SOAP-based web services. Conversely, Amazon has recently released a program for its partners based upon pure XML, HTTP and URIs. eBay has long had an XML/HTTP/URI-style service.

In this article I demonstrate that Google's choice was technologically poor, compared to that of eBay and Amazon. I will show that a Google API based on XML, HTTP and URIs can be simpler to use, more efficient, and more powerful.

Why Google Is Special

In previous articles, I have demonstrated how SOAP-based services can be re-engineered around pure HTTP. Although the uptake of web services has been slow, I cannot suggest reinventions of each and every one of them as an XML/HTTP/URI-style service, but the new Google API is special for a variety of reasons.

  • Google has had an XML/HTTP/URI-style API in the past. The choice to use SOAP would seem to indicate a need to move beyond HTTP. My experiment demonstrates that this is not so.
  • Google's service seems almost unique in how simply and clearly one can define an XML/HTTP/URI interface. The new API is an obfuscating SOAP wrapper over a simple XML/HTTP/URI service.
  • Google the corporation has always been known for its technological acumen and general cluefulness. I think that it can be convinced to do the right thing and restore the HTTP interface (perhaps alongside the SOAP interface).

In addition, I think that the Google move is important symbolically. I take the SOAP-ifying of Google as a sign that the web services hype has now reached overdrive. I regularly hear customers say, "We have a working XML system but we know we'll have to move to SOAP soon." They are going to migrate their working systems to an unproven technology with a questionable design because they feel the dominance of SOAP is inevitable.

Google's SOAP API

Let's take a look first at Google's SOAP API. There are three methods.

doGoogleSearch searches the Google archive and returns an XML document with information about the query (e.g. number of hits) and with a list of result elements.

doSpellingSuggestion requests a spelling correction suggestion. For instance it would suggest that "britney speers" be corrected to "britney spears". It takes only a key and string as parameters and returns only a string for the suggestion.

doGetCachedPage requests a cached page from Google. It takes a key and a URI as parameters and returns base64-encoded data for the cached page.

Below is a complete message from client to server using the SOAP API. It searches for the query string "constantrevolution rules xml":

POST /search/beta2 HTTP/1.1
Accept-Encoding: identity
Content-Length: 914
SOAPAction: urn:GoogleSearchAction
Content-Type: text/xml; charset=utf-8

<?xml version='1.0' encoding='UTF-8'?>
        <ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch"
            <key xsi:type="xsd:string">0000</key>
            <q xsi:type="xsd:string">constantrevolution rules xml</q>
            <start xsi:type="xsd:int">0</start>
            <maxResults xsi:type="xsd:int">0</maxResults>
            <filter xsi:type="xsd:boolean">true</filter>
            <restrict xsi:type="xsd:string"></restrict>
            <safeSearch xsi:type="xsd:boolean">false</safeSearch>
            <lr xsi:type="xsd:string"></lr>
            <ie xsi:type="xsd:string">latin1</ie>
            <oe xsi:type="xsd:string">latin1</oe>

The message starts with HTTP headers because Google runs SOAP over HTTP. The only SOAP-specific header is the SOAPAction header which is theoretically useful for filtering SOAP traffic. Practically speaking it is optional and thus not a good basis for filtering.

The XML part of the message starts with some namespace declarations and a SOAP encodingStyle declaration, which is more or less boilerplate. The SOAP Body element can also be treated as boilerplate. Inside it is the method name, "doGoogleSearch".

Next there's an ordered list of parameters. The allowed parameters for the doGoogleSearch method are "key" (userid/password), "q" (query text), "start" (where to start returning in the results), "maxResults" (number of allowed results), "filter" (filter out very similar results), "restrict" (country or topic restrictions), "safeSearch" (pornography filter), "lr" (language restrict), "ie" (input encoding) and "oe" (output encoding). None of the parameters are optional.

The "key" parameter is special. It is more or less a password assigned to a particular software developer. The Google API seems not to support SSL so these always pass across the wire in cleartext.

For this particular query I set maxResults to 0 so that I won't actually get any results. Google returns a lot of XML metadata about the query itself so working with query "hits" would make my examples too long.

As you can see, parameters are strongly typed. Although the client and server know the types in advance, most SOAP toolkits will inline the types into the message. As you will see, this is entirely unnecessary.

HTTP-ifying the API

To reinvent this as an HTTP-based method, I merely have to translate the parameters into "query parameters" in a URI. A couple of hundred lines of Python serve to implement the mapping from these query parameters into Google SOAP (for all three methods). Of course one could have done the same thing in ASP, JSP, PHP, etc.

Here is the sort of URI used to address my version of the service:


Nobody (even a programmer) ever needs to look at this URI. It can be autogenerated, just as the SOAP message is. For instance here is a complete Python program to generate the URI and fetch the result:

import urllib

params = {"key":"0000",
          "q":"constantrevolution rules xml",
encoded_params = urllib.urlencode(params)
url = "http://mymachine/cgi/" + encoded_params
data = urllib.urlopen(url)

That's pretty simple, but I'll show later that by building on a WSDL service description we can make it even simpler.

HTTP deals with optional arguments much more gracefully than most WSDL-based SOAP toolkits, so I've left out any of the arguments that can be omitted. My script infers them just as the older XML/HTTP/URI Google API did.

Let's zoom in on the special key parameter in the URI. Remember that this is a form of authentication. HTTP has a built-in authentication mechanism, so really that would be a better way to handle it. SOAP does not support authentication yet, and authentication is (as of this writing) still an open issue for SOAP's RPC-over-HTTP binding. Of course for ultimate privacy (an an increased performance cost), I would recommend SSL-encrypting the entire message. I will leave key as a parameter for simplicity and to parallel the SOAP version more closely.

Here is the message that gets sent on the wire:

GET /cgi/ HTTP/1.0
Host: mymachine
User-agent: Python-urllib

That's it.

Now some might complain that the URI query parameters are just strings, whereas the SOAP parameters were strongly typed. This is not necessarily true. On the wire, of course, the parameters are just strings, just as SOAP messages are just strings. Only at the client and the server are the strings interpreted as other types. The client must know the types of the parameters in order to have made the call. The server must know the types of the parameters in order to have implemented the service. Given that both parties already know the types, it is a waste of bandwidth to declare the types of the parameters in each and every message. Even SOAP does not require it; it's merely a common SOAP idiom.

Which leaves the question of how types are communicated from the service provider to the programmer writing the client. Later I will show a way to declare the types strongly and statically enough to satisfy the most ardent Java or C# masochist.

Handling the Response

Here is the Google SOAP API's response message (formatted for readability, the original is available).

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 01:41:08 GMT
Content-Length: 1325
Content-Type: text/xml; charset=utf-8

<?xml version='1.0' encoding='UTF-8'?>
      <return xsi:type="ns1:GoogleSearchResult">
        <searchTime xsi:type="xsd:double">0.071573</searchTime>
        <endIndex xsi:type="xsd:int">0</endIndex>
        <searchTips xsi:type="xsd:string"></searchTips>
        <searchComments xsi:type="xsd:string"></searchComments>
        <startIndex xsi:type="xsd:int">0</startIndex>
            rules xml</searchQuery>

Note that the resultElements element is empty because we asked for zero hits. I warned you that we would have quite a bit of XML to work with even without looking at any hits.

HTTP allows any media type in the response to a message. It could return any XML vocabulary whatsoever. The WSDL definition of the Google SOAP API already embeds a simple schema. I will use that as the basis of a schema for responses to search requests. I merely have to remove a few SOAP-isms (there is an SOAP-Enc:arrayType attribute, the SOAP-ENV:Envelope element, etc.) and choose a new root element type (I chose searchResult).

I've called the new language GoogleML. I've written a tiny XSLT stylesheet, pureGoogle.xsl, which translates GoogleSOAP into GoogleML. The resulting documents are smaller and simpler.

And here's a GoogleML equivalent to the SOAP response:

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 02:29:56 GMT
Content-Type: text/xml

<searchResult xmlns="">
    <searchQuery>constantrevolution rules xml</searchQuery>

HTTP provides the envelope so the redundant SOAP:Envelope would be redundant. The types are known in advance so they are stripped (although I could just as easily have left them in). GoogleML documents are not constrained to the subset of XML supported by SOAP. They may use any feature in standard XML, including a DOCTYPE, a DTD and processing instructions.

Handling the Other Methods

Next let's look at the getCachedPage method. Here is the original SOAP request (again formatted, original here).

POST /search/beta2 HTTP/1.1
Accept-Encoding: identity
Content-Length: 577
SOAPAction: urn:GoogleSearchAction
Content-Type: text/xml; charset=utf-8

<?xml version='1.0' encoding='UTF-8'?>
    <ns1:doGetCachedPage xmlns:ns1="urn:GoogleSearch" 
      <key xsi:type="xsd:string">0000</key>
      <url xsi:type="xsd:string">

Now compare that to an HTTP-style cachedPage request:

GET /cgi/ HTTP/1.0
Host: mymachine
User-agent: Python-urllib/1.15

Believe it or not, there is an even more dramatic improvement in the response. Here is the SOAP response (I have elided the majority of the embedded base64-encoded data and reformatted; unformatted version).

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 03:01:40 GMT
Content-Length: 10744
Content-Type: text/xml; charset=utf-8

<?xml version='1.0' encoding='UTF-8'?>

The SOAP API must base64-encode the data because the result is HTML (not well-formed XML) and perhaps could eventually include binary formats like Word documents.

Here is the HTTP-style response:

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 03:44:34 GMT
Content-Type: text/html


Because HTTP has no problem directly embedding HTML (or any other textual or binary data type), there is no reason to base64-encode the data. Base64 data is always more verbose than unencoded binary data; the HTTP/URI version of the service will always save bandwidth and CPU power. By the way, Google already provides a service to get a cached page using exactly this technique.

For brevity, I'll skip comparing the doSpellingSuggestion requests and go directly to the responses. Consider the old-style SOAP doSpellingSuggestion response (unformatted version):

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 03:01:40 GMT
Content-Length: 10744
Content-Type: text/xml; charset=utf-8

<?xml version='1.0' encoding='UTF-8'?>
      <return xsi:type="xsd:string">britney spears</return>

Compare that with the HTTP version:

HTTP/1.1 200 OK
Date: Thu, 18 Apr 2002 04:08:27 GMT
Content-Type: text/plain

britney spears

I could have used XML for the response, but it's overkill for this job.

Statically Declaring Types

Many people assume that the difference between HTTP-based services and RPC-based services is that the former are loosely or dynamically typed and the latter are strongly or statically typed. That's not especially true. The choice between HTTP and SOAP is a choice between protocols. The decision to statically type-check information passing across the wire has more to do with service description. The two issues are totally separate.

I've already demonstrated how one can strongly type-declare the responses to HTTP-based services using W3C XML Schema. If you want to type-declare the URI query parameters, then you can use a language designed for type-declaring HTTP-based services like Web Resource Description Language (WRDL). WRDL is still under development, but you can already solve this problem today using the more popular WSDL.

Although WSDL is most often used with SOAP, it can in fact type-declare the parameters for simple HTTP services. Here is the relevant bit of a WSDL for my HTTP version of the doGetCachedPage method:

<operation name="doGetCachedPage">
  <http:operation location="/cached_page"/>
   <mime:content type="text/html"/>

It turns out that WSDL's handling for the <http:urlEncoded/> works almost perfectly. It gets the parameter names from the operation's part names. If you know WSDL, then that will probably be clear to you. If not, don't worry about it, it won't affect your understanding of what follows. The input description for the other two methods is identical, so we will concentrate on the output elements.

The output for the doSpellingSuggestion has a media-type of "text/plain":

<output><mime:content type="text/plain"/></output>

Finally, the output for doGoogleSearch is in XML so we declare that directly:

<output><mime:mimeXml part="searchResult"/></output>

This refers to a part named searchResult which is based upon an element type of the same name.

I do not want to oversell WSDL's HTTP features. You cannot define sophisticated HTTP-based web services with WSDL. It falls down as soon as a web resource generates links to another web resource. WSDL cannot express the data type of the target resource. In other words it can describe only one resource and not the links between resources. SOAP lacks a first-class concept of resource and especially lacks a syntax for linking them. It is thus not surprising that WSDL inherits this flaw. Nevertheless, I hope that this weakness will be corrected in future versions of WSDL. In the meantime, this is the primary reason for the existence of the WRDL language.

But you do not have to wait for WRDL. WSDL is sufficient for Google's current API because the API does not make use of hyperlinks. This is a common failing of SOAP-based APIs which follows from the component-centric thinking that SOAP encourages.

What this means practically is that it is possible to generate statically typed APIs for languages like C# and Java for the Google/HTTP interface. For instance, I can generate a C# interface from an HTTP-based WSDL description. The strongly-typed code is functionally identical to the SOAP version. The C# generated from the HTTP/WSDL can have exactly the same strongly typed interface as the C# for the SOAP/WSDL version.

Here is the strongly typed C# code to do a Google search. The only thing that has changed from the version shipped with the Google API is the class name:

PureXMLGoogleHTTPBinding s = new
// Invoke the search method
GoogleSearchResult r = s.doGoogleSearch(keyBox.Text, 
    0, 1, false, "", 
    false, "", "", "");

Strong type checking (in C# and VB.NET) and easy API use (in a language like Python or Perl) are completely unrelated to the choice of protocol. It is .NET's WSDL and XML Schema features which handle the strong type checking, not its SOAP features. According to Don Box (one of the key inventors of SOAP), "At this point, I believe most SOAP plumbers have conceded that XML Schema will be the dominant type system and metadata format for interop."

HTTP works equally well with XML Schema, RELAX NG, Schematron and DT4DTDs because it strongly separates protocol data (which is in MIME format) from payload data (which is in XML).

In the spirit of truth in advertising, to get it actually working I did have to work around several bugs in Microsoft's WSDL toolkit. I hope that WSDL implementors will its HTTP binding seriously and implement it properly. Once the bugs are fixed, Microsoft's WSDL/HTTP interface to the service for C# and VB programmers will be the same in every detail as the WSDL/SOAP interface. This is true of any complete implementation of the WSDL specification because it is WSDL and XML Schema that define the service's interface, not SOAP.

In the meantime, the bugs really do not matter because Google ships a .NET interface to the service in its toolkit. Google could easily ship an HTTP-based interface. To client code there would be no difference. The same argument applies to the supplied Java binding. It would be relatively simple to ship an HTTP-based binding instead of the existing SOAP one.

Dynamically Declaring Types

Some people are not willing to go to the extra effort of publishing explicit service descriptions (e.g. WSDL or WRDL). That's fine. There are a variety of inline-type annotation techniques you could use. Examples include the XMI, MIME-RPC, XML-RPC, WDDL, and even SOAP's "section 5" encoding. No one of these has become the standard because the common wisdom in the strongly typed XML world is that it will be much more common to express the types in some form of external schema such as XML Schema.

In other words, dynamically parsing XML into language-native data types is a problem that has been solved over and over again. If you happen to prefer the SOAP solution, then use the SOAP encoding. My opposition is to the SOAP-RPC protocol, not the SOAP encoding.

Service as API

An important issue in all of this is ease-of-use. If it is no longer simple for programmers to use the service, then it will fail.

If you are a Java or C# programmer, you use the client bindings provided by Google. The details of the protocol are invisible to you. You can skip the rest of this section if you wish.

What if you are using one of the few other languages (e.g. Perl or Delphi) with a decent SOAP/WSDL toolkit? If the toolkit is complete then it will support WSDL's HTTP binding. You can use exactly the same API to access the service as if you were using the SOAP version. Admittedly, toolkits supporting the HTTP bindings are few and far between, but that can, and hopefully will, change quickly. Google could lead the way here.

What if you are using a language with no special toolkit at all, just a standard programming language distribution or IDE? Obviously the HTTP solution wins because SOAP and WSDL support is not yet embedded in most programming language distributions. It takes five lines of Java to get a cached page or spelling correction from my version of the service. You don't need even an XML parser installed to handle two of the three methods.

Other Non-problems

You might think that the URI form would have problems with internationalization. That is not so. Google already supports internationalized URIs. How else could non-English speakers search the Web? You might think that the URI-based API would have trouble with large queries. That is also not true. Google limits the lengths of queries explicitly, even for SOAP messages. Google's limits are well within the safe reaches of URIs. There are ways to use HTTP to deal with arbitrarily large queries so that Google can loosen their limits later without outgrowing HTTP.

Now your application may have different requirements than Google's so the analysis may be quite different. HTTP is not necessarily better than SOAP-RPC for every application in the world. In my experience, however, the HTTP solution is better for any project where a public (as opposed to individual or departmental) interface is needed for a service. If you believe you have a counter-example (a concrete, not abstract, counter-example), I'd like to discuss it. Every time I convert a service, SOAP advocates say: "yes, sure, that service would be better done as HTTP, but there are others that are not." I am still looking for evidence of this.

Advantages of the HTTP API

Let's consider the benefits of the HTTP version. First, I have already discussed how HTTP has standardized ways of doing authentication and of handling non-XML (including non-textual) data. The message size is a tiny fraction of the original. More important, it is much easier for a typical system security administrator to filter, log, monitor and read.

The CPU load is also likely to be significantly smaller. I shudder to think how many cycles are being wasted around the world creating redundant SOAP messages for Google (sometimes base64'ing data) and then parsing them again on the other side. Remember that each and every message is also an HTTP message and must be built and parsed as such. Readers in California should remind Google that electricity is precious. They could switch to pure HTTP and turn off a couple of wasted server clusters.

Users of the HTTP version have no need to install a SOAP implementation like .NET's or Apache's. They can use any HTTP implementation, including Internet Explorer, Netscape, Mozilla, Lynx, Opera, wget,, Python's httplib.HTTPConnection, Perl's LWP, etc. In fact, you could easily test the API through a plain old HTML form.

<p>Get cached page:</p>
<form method="GET" action="http://mymachine/cgi/">
Key: <input name="key">
Url: <input name="url">
<input type="submit">

The HTTP version is highly amenable to caching. You can set up a standard HTTP cache on your local network or use one at your ISP. Google could also do server-side caching using something as simple as Squid. Of course caching of SOAP is also possible if you write the cache implementation yourself. With HTTP, you can install any one of a variety of free or commercial caching proxy implementations.

The HTTP version could use the little-known feature of HTTP known as content negotiation. "Conneg", as it is known to its fans, allows each URI to map to a resource that can deliver up various representations of itself. For instance the service could return the same URI in GoogleML, SOAP, XML-RPC and HTML, depending on the preferences of the caller as stated using an "Accept" header.

The Web's Crowning Glory

But all of these advantages are like the tiny diamonds that ring the big rock. The most important advantage is that an HTTP version is part of the Web rather than merely being behind it. This point is subtle but the most central. A piece of information is on the Web if it has a URI, accessible through a Web protocol or is embedded in an accessible document. When Google exposes its service through SOAP, it is behind the Web because the object with the web URI is the SOAP component ("endpoint"), not the actual query results. I need to go through the component to get to the data, like making a phone call through an operator instead of dialing direct. But in the XML/HTTP/URI way of thinking, every possible query result has its own URI and thus is a first-class web data object.

This assigning of URIs is incredibly powerful. First, it means that in standard hypertext documents it is possible to link to query results. There are thousands of links around the Web to Google search result resources. Google's HTML incarnation is an important part of the web community as a service, but it is also an important part of the Web through the thousands of virtual "resource documents" it hosts like:

<a href="">Search Google for links about XML</a>

There are thousands of links to these documents scattered around the Web. So Google is both a service and a shared information base.

It is now time to move this concept into XML. You may recall the days when XML was intended to be a way of publishing structured information on the Web. One of the great virtues was supposed to be that result sets could be more compact and could be manipulated (e.g. sorted and filtered) using client-side scripting. Somehow we got sidetracked with RPC models and SOAP, but the core idea of delivering XML to the desktop is alive and well in Internet Explorer and Mozilla. We still have not achieved that and will not until machines and humans use the same access protocol: HTTP.

But URI addressability is important even for information that will never be seen by humans. Addressable XML can be composed into extremely sophisticated "compound-documents". Referring to context-providing or historical documents is a necessary component of any non-trivial business transaction.

There are a variety of web standards designed to make this easier, like XInclude, XSLT, XPointer, XLink, RDF and XML Topic Maps. For instance, consider a document that uses XInclude to aggregate query results:

<document xmlns:xi="">
<xi:include href="http://mymachine/cgi/
<xi:include href="http://mymachine/cgi/

Now consider an XSLT template that does the inclusions:

<xsl:template match="xi:include">
    <xsl:apply-templates select="document(@href)"/>

And then a template in another XSLT transform could collect the URL and snippet elements in the merged data, sort them by URL and transform them to HTML:

<xsl:template match="/">
    <xsl:for-each select="//goog:item">
        <xsl:sort select="goog:URL"/>
        <xsl:copy-of select="goog:URL"/>
        <xsl:copy-of select="goog:snippet"/>

Next consider an RDF assertion that related the query results to an individual's home pages:

<rdf:Description about=''>

Don't worry if you are uninterested in the particular technologies. Many great hyperlinked applications do not use any of them. After all, using links in XML need be no harder than putting a URL in an attribute value. The Web's strength is that it allows us to address, link, compare, relate and otherwise combine resources. You can use standardized syntaxes or proprietary ones based on your particular problem domain.

The point that has not yet filtered through to the mainstream of web services implementors is that linking is just as important for machine-to-machine applications as it is for human-facing applications. If it is impossible to link to SOAP-exposed resources, then they are less powerful and useful than HTTP-exposed ones. Until SOAP has an addressing mechanism as rich as HTTP URIs, SOAP is less, rather than more powerful than HTTP.

SOAP-based services are called "Web Services" because their proponents wish to partake of the Web's success -- yet they don't build on its core technologies, URIs and HTTP. Somehow many have equated SOAP and Web Services but HTTP has been in Service on the Web for more than a decade now and has not yet hit its prime. One other question for you to ponder. If, in general, most deployed Web Services use XML Schema for definition of types, language-specific databinding techniques for interpretation of types and HTTP for the transport of data, then what exactly is SOAP doing?

What Would Frodo Do?

Back in the days before XML, when we used SGML, we enthusiasts were like Hobbits in the Shire, hearing tales of the distant wars between Wizards (tools vendors), Dragons (database vendors) and Armies (MIS departments). But we were relatively sheltered from them. Those days have long since passed.

What we need to do is gather together a fellowship of like-minded Hobbits, Dwarves, Elves and men and go on a quest to educate the world about the limitations of SOAP-RPC interfaces. You can join the quest by signing the petition available, and if you don't mind registering for Google Groups, you can also participate in the thread discussing this topic. Make sure to spread the topic to other mailing lists, weblogs and discussion forums.

XML users have a special understanding of the possibilities of the marriage of hyperlinks, XML and the Web. We have a responsibility to go out of the Shire and educate the other residents of Middle Earth. Google is like the wizard Saruman, a benign and powerful force inexplicably turned from the path of virtue. Nevertheless, I am confident that it can be won back. The white robe awaits the delivery of Google API beta 3 with strong support for URI-addressed XML delivered over HTTP.