XML.com: XML From the Inside Out
oreilly.comSafari Bookshelf.Conferences.

advertisement

Fun with Amazon's Simple Queue Service

January 05, 2005

Amazon.com has been exposing itself again, much to the delight of developers, entrepreneurs, and media junkies. This time, they've revealed Amazon Simple Queue Service (SQS), a basic message queue service with a web services interface. Originally developed and used in-house by internal Amazon developers, the API is now publicly exposed so that everyone can join in the fun.

Message queues are normally the stuff of big-business middleware and operating system internals, but when you have easy access to a high-performance, high-bandwidth, network-accessible message queue service with a simple API, well, there are a lot of potential applications. Importantly, Simple Queue Service 1.0, which is in beta now, is free for developers to play with. Amazon will start charging for use when it is released (although no price has been set, I'm told that it will be available at "a low cost that will not restrict innovation," whatever that means). But for now, it's totally free, so let's check it out.

Accessing Amazon Web Services

Like other Amazon Web Services, you can access Simple Queue Service in three different ways: by using an XML-over-HTTP (what Amazon calls a REST) interface, a SOAP interface, or by taking advantage of Amazon's XSLT engine. For this article, I'm focusing on using the REST API. But, before you can access any of Amazon's Web Services, you need to obtain an Amazon Subscription ID. Don't worry, it's totally free. You can get one here. The subscription ID is used by Amazon to generate usage statistics and to identify developers who may attempt to abuse their generous free offerings. What constitutes abuse or inappropriate use of the various Amazon Web Services? Read the license agreement. I'm not a lawyer, but basically, no porn and don't average more than one request per second. Capiche? Failure to adhere to the license agreement can get your originating IP address blacklisted by Amazon.

The API

The API for Amazon Simple Queue Service is, as the name suggests, quite simple and easy to use. Their are seven methods (Amazon calls them "operations," which is what I'll call them) which are summarized in table 1-1. For more specific details, you can refer to Amazon's online documentation.

Table 1-1 Amazon Simple Queue Service Operations
Operation Description
CreateQueue Create a message queue
DeleteQueue Delete a message queue
ListMyQueues List my messsage queues
Read Read an entry, or entries, from a queue
Enqueue Add an entry to a queue
Dequeue Delete an entry, or entries, from a queue
ConfigureQueue Modify details about a queue

Structure of Simple Queue Service Requests Using REST

Using Amazon's REST interface, I could demonstrate the entire API without using a programming language at all. That's because Amazon chose to implement SQS using only HTTP GET requests (no PUTs, DELETEs, or POSTs). I could simply construct the appropriate URLs and put them into my favorite web browser to execute them. While some might disagree with this design (which is precisely what Joe Gregorio has done in this week's The Restful Web column-- Editor) about Amazon's non-RESTy choice to use only HTTP GETs for SQS, the fact is that it makes life very simple for developers. All SQS REST requests have the same basic URL structure:

http:/webservices.amazon.com/onca/xml?Service=AWSSimpleQueueService&
    SubscriptionId=[Your Subscription ID Here]&
Version=[Version number]&Operation=[Operation from Table 1-1]
    &[some parameter]=[some value]&[some parameter]=[some value]... 

A few things to point out about the SQS request structure:

  • All requests start with the same prefix: http:/webservices.amazon.com/onca/xml?Service=AWSSimpleQueueService
  • All requests require a Subscription ID: SubscriptionId=001VHHVC74XFD88KCY82
  • Though optional, it's a good idea to include the Version of Simple Queue Service that you're using: Version=2004-10-14. Yes, the version is a date. October 14th, 2004 is the first version of Simple Queue Service. Subsequent versions will have newer dates. To find the date of the latest version, look at the date in the targetNamespace string in the SQS WSDL.
  • All requests must specify an operation from Table 1-1. For example: Operation=CreateQueue
  • All requests are case-sensitive. If you use Operation=createqueue instead of Operation=CreateQueue, the request will fail

Simple Queue Service Design and Limitations

In SQS, a queue is a list of entries. An entry is a blob of data. When you create a queue, it's empty until you put entries on it. You can then read the entries you put on the queue. Entries remain on the queue until you explicitly delete them. The ordering of the queue is FIFO (First In, First Out). However, because access to the queues is heavily cached for performance reasons, Amazon does not guarantee FIFO access--occasionally, SQS requests may return results in a slightly different order than FIFO.

You may specify a Read Lock on any queue you create. The Read Lock specifies how long (in seconds) an entry in a queue will be unavailable to subsequent read operations after it is returned by a read operation. The default Read Lock for queues is 60 seconds. The minimum value is 0 seconds. Read Lock can be thought of as an advisory locking mechanism that's useful for cases where you may have several processes reading from a queue, but you only want one process to handle a particular entry.

Each queue you create is tied to your Subscription ID. You can only access queues that were created by your Subscription ID. So, if you want to make sure that others don't access your queues, you need to make sure that they don't find out your Subscription ID. For more privacy, you can make requests via HTTPS.

It'd be way cool if I could create millions of queues stuffed with my favorite song lyrics, or my copious notes about the bogus information in Texas high school history textbooks, but, while Simple Queue Service may be free for the moment, it's not that generous. When using SQS, the following limitations apply to all queues created by a single Subscription ID:

  • Maximum number of queues you can create: unlimited (This is what the Amazon folks tell me--care to test their assertion?)
  • Total number of queue entries: 4000 (combined total of all your queues)
  • Maximum size of a single queue entry: 4Kbytes
  • Maximum length of a Readlock: 30 days (2,592,000 seconds)

Creating a Queue

You can create a queue named "Jason" by dereferencing the following URL (broken across lines for readability). If you don't replace my Subscription ID, 001VHHVC74XFD88KCY82, with your own, then this request will generate an error because I (or someone else reading this article) already created the queue.

http:/webservices.amazon.com/onca/xml?Service=AWSSimpleQueueService
    &Version=2004-10-14&SubscriptionId=001VHHVC74XFD88KCY82
    &Operation=CreateQueue&QueueName=Jason

If it succeeds, you'll see this in your web browser:


<?xml version="1.0" encoding="UTF8"?>
<CreateQueueResponse 
  xmlns="http://webservices.amazon.com/AWSSimpleQueueService/2004-10-14">
  <OperationRequest>
    <HTTPHeaders>
      <Header Name="UserAgent" Value="Mozilla/4.0 (compatible; MSIE 6.0; 
      Windows NT 5.1)"/>
    </HTTPHeaders>
    <RequestId>054FT186J7SQ6CKZDJC8</RequestId>
    <Arguments>
      <Argument Name="Service" Value="AWSSimpleQueueService"/>
      <Argument Name="QueueName" Value="Jason"/>
      <Argument Name="SubscriptionId" Value="[Your Subscription Id]"/>
      <Argument Name="Version" Value="2004-10-14"/> 
      <Argument Name="Operation" Value="CreateQueue"/>
    </Arguments>
  </OperationRequest>
  <CreateQueueResult>
    <Request>
      <IsValid>True</IsValid>
      <CreateQueueRequest>
        <QueueName>Jason</QueueName>
      </CreateQueueRequest>
    </Request>
    <QueueId>0KJ2NTM8MW1QNJQY2YZB</QueueId>
  </CreateQueueResult>
</CreateQueueResponse>

If you didn't replace my Subscription ID with yours, you might see something like this:


<?xml version="1.0" encoding="UTF-8"?>
<CreateQueueResponse 
 xmlns="http://webservices.amazon.com/AWSSimpleQueueService/2004-10-14">
  <OperationRequest>
    <HTTPHeaders>
      <Header Name="UserAgent" Value="Mozilla/4.0 (compatible; MSIE 6.0;
       Windows NT 5.1)"/>
    </HTTPHeaders>
    <RequestId>04CGJDQDS8A1QK5091N6</RequestId>
    <Arguments>
      <Argument Name="Service" Value="AWSSimpleQueueService"/>
      <Argument Name="QueueName" Value="Jason"/>
      <Argument Name="SubscriptionId" Value="001VHHVC74XFD88KCY82"/>
      <Argument Name="Version" Value="2004-10-14"/> 
      <Argument Name="Operation" Value="CreateQueue"/>
    </Arguments>
  </OperationRequest>
  <CreateQueueResult>
    <Request>
      <IsValid>True</IsValid>
      <CreateQueueRequest>
        <QueueName>Jason</QueueName>
      </CreateQueueRequest>
    </Request>
    <SimpleQueueServiceError>
      <ErrorCode>AWS.SimpleQueueService.QueueNameExists</ErrorCode>
      <ReasonText>
       A queue with that name already exists for this subscription.
      </ReasonText>
    </SimpleQueueServiceError>
    <QueueId>0KJ2NTM8MW1QNJQY2YZB</QueueId>
  </CreateQueueResult>
</CreateQueueResponse>

Let's talk about a few notable things revealed by these responses.

First, note that all responses are UTF-8 encoded. Your requests should also be UTF-8 encoded.

<?xml version="1.0" encoding="UTF-8"?>

The operation we used, along with the word "Response", and the default namespace identifier are returned next:

<CreateQueueResponse 
 xmlns="http://webservices.amazon.com/AWSSimpleQueueService/2004-10-14">

The OperationRequest block is next. The HTTP UserAgent header is echoed back. The RequestId is a unique ID that is generated for each request by Amazon. If you encounter serious problems with Amazon Queue Service, you can report this ID to Amazon. Your request arguments are then echoed back. These are useful if you are using XSLT style sheets and want to pass parameters into your style sheet from the request.


 <OperationRequest>
    <HTTPHeaders>
      <Header Name="UserAgent" Value="Mozilla/4.0 (compatible; MSIE 6.0;
       Windows NT 5.1)"/>
    </HTTPHeaders>
    <RequestId>04CGJDQDS8A1QK5091N6</RequestId>
    <Arguments>
      <Argument Name="Service" Value="AWSSimpleQueueService"/>
      <Argument Name="QueueName" Value="Jason"/>
      <Argument Name="SubscriptionId" Value="001VHHVC74XFD88KCY82"/>
      <Argument Name="Version" Value="2004-10-14"/> 
      <Argument Name="Operation" Value="CreateQueue"/>
    </Arguments>
 </OperationRequest>

The actual results are in the final block, which has the container element CreateQueueResult, the operation name plus the word "Result." The "Request" block that follows is just your request arguments echoed back. The IsValid tag is always present and has the value of either "True" or "False." If it's "True," Amazon's server considers your request valid, which means that the syntax is correct. If it's "False," then Amazon's server found a problem with your request syntax. The container tag CreateQueueRequest contains the CreateQueue operation parameters.

Following the Request block is the actual response, in this case, the Queue ID of the queue created by Amazon.


<CreateQueueResult>
    <Request>
      <IsValid>True</IsValid>
      <CreateQueueRequest>
        <QueueName>Jason</QueueName>
      </CreateQueueRequest>
    </Request>
    <QueueId>0KJ2NTM8MW1QNJQY2YZB</QueueId>
</CreateQueueResult>

Had the request failed, as in the second example above, the CreateQueueResult block would look like this:


 <CreateQueueResult>
    <Request>
      <IsValid>True</IsValid>
      <CreateQueueRequest>
        <QueueName>Jason</QueueName>
      </CreateQueueRequest>
    </Request>
    <SimpleQueueServiceError>
      <ErrorCode>AWS.SimpleQueueService.QueueNameExists</ErrorCode>
      <ReasonText>
       A queue with that name already exists for this subscription.
      </ReasonText>
    </SimpleQueueServiceError>
    <QueueId>0KJ2NTM8MW1QNJQY2YZB</QueueId>
 </CreateQueueResult>

The request was still a valid request (IsValid is "True"), but since a queue with the name "Jason" already existed for that Subscription ID, the request failed. Queue names must be unique for a given Subscription ID. When a request fails, a SimpleQueueService Error block appears instead of the QueueId. An Error block always contains an ErrorCode and a ReasonText. A complete list of possible error codes is available in the documentation.

Pages: 1, 2

Next Pagearrow







close