XML.com: Beep BEEP!

Beep BEEP!

October 16, 2002

This article is the last in a series examining how one might go about sending binary data as part of a SOAP message. This month we look at BEEP, the Blocks Extensible Exchange Protocol.

The primary inventor of BEEP is Marshall Rose, a long-time "protocol wonk" within the IETF community. Marshall has authored more than 60 RFC's, covering everything from core SNMP details to a DTD for IETF RFCs.

Like SOAP 1.2, BEEP is described in a transport-neutral manner and is defined in RFC 3080. The most common transport is TCP, and the TCP mapping is found in RFC 3081. If you are at all interested in network protocols, you should read RFC 3117, "On the Design of Application Protocols". At the BEEP site you can find HTML versions of BEEP and related RFCs generated from the DTD mentioned above.

Related Reading

BEEP: The Definitive Guide
Developing New Applications for the Internet
By Marshall T. Rose

BEEP actually addresses a wider range of problems than just SOAP and binary data. According to the RFC, BEEP is "a generic application protocol kernel for connection-oriented, asynchronous interactions." In other words, it's like an application-level transport protocol. Unlike many application protocols, it supports more than just lockstep request-response interchanges, which makes it suitable for applications such as event logging (see RFC 3195, for "Reliable Syslog"), and general peer-to-peer interactions.

One of the more interesting implications of BEEP's design principals is that it supports multiplexing -- that is, multiple channels communicating over a single transport stream. BEEP allows different applications, or multiple instances of the same application, to use the same stream for independent activities, including independent security mechanisms. For example, a single BEEP-over-TCP link between a browser and a web server would efficiently allow the browser to fetch a page over SSL-encrypted HTTP, while also streaming in multiple images over unencrypted HTTP.

A BEEP message is the complete set of data to be exchanged. BEEP defines three styles of message exchange, distinguished by the type of reply the server (or, more accurately, the message recipient) returns:

Message/reply, the server performs a task, and sends a positive reply back.
Message/error, the server does not perform the task, and sends a negative reply back.
Message/answer, the server sends back zero or more answers, followed by a termination indicator. This style is appropriate for streaming data, for example.

Messages are divided into frames. The specification says that a message is normally sent in a single frame. Frames consist of a header, a payload, and a trailer. The header and trailer are encoded in printable ASCII and terminated with a CRLF -- in other words, typical IETF protocol style. The payload is an arbitrary set of octets.

The frame headers are almost identical and are structured as follows,

type SP channel SP msgno SP more SP seqno SP size CR LF

where SP, CR, and LF stand for the ASCII space, carriage-return, and linefeed characters respectively.

The type specifies the message type and is a three-byte strings -- MSG (message), RPY (reply), ANS (answer), ERR (error), or NUL (terminator) -- which correspond to the message types listed in the exchange patterns above.

The channel identifies the multiplex channel of the communication and is the printed form of a number between zero and 2³¹ - 1. BEEP reserves channel zero for management tasks, like creating new channels. So the simplest BEEP application will need two channels and is,therefore, likely to end up being multiplex-ready.

The msgno uniquely identifies the message. It's a number in the same format as channel and acts like a session identifier: a reply to a given message will have the same msgno. A msgno cannot be reused until the final response packet -- RPY, ERR, or NUL -- has been received.

The more indicator is a period if this is the only or last frame of a message. It's an asterisk if at least one other frame follows for this message. Note that frames can have a zero-length payload, so that final EOF can be sent by itself.

The seqno is an unsigned 32-bit number (i.e., twice the size of the other number fields) and specifies the offset of the first octet in the current payload. This is different from the conventional use of the term "sequence number"; perhaps "offset" might have been a better choice. In most cases, the seqno of frame n will be the sum of the payload lengths of the prior n - 1 packets. Some applications, however, might want to efficiently use BEEP to omit large sets of default values. For example, a distributed file system protocol could avoid sending a large number of all-zero-byte frames.

If a server is sending multiple replies to the client -- the message/answer message pattern -- there will be an ansno field to identify each answer. The combination of ansno and more is similar to nested messages within DIME and the MB/MF/CF bits.

The final field is the size, which has the obvious meaning of specifying the number of bytes in the message payload. While BEEP is often used for XML or other textual data, payloads can be arbitrary. Payloads are MIME objects in that they may have MIME headers describing the type and encoding of the payload. BEEP adds two rules: the default Content-Type is octet-stream, and the default transfer encoding is binary. This seems like a reasonable trade-off; binary data has no overhead, and text data has a standard, but minor overhead to describe its type.

There are a set of simple constraints among the various fields that let implementations do a large number of error-checks. Over a dozen are documented in the RFC.

BEEP initial connection follows standard IETF practice: the client (or initiator) connects to a remote port, and the server (or listener) responds with a greeting message. The greeting message includes a profile element in which the server lists its supported facilities. For example, after receiving a connection, the following greeting indicates SOAP and TLS (SSL) support:

RPY 0 0 . 0 143
Content-Type: application/beep+xml

<greeting>
  <profile uri="http://iana.org/beep/TLS"/>
  <profile uri="http://iana.org/soap"/>
</greeting>
END

From the frame header, we see that this is a reply (to the initial TCP connection), that it is the first message on the management channel, and that it is completely contained in this frame. This is a simple greeting; optional elements include codeset localization and identification of management features.

The client now acknowledges the greeting message, immediately indicating that it wants to start a SOAP channel:

RPY 0 0 . 0 51
Content-Type: application/beep+xml

<greeting/>
END
MSG 0 1 . 0 138
Content-Type: application/beep+xml

<start number='1' serverName='www.example.com'>
  <profile uri='http://iana.org/beep/soap'/>
</start>
END

The number attribute identifies the channel, and the serverName attribute identifies the "virtual server" the client wishes to work with. A client can specify multiple profiles, and the server will determine which ones it can support and send them back in its reply:

RPY 0 1 . 221 79
Content-Type: application/beep+xml

<profile uri='http://iana.org/beep/soap'/>
END

Now both parties are using a single TCP connection for management and plain text SOAP communication. BEEP is a fairly simple protocol, allowing you to build powerful communication infrastructures with modest work.

The full SOAP-over-BEEP specification is RFC 3288. It says that, once the channel has been opened, both sides enter a "boot" phase. During this phase information equivalent to the classic SOAP-over-HTTP headers is sent -- that is, the URI that would be POST'd to. It's also a placeholder for future feature negotiation and has a trivial format:

<bootmsg resource='StockQuote'>

As an optimization, BEEP allows a boot message and its reply to be piggy-backed into the channel start message, avoiding extra network round-trips (and latency). In practice, then, here is what a real start message and its reply would look like:

MSG 0 1 . 0 197
Content-Type: application/beep+xml

<start number='1' serverName='www.example.com'>
  <profile uri='http://iana.org/beep/soap'>
  <![CDATA[<bootmsg resource='StockQuote'/>]]>
  </profile>
</start>
END

RPY 0 1 . 0 112
Content-Type: application/beep+xml

<profile uri='http://iana.org/beep/soap'>
<![CDATA[<bootrpy/>]]>
</profile>
END

More from Rich Salz

Having established communication, simple attachments become trivial: send the message as multi-part MIME, as specified by the SOAP with Attachments Note. This is, in fact, the only attachment scheme currently specified in RFC 3288, which is unfortunate. SOAP should be able to take full advantage of the powerful asynchronous multiplex communications core provided by BEEP. It would seem that only two things are needed: first, a URN scheme for naming BEEP channels; second, that a SOAP message could refer to data coming in over separate channels, streaming in as it becomes available. Now, that would be cool.

I'll close with the following plea: Don, pick BEEP. which I'll explain. Back in February, Don Box pointed out that HTTP has a number of limitations for use as a web services protocol. His criticisms are right-on. It's my sincere hope that we don't get a new protocol: BEEP seems to meet all his requirements.

Are you using BEEP yet? Share your experience in our forum.
(* You must be a member of XML.com to use this feature.)
Comment on this Article

Titles Only

Newest First

Carpet Cleaning Marina Del Rey 1-877-364-5264
2009-06-30 22:40:25 carpetcare [Reply]

Carpet Cleaning Marina Del Rey 1-877-364-5264
Put an unmatched carpet cleaners to work for you. Upholstery includes Leather sofa, Mattress cleaning, Rugs Cleaning, commercial carpet cleaning, Fabric Lounge Suite Cleaning, Leather Lounge Suite Cleaning, floor Tile and Grout Cleaning, Mattress Cleaning, Wet Shmapoo Carpet Care, Water Damage Restoration, Air Duct Cleaning for offices, homes, restaurants, clubs and hotels — We will coordinate all scheduling and services involved with your cleaning with you. Los Angeles Carpet Cleaning also provides customer service and quality assurance throughout your entire cleaning so you can sleep at night knowing you are in the right hands.

Carpet Cleaning Santa Monica 1-310-925-1720
2009-06-30 22:39:19 carpetcare [Reply]

Carpet Cleaning Santa Monica 1-310-925-1720
Put an unmatched carpet cleaners to work for you. Upholstery includes Leather sofa, Mattress cleaning, Rugs Cleaning, commercial carpet cleaning, Fabric Lounge Suite Cleaning, Leather Lounge Suite Cleaning, floor Tile and Grout Cleaning, Mattress Cleaning, Wet Shmapoo Carpet Care, Water Damage Restoration, Air Duct Cleaning for offices, homes, restaurants, clubs and hotels — We will coordinate all scheduling and services involved with your cleaning with you. Los Angeles Carpet Cleaning also provides customer service and quality assurance throughout your entire cleaning so you can sleep at night knowing you are in the right hands.

Another Nit
2002-10-22 07:55:27 Ward Harold [Reply]

Each peer sends its "greeting" message as soon as the connection is established, i.e., neither peer waits to receive the other's greeting before it sends its own greeting. So both of the following traces are valid:

S: <waiting for connection>
C: <open connection>
S: RPY 0 0 . 0 110
S: Content-Type: application/beep+xml
S:
S: <greeting>
S: <profile uri='http://iana.org/beep/TLS' />
S: </greeting>
S: END
C: RPY 0 0 . 0 52
C: Content-Type: application/beep+xml
C:
C: <greeting />
C: END

and

S: <waiting for connection>
C: <open connection>
C: RPY 0 0 . 0 52
C: Content-Type: application/beep+xml
C:
C: <greeting />
C: END
S: RPY 0 0 . 0 110
S: Content-Type: application/beep+xml
S:
S: <greeting>
S: <profile uri='http://iana.org/beep/TLS' />
S: </greeting>
S: END

Also, note that the "client" is free to advertise its own profiles, i.e., it could send something like:

<greeting>
<profile uri='http://my.org/QUOTESTREAM' />
</greeting>

to indicate that it supports the QUOTESTREAM profile. The server could then start a QUOTESTREAM channel to the client if it wanted to.
- Another Nit
  2008-04-12 16:35:48 Laura.Nguyen [Reply]
  
  Love it
  
  http://www.instantz.net

Minor Correction
2002-10-21 16:21:19 Ward Harold [Reply]

Security and privacy in BEEP are per-session, not per-channel. Thus you couldn't actually do a TLS encrypted request while simultaneously streaming unencrypted images in the same session.

One of the nice things about BEEP is that it takes the "upgrade" approach to TLS usage rather than the "separate port" approach. That means that you don't have to define two ports for BEEP-based protocols; instead, a peer can "tune" an existing connection for privacy - using TLS or a SASL defined mechanism like DIGEST-MD5 - on demand at any time during the session.

are we just adding more layers?
2002-10-21 06:03:42 Taylor Cowan [Reply]

"For example, a single BEEP-over-TCP link between a browser and a web server would efficiently allow the browser to fetch a page over SSL-encrypted HTTP,..."

So we've got soap, over beep, over ssl, over http, over tcpip? Seems like what we really want is just good old rmi over tcpip. I'll just wait till the "protocol wonks" have it abstracted back out to method calls.

Taylor
- are we just adding more layers?
  2002-10-21 06:14:03 Edd Dumbill [Reply]
  
  BEEP actually cleans things up significantly and is worth taking a closer look at. It doesn't go anywhere near HTTP, and security (TLS) isn't a layer, but an option that you can switch in. SOAP is just the format of the payload you send.
  
  I'm pretty fond of BEEP, but I fear that it's the misfit kid on the block, lacking the designer clothing of its more-promoted rivals.

BEEP and HTTP
2002-10-17 16:49:49 Mark Baker [Reply]

It's funny to hear BEEP promoted over HTTP, for several reasons.

First, is because they're different types of protocols; BEEP is a presentation layer protocol, where HTTP is application layer. So any comparison (almost, see below 8-) is apples-to-oranges.

Second, both attempt to solve a similar problem; how can we build commonality between the multitude of application protocols out there, and those yet to come? BEEP's answer is to build a common lower layer on which application protocols can be built, while HTTP's answer is to be an application protocol whose semantics are so general as to be able to engulf many other protocols (as FTP and Gopher have been engulfed). The advantage of BEEP's approach is that a large number of application protocols can be built this way, say 90%. And while HTTP is perhaps only suitable for, say, 70% of them, it's principle (huge!) advantage is that it defines interoperability *between* applications. For example, Yahoo can link to both stock quotes and sports scores, while a BEEP based "stock quote" application has no meaningful way to interoperate with a BEEP based "sport scores" app. This means that there are minimal, if any, network effects generated by BEEP's common layer, whereas HTTP's general application semantics induces huge effects. That's why I don't think BEEP will see much success, and why HTTP will continue to grow in use, even beyond browsers and humans.
- BEEP and HTTP
  2002-10-21 16:02:57 Ward Harold [Reply]
  
  To the extent that it's a useful characterization BEEP actually fits better at the OSI Session Layer. HTTP could be reasonably placed at layer 6 as well:
  
  Application --> Browser
  Presentation --> HTML
  Session --> HTTP
  
  Whatever. I'm not a big fan of OSI Layer pigeon holing so let's move on to your second statement.
  
  You claim both HTTP and BEEP address the problem of building "commonality between the multitude of application protocols out there". This is absolutely false. HTTP was designed as part of a software architecture designed to meet the needs of an Internet-scale distributed hypermedia application ["Principled design of the modern Web architecture" Fielding, Taylor]. BEEP was designed to be a "generic application protocol kernel for connection-oriented, asynchronous interactions" [RFC 3080]. HTTP has been pressed into service, with mixed results, as a generic application protocol substrate but that's not how it was designed to be used.
  
  Regarding "network effects", if the stock quote and sports scores services both support SOAP (over BEEP) and publish WSDL descriptions of their interfaces then the same application can access both services. In fact, if they are hosted on the same server I can use a single session to retrieve scores and quotes. Further, I can define an additional BEEP profile that allows the quote service to stream real-time quotes back to the client - again, using a single session.
  
  In programming language terms HTTP is Basic while BEEP is C.
  - BEEP and HTTP
    2002-10-25 11:33:58 Mark Baker [Reply]
    
    HTTP is an *application* protocol. That should be your first hint that it's layer 7. 8-) The application code, like a browser, isn't part of the protocol stack because it's not a protocol; it's just code that uses the application semantics of the application protocol. I'm amazed how many people don't understand this.
    
    To refute my point about generality, you claim "HTTP was designed as part of a software architecture designed to meet the needs of an Internet-scale distributed hypermedia application", which is entirely true. But a "distributed hypermedia application" is exactly what a generalization of other forms of distributed computation looks like.
    
    As for network effects, having an app that know what the methods in one WSDL document means, and can even parse other WSDL documents, doesn't in any way help them understand what the methods in the other WSDL documents mean. There are no network effects there. On the other hand, having an app that knows what GET, PUT, POST means, can help it interact with anything that presents that same interface. This is REST's uniform interface constraint, and the reason Web services are not being deployed (and will never be deployed) in quantity on the Internet.

Don Wields a lot of power
2002-10-17 09:15:33 Doug Ransom [Reply]

Don Box wields too much power.

Don likes XML Schema, so XML Web Services are based on Schema (i.e. instead of schematron, etc.).

I support BEEP, hope the Don picks it. Also, I hope the Don convinces the .Net (dotnet) team to allow developers to bind any protocol they choose to the .net web service wsdl/dispatch mechanism.
- Don Wields a lot of power
  2008-01-31 03:32:24 sb100 [Reply]
  
  It does there's an excellent article on it here
  
  http://www.studentbunk.com/student-accommodation/London.html