XML.com: Doing HTTP Caching Right: Introducing httplib2

Doing HTTP Caching Right: Introducing httplib2
by Joe Gregorio | Pages: 1, 2, 3, 4

Introducing httplib2

Introducing httplib2, a comprehensive Python HTTP client library that supports a local private cache that understands all the caching operations we just talked about. In addition it supports many features left out of other HTTP libraries.

HTTP and HTTPS

HTTPS support is available only if the socket module was compiled with SSL support.

Keep-Alive

Supports HTTP 1.1 Keep-Alive, keeping the socket open and performing multiple requests over the same connection if possible.

Authentication

The following three types of HTTP Authentication are supported. These can be used over both HTTP and HTTPS.

Caching

The module can optionally operate with a private cache that understands the Cache-Control: header and uses both the ETag and Last-Modified cache validators.

All Methods

The module can handle any HTTP request method, not just GET and POST.

Redirects

Automatically follows 3XX redirects on GETs.

Compression

Handles both compress and gzip types of compression.

Lost Update Support

Automatically adds back ETags into PUT requests to resources we have already cached. This implements Section 3.2 of Detecting the Lost Update Problem Using Unreserved Checkout.

Unit Tested

A large and growing set of unit tests.

See the httplib2 project page for more details.

Next Time

Next time I will cover HTTP authentication, redirects, keep-alive, and compression in HTTP and how httplib2 handles them. You might also be wondering how the "big guys" handle caching. That will take a whole other article to cover.

Share your experience in our forums.
(* You must be a member of XML.com to use this feature.)
Comment on this Article

Titles Only

Newest First

Lockout Services Los Angeles Locksmith 1-310-925-1720
2009-06-30 17:38:56 carpetcare [Reply]

Lockout Services locksmith Los Angeles 1-310-925-1720
Locksmith services Los Angeles, including locks installation, doors locks repair, doors locks rekey, locks and keys products or services the best value and commitment to customers 100 satisfaction guaranteed.
On this website you can find local los angeles locksmith in your area.

Commercial Locksmithing, Specializing in: Banks Office, Apartment Building, New Homes, Condominiums, Retail Stores, Banks Industrial Facilities, Locks and Padlocks, Pharmacies Grocery Stores, Restaurants, Retail stores, Schools, Storage Warehouse.
Service includes Lockout, locked out need locksmith fast response, Deadbolt Locks repair installation Changed, Installed & Repaired, Re-keys & Master Key Systems, rekey ddors locks, High Security Locks Systems, Home Security Safes, Intercom Systems Repair & Installation, Fire Proof Panic Bars Repaired & Installed, Peephole Installation, High Security Cylinder Changed & Re-Keyed, Closed Curcuit Television CCTV, Card Access Control Systems, Panic Lock Devices, Safes,Windows,Glass Doors & Gates, Padlock, Combination Lock, Electronic Key, Magnetic Keys, Electronic Keypad and Keyless Entry, File Cabinet & Lock Picking.

locksmith Los Angeles 1-310-925-1720
2009-06-30 17:38:09 carpetcare [Reply]

locksmith Los Angeles 1-310-925-1720
Locksmith services Los Angeles, including locks installation, doors locks repair, doors locks rekey, locks and keys products or services the best value and commitment to customers 100 satisfaction guaranteed.
On this website you can find local los angeles locksmith in your area.

Commercial Locksmithing, Specializing in: Banks Office, Apartment Building, New Homes, Condominiums, Retail Stores, Banks Industrial Facilities, Locks and Padlocks, Pharmacies Grocery Stores, Restaurants, Retail stores, Schools, Storage Warehouse.
Service includes Lockout, locked out need locksmith fast response, Deadbolt Locks repair installation Changed, Installed & Repaired, Re-keys & Master Key Systems, rekey ddors locks, High Security Locks Systems, Home Security Safes, Intercom Systems Repair & Installation, Fire Proof Panic Bars Repaired & Installed, Peephole Installation, High Security Cylinder Changed & Re-Keyed, Closed Curcuit Television CCTV, Card Access Control Systems, Panic Lock Devices, Safes,Windows,Glass Doors & Gates, Padlock, Combination Lock, Electronic Key, Magnetic Keys, Electronic Keypad and Keyless Entry, File Cabinet & Lock Picking.

Cookies
2006-02-16 04:15:03 Stoune2006 [Reply]

Good job, i'm do the same in my last project manualy, but this library can be more valuable if added support for cookies

I want one
2006-02-09 07:07:16 Damian Cugley [Reply]

I've got an application I use to keep local copies of web resources up-to-date, which coincidentally I recently wrote up on my site (http://www.alleged.org.uk/pdc/2006/02/04.html). It handles ETag and Last-Modified, but does not (yet) examine Cache-Control and Expires headers. From the sounds of things I ought to be able to throw most of it away and use httplib2 instead, which will save me the trouble of finishing off those missing features... :-)

Having an HTTP client library that groks caching should be seen as an essential system service
these days. Python's runtime is not alone in offering a standard solution that leaves caching and redirection as an exercise for the caller.

HTTP Versioning
2006-02-03 16:41:55 Mark Nottingham [Reply]

One other thing that's important to understand; just because you get a message saying it's HTTP/1.0, it doesn't mean it can't contain things like Cache-Control. During the long (and still ongoing, as you note) transition to HTTP/1.1, many implmentations selectively use things like the Cache-Control header (which was easy to do), while holding back on things like chunked encoding (which is much harder to code), so they advertised themselves as capable of HTTP/1.0.

This is perfectly OK; HTTP versioning allows that. A good read is RFC2145, "Use and Interpretation of HTTP Version Numbers."

Implementations
2006-02-03 14:08:28 Mark Nottingham [Reply]

Nice article, Joe!

One thing to keep in mind is that the level of support for these mechanisms varies. While Last-Modified validators are supported by pretty much every cache, ETags aren't; last time I looked, most of the big Web proxy caches didn't use them. A similar story can be seen on the client side.

Similarly, Vary headers can act like cache-busters; in my testing (which was admittedly a while ago), not a single widely-used Web cache would store a representation with a Vary header; they treated it as a no-store.

Finallly, don't always assume your server will handle caching or other protocol functions for you; they often don't. See: http://www.mnot.net/papers/capabilities.html (also old; I really need to do another round of testing).

httplib2 looks exciting; I'll definately have a play. One of the biggest problems that I see out there is tool support; there just aren't many good server-side or client-side tools that support RESTful applications.
- Implementations
  2006-02-03 17:47:51 Joe Gregorio [Reply]
  
  Mark,
  I've tried to be as careful as I could in my implementation. For example, I will use all validators that a server sends; if it sends just an ETag I use that, if a server sends just Last-Modified I use that, if a server sends both then I will send back both when validating. Httplib2 also adds in a "Cache-Control: no-cache" header if a request is sent with "Pragma: no-cache".
  Vary headers can be a problem for intermediaries, but I don't think there is anything much I can do about that as a client library.
  - Implementations
    2006-02-03 20:56:07 Mark Nottingham [Reply]
    
    That's great; I wouldn't think you'd do a bad job; just pointing out that people shouldn't have very high expectations for other implementations.
    
    Cheers,

Correction
2006-02-02 07:12:42 JimDabell [Reply]

> For example, if a server did do content negotiation then the Content-Type: header would be different for the different types of responses, depending on the type of content negotiated. In that case the server can add a Vary: content-type header, which causes the cache to consider the Content-Type: header when caching responses from that URI.

This reads as if you think it's the varying *response* header that should be listed in the Vary header, when in actual fact it's the varying *request* header that should be listed. So Vary: Content-Type is almost certainly wrong, you want Vary: Accept if you are doing content negotiation.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.44
- Correction
  2006-02-02 07:22:52 Joe Gregorio [Reply]
  
  Jim,
  Thanks for pointing that out, I have submitted a correction for the article.