Hacking the XML in Your TiVo
February 15, 2006
When I was a kid, our television set picked up just a few broadcast channels over the wire aerial mounted on our roof. This offered us a limited choice of what to watch, so we saw a lot of I Love Lucy reruns. Now that we have hundreds of channels over the satellite dish, libraries of DVDs and videotapes, Netflix, a great local video store, increasing access to video over the Web, and especially a TiVo, what do we save up on the TiVo to watch together as a family besides The Simpsons and Monty Python's Flying Circus? Old I Love Lucy episodes.
We don't watch every recorded episode. The TiVo offers various options for finding
episode titles and plot summaries of what it recorded, so that we can find Simpsons
episodes we've never seen and alert our kids to particularly classic Monty Python
and I Love Lucy episodes. The simplest way to see what you have is to use your TiVo
remote to bring up the Now Playing menu. Things get more interesting when your
TiVo—which is ultimately a Linux box with a big hard disk and a timer-triggered
channel changer—has a wireless network adapter that connects to your home network.
HTTP server built into the TiVo lets you browse web page versions of TiVo metadata
browser on the same network, and a RESTful API lets you pull XML versions of the same
information. With some short stylesheets and a little help from the free
utility, I wrote a simple application that puts onto my weblog a "TiVoRoll" of what shows our TiVo's
been recording, and another app that gives me an Atom feed showing which episodes
Before I describe how to do this, I'm making two assumptions:
Your TiVo is on your home network. As the TiVo documentation explains, you just plug a network adapter into the USB port on the TiVo's back panel and run a setup routine. Make sure that you get your network adapter from someplace where you can return it if necessary, because the TiVo can be very picky about these, sometimes accepting and rejecting different adapters of the same make and model due to slight differences in the version number. I found the website tivocards.com to be very helpful.
You're using one of the Series 2 TiVos that have been available for over four years now. For information on great hacks that you can do with both Series 2 and Series 1 TiVos, see the O'Reilly book TiVo Hacks.
To retrieve data from the HTTP server in your TiVo, you first need to find out its IP address. From the TiVo main menu, pick Messages & Settings, Settings, and then Phone & Network to display the Network Settings screen, which will have your IP address. This number was dynamically assigned and may change if you reboot some part of your home network, so don't hardwire this IP address into your code when creating an application around it. The same TiVo screen also shows your MAC ID (formerly the MAK, for Media Access Key), which will be your password to gain access to the TiVo's metadata, so note that as well.
Browsing the TiVo HTTP Server
Let's say that your TiVo's IP address is 192.168.2.103. To see its web server's home page, point a browser on one of your home network's machines to https://192.168.2.103. (Remember to make it https and not http, which will give you a screen that tells you nothing about what you have on your TiVo.) You may get a dialog box saying "Web Site Certified by an Unknown Authority" and one saying "Domain Name Mismatch," and then you'll be prompted for a username and password. Your username is "tivo", and your password is the MAC ID that you copied from your TV screen when you got the TiVo's IP address. The default web page redirects your browser to https://192.168.2.103/nowplaying/index.html, which summarizes the shows you have recorded:
Clicking "folder" on the right lists individual episodes of a show that you have recorded, and clicking "classic" at the bottom lists all the individual episodes on the TiVo hard disk.
The third method for finding out about recorded episodes, besides the web page interface and using the remote to display the Now Playing list on your television, is the most interesting: with a URL full of parameters that tell the TiVo's HTTP server to give you an XML version of the information. Pointing your browser to the URL
(with your own TiVo's IP address substituted if necessary) displays something like this:
The parameters in this URL are documented in the TiVo Home Media Option Music and Photos Server
Protocol Specification, an Acrobat file. The free Home Media Option lets your TiVo
display pictures and play music stored on another computer on your home network. Its
documentation mostly describes how to retrieve XML about your picture and music files
well as the files themselves, as we'll see), but by specifying the appropriate
Container parameter, as shown in the URL above, you get Now Playing metadata.
The protocol specification describes other commands besides
none looked particularly interesting to me. For example,
QueryServer tells you
the version of the TiVo server and nothing else.
Command-line Retrieval of TiVo XML
The XML retrieved with this interface doesn't do us much good in a browser. To do something interesting with it, we want to pull the XML down from the server and pass it to an XSLT stylesheet that transforms and stores the result in a file that can be passed to the next step in our application. Remember, though, about the security dialog boxes: if you used the browser to retrieve the XML version without trying the HTML example first, you had to go through the same security dialog boxes, and a non-browser tool for retrieving the XML must negotiate the same issues.
wget tool, which is available for Windows and included with most Linux
distributions, makes this easy. In the words of its home page, it's a "free software
package for retrieving files using HTTP, HTTPS and FTP, the most widely used Internet
protocols. It is a non-interactive command-line tool, so it may easily be called from
cron jobs, terminals without X-Windows support, etc." (Fans of
curl will find that it can pull XML out of
their TiVos as well.)
Passing a URL as a single parameter to
wget pulls down the named file to your
hard disk. For example, entering the following code downloads Google's current
index.html file to your current directory:
Dozens of optional command-line switches let you customize
wget's behavior for
more complex situations such as negotiating the security checks that your TiVo puts
way of a straight HTTP request. The
--http-password parameters are self-explanatory. We'll supply the word "tivo"
and the MAC ID for these values, as we did when we pointed a browser at the TiVo box
asked for a username and password. To get past the "Web Site Certified by an Unknown
Authority" warning, we'll add the
--no-check-certificate switch. The
-O switch is handy for setting a specific output filename, because while
wget had an easy enough time figuring out that Google's homepage was in a
file named index.html, the filename that it infers from a REST URL full of
ampersands and percent signs can get messy. The following
wget command line
works identically under Windows and Linux:
wget --no-check-certificate --http-user=tivo --http-password=your-MAK-here -O nowplaying.xml "https://192.168.2.103/TiVoConnect? Command=QueryContainer&Container=%2FNowPlaying&Recurse=Yes"
(This single command is split into three lines to fit here. When rejoining the pieces,
sure to put a space after your
http-password value but none after the question
TiVoConnect.) This command downloads the list of recorded episodes
and stores them in the nowplaying.xml file. The quotes around the URL may not be
necessary, but they make things a little less confusing for the Linux shell, where
ampersands indicate that a job should run in the background.
Recurse=Yes parameter in the URL tells the TiVoConnect application to go
through the entire tree of information in the
NowPlaying container, which is
why it gets information about all the individual recorded episodes. With
Recurse set to
No, it lists only those shows whose episodes the
TiVo records, as shown in the first illustration above. This is what we want for our
wget --no-check-certificate --http-user=tivo --http-password=your-MAK-here -O tivoroll.xml "https://192.168.2.103/TiVoConnect? Command=QueryContainer&Container=%2FNowPlaying&Recurse=No"
wget has pulled down an XML version of the information we want, we can
use XSLT to convert it to the format we want. For the TiVoRoll added to my weblog,
stylesheet is pretty short and simple:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:t="http://www.tivo.com/developer/calypso-protocol-1.5/" xmlns="http://www.w3.org/1999/xhtml" exclude-result-prefixes="t" version="1.0"> <xsl:output omit-xml-declaration="yes"/> <xsl:template match="t:TiVoContainer"> <div class="tivoroll"> <p> <xsl:apply-templates/> </p> </div> </xsl:template> <xsl:template match="t:Item/t:Details/t:Title"> <xsl:value-of select="."/><br/> </xsl:template> <xsl:template match="text()"/> </xsl:stylesheet>
All text nodes are suppressed because the second template rule uses an
xsl:value-of instruction to get the text that it needs: the name of each show
Note that everything in the TiVo source XML file is in the
http://www.tivo.com/developer/calypso-protocol-1.5/ namespace declared at the
beginning of that file, so the stylesheet must declare that before referencing specific
elements in the source document.
Using the XML that I pulled off of my TiVo just now as a source document, the stylesheet above created the following result:
<div xmlns="http://www.w3.org/1999/xhtml" class="tivoroll"> <p>Monty Python's Flying Circus <br/>I Love Lucy <br/>Daria <br/>The Colbert Report <br/>The Daily Show With Jon Stewart <br/>The Simpsons <br/>NHL Hockey <br/> </p> </div>
Once I FTP this to the appropriate directory on my weblog's server, a Movable Type
$MTInclude instruction incorporates it into the right-hand column of
my weblog's index page.
Let's review the steps by looking at the Linux shell script that drives it. (To run
yourself, first substitute your appropriate values for
your-MAK-here in the shell script. This zip file stores all the
scripts and stylesheets used to make this happen. It includes Windows batch file equivalents
of the tivoroll.sh and nowplaying.sh files.)
#! /bin/sh export TiVoIP=your-TiVoIP-here export MAK=your-MAK-here # pull the TiVoRoll XML down from the TiVo wget --no-check-certificate --http-user=tivo --http-password=$MAK \ -O /tmp/tivoroll.xml "https://$TiVoIP/TiVoConnect?Command=QueryContainer\ &Container=%2FNowPlaying&Recurse=No" # Convert it to HTML markup xsltproc -o /tmp/tivoroll.html tivoroll2p.xsl /tmp/tivoroll.xml # FTP the result to a public web server ftp -n snee.com < tivoroll.l.ftp
wget utility pulls down the XML we want, the
xsltproc creates HTML from the XML, and
ftp uses a
script to automate the process of putting the HTML into the right directory on the
snee.com server where my weblog resides.
My application that creates an Atom feed of the Now Playing list of saved episodes
almost identical to the one that creates a TiVoRoll for my weblog. The driver shell
Recurse=Yes version of the URL to
wget to get the details
about the stored episodes, then calls
xsltproc with an XSLT stylesheet that
creates an Atom 1.0 file, and it finishes by using an FTP script that puts the result
XSLT pass into a different directory on the web server (http://www.snee.com/rss/nowplaying.atom, in case you're interested in seeing what's on
our TiVo). A
cron job on
my home Linux box runs both scripts before I get up each morning.
Querying the TiVo Desktop
The TiVo Desktop is a Windows program that, when run on your home network, lets you use your TiVo to display pictures and play music on your TV that it retrieves from the computer running the TiVo Desktop. Computers on your network can talk to the TiVo Desktop using URLs like those shown above to find out what pictures and music are available.
Because you'll be sending your queries to the Windows machine and not to the TiVo,
first find out its IP address. The
ipconfig command-line utility that comes
with Windows shows you this and related information. For mine, it was 192.168.2.102,
following URL retrieved XML that listed the high-level containers:
(Note that the URL scheme prefix is http and not https and that a port number of 8080 is included in the URL.) Here is the XML that it retrieved:
<?xml version="1.0" encoding="ISO-8859-1" ?> <TiVoContainer> <Details> <Title>BLACKDELL</Title> <ContentType>x-container/tivo-server</ContentType> <SourceFormat>x-container/folder</SourceFormat> <TotalItems>2</TotalItems> </Details> <Item> <Details> <Title>Bob's Music on BLACKDELL</Title> <ContentType>x-container/tivo-music</ContentType> <SourceFormat>x-container/folder</SourceFormat> <LastChangeDate>0x43BB4121</LastChangeDate> </Details> <Links> <Content> <Url>/TiVoConnect?Command=QueryContainer&Container=%2FTivoMusic</Url> <ContentType>x-container/tivo-music</ContentType> </Content> </Links> </Item> <Item> <Details> <Title>Bob's Photos on BLACKDELL</Title> <ContentType>x-container/tivo-photos</ContentType> <SourceFormat>x-container/folder</SourceFormat> <LastChangeDate>0x43D40340</LastChangeDate> </Details> <Links> <Content> <Url>/TiVoConnect?Command=QueryContainer&Container=%2FTivoPhotos</Url> <ContentType>x-container/tivo-photos</ContentType> </Content> </Links> </Item> <ItemStart>0</ItemStart> <ItemCount>2</ItemCount> </TiVoContainer> <!-- Copyright (c) 2003-2005 TiVo Inc. All rights reserved.-->
I had given that computer the name of blackdell to distinguish it from our older, off-white Dell that is now running Ubuntu Linux, which is the machine I used to test these queries. Because I had shared the My Music and My Photos folders from my "Bob" account on the blackdell computer, the two containers listed by the URL above had titles of "Bob's Music on BLACKDELL" and "Bob's Photos on BLACKDELL." Below these titles you can see relative URLs that point to the contents of these containers. The following URL (split into two lines here) showed that the TivoPhotos container had containers named My Photos.lnk and My Pictures.lnk:
Those have their own URLs too, and using those and the same technique you can drill down to your actual listings of photos. These image files will have URLs that let you retrieve them; for example, the following URL retrieved I102_0248.JPG and displayed it on the Ubuntu machine's screen:
The API documention lists additional parameters to control the display of the image. For example, this URL displays the I102_0248.JPG image rotated 90 degrees:
The ability to see directory listings and to retrieve JPEG and MP3 files from one home network machine to another is not in itself exciting. It's ironic that the documentation explaining how do to this with the TiVo Home Media Option API holds the key to querying the TiVo box itself, which is much more interesting. The fact that we can do this with a REST API makes it simple to play with, and makes it that much easier to script the integration of your personal TiVo data into the other metadata of your life.