XML.com: XSLT Recipes for Interacting with XML Data

Published on XML.com http://www.xml.com/pub/a/2003/08/13/udell.html
See this if you're having trouble printing code examples

XSLT Recipes for Interacting with XML Data
By Jon Udell
August 13, 2003

In last month's column, "The Document is the Database", I sketched out an approach to building a web-based application backed by pure XML (and as a matter of fact, XHTML) data. I've continued to develop the idea, and this month I'll explore some of the XSLT-related recipes that have emerged.

Oracle's Sandeepan Banerjee, director of product management for Oracle Server Technologies, made a fascinating comment when I interviewed him recently. "It's possible," he said, "that developers will want to stay within an XML abstraction for all their data sources". I suppose my continuing (some might say obsessive) experimentation with XPath and XSLT is an effort to find out what that would be like.

It's true that these technologies are still somewhat primitive and rough around the edges. Some argue that we've got to leapfrog over them to XQuery or to some XML-aware programming language in order to colonize the world of XML data. But it seems to me that we can't know where we need to go until we fully understand where we are. And there's no better reality check than a practical application. In order to set the stage for this month's installment, let's review the XHTML data formats for an evolving application that gathers and displays information about conference speakers and sessions. Here's the format of the speakers file:

<?xml version="1.0"?>
<body>
<style>
.speaker { margin-bottom: 10px }
.speakerName { font-weight: bold }
.speakerTitle { font-style: italic }
</style>
<speakers>
<div class="speaker" email="jon_udell@infoworld.com">
<div class="speakerName">
Jon Udell
</div>
<div class="speakerTitle">
lead analyst, InfoWorld
</div>
<div class="speakerBio">
<span class="bio"><p>See http://udell.roninhouse.com/bio.html</p></span>
</div>
</div>
</speakers>
</body>

Figure 1. The speakers "database"

And here's the format of the sessions file:

<?xml version="1.0"?>
<body>
<style>
.session { margin-bottom: 10px }
.sessionTitle { font-weight: bold }
.sessionTrack { font-style: italic }
</style>
<sessions>
<div class="session" id="01" day="3" start="10:30" end="12:00">
<div class="sessionTitle">Grade-school CMS lessons</div>
<div class="sessionTrack">general</div>
<div class="speakerEmail">jon_udell@infoworld.com</div>
<div class="speakerEmail">paul@zope-europe.org</div>
<div class="sessionDescription">
<span class="description"><p>Paul's introduction, Jon's keynote</p></span></div>
</div>
</sessions>
</body>

Figure 2. The sessions "database"

More from Jon Udell

The Beauty of REST

Lightweight XML Search Servers, Part 2

Lightweight XML Search Servers

The Social Life of XML

Interactive Microcontent

Note that both of these formats suffer from a syndrome that might be called "div-itis". I've chosen to maintain a browser-friendly XHTML format to ensure that the raw data files are always viewable without special transformation. In fact, that was never completely true: the files contain some fictitious attributes (e.g. 'id' and 'start' in the speakers file) that aren't kosher in HTML, and don't display in the browser.

On balance, I still think it's handy to be able to browse the data file directly. But the techniques I'm exploring in the last column and this one don't require use of XHTML. Indeed, they'll work more easily when element names, rather than attributes, carry the semantics.

It's crucial to be able to visualize data. As browsers are increasingly able to apply CSS stylesheets to arbitrary XML, the XHTML constraint becomes less important. The Microsoft browser has been able to do CSS-based rendering of XML for a long time. Now Mozilla can too. Safari doesn't, yet, but I'll be surprised if it doesn't gain that feature soon. So while I'm sticking with XHTML for now, that may be a transient thing. Of more general interest are the ways in which XPath and XSLT can make XML data (of any flavor) interactive.

An XSLT-driven form generator

Here's a form, driven by both of the data sources, that's used to update information for a session:

Screen shot.
Figure 3. The session update form.

The script that builds the form is based on the Zope XSLT wrapper we explored last month. It accepts one argument: the id of a session. That id is used to create a series of form widgets of different types, drawing on different data sources:

name	type	data source(s)
sessionDay	SELECT	current session element and ZODB property
sessionTitle	INPUT	current session element
sessionTrack	SELECT	current session element and ZODB property
sessionStart	SELECT	current session element and ZODB property
sessionEnd	SELECT	current session element and ZODB property
sessionSpeakers	SELECT	current session element and related speaker elements
sessionDescription	TEXTAREA	current session element

Figure 3. Widget types and data sources for the session update form.

Two of these widgets -- sessionTitle and sessionDescription -- are easily made using XPath expressions to pick out the corresponding data from the sessions file, as shown in bold Figure 5. But the rest require more spelunking.

datafile = 'sessions.html'
id = context.REQUEST.form['id']

xsl = '''%s

<xsl:template match="//*[@id='%s']">
<form method="post" action="updateSession">
<div><input type="hidden" name="sessionId" value="{@id}"/></div>
<div>sessionDay: <select name="sessionDay">
%s
</select></div>
<div>sessionTitle: <input size="70" 
      name="sessionTitle" value="{*[@class='sessionTitle']}" /> </div>
<div>sessionTrack: <select name="sessionTrack">
%s
</select></div>
<div>sessionStart: <select name="sessionStart">
%s
</select></div>
<div>sessionEnd: <select name="sessionEnd">
%s
</select></div>
<div>sessionSpeakers: %s </div>
<div>sessionDescription: <textarea rows="10" cols="70" name="sessionDescription">
<xsl:copy-of select="*[@class='sessionDescription']/*" />
</textarea></div>
<div><input value="updateSession" type="submit"/></div>
</form>
</xsl:template>

<xsl:template match="text()" />

</xsl:stylesheet>'''

xsl = xsl % (context.xsltPreamble(), id,
             context.makeList('sessionDays', "@day"),
             context.makeList('sessionTracks', "*[@class='sessionTrack']"),
             context.timesAsSelectWidget ( context.sessionsAsDict()[id]['start'] ),
             context.timesAsSelectWidget ( context.sessionsAsDict()[id]['end'] ), 
             context.speakersAsSelectWidget(id),
             )

result = context.xslt(xsl, datafile)

result = context.adjustSelectWidget ( result )

Figure 5.

Because this is a Zope application, the SELECT widgets for sessionDay and sessionTrack can use lists of values stored as ZODB properties. That means an authorized administrator -- a non-programmer, presumably -- can edit the lists using Zope's through-the-web management interface.

It's easy to fetch a list of values from ZODB and turn them into a SELECT widget. What's trickier is to set the selected item of the list to match some XSLT-derived value. If XSLT could call out to Python extension functions, that'd be wonderful. A couple of years ago, I wrote a column in which I explored a similar scenario -- calling out to Perl from an XSLT stylesheet. The idiom looks like this:

<msxsl:script language="PerlScript" implements-prefix="user">
<![CDATA[
function myFunction()
{
   return "myValue";
}

</msxsl:script>

<xsl:value-of select="user:myFunction()"/>

Figure 6. Calling PerlScript from XSLT, circa 2001.

I thought this way of writing XSLT extension functions would have been standardized by now, but for reasons not clear to me, it hasn't been. I'm bummed, and so is XSLT guru Jeni Tennison:

To give an example, say that I need an extension function to get a directory listing, so that I can tell whether a particular document is available before I use the document() function. I can't write this my:directory-listing() extension function in XSLT, with xsl:function, because XPath doesn't give me that kind of access to the system environment. I could, however, write it in JavaScript or Java.

This function doesn't rely very much on whatever language binding I use, so the language binding isn't particularly relevant - I can use the same code for all the processors that support JavaScript, and all the processors that support Java. Even if the code did require some standard language binding, it's very feasible that all processor implementors have agreed on the same language binding for the same language -- they tend to be fairly cooperative on the whole ;)

But now I'm stuck. I want my stylesheet to work across processors, but each processor has its own way of doing the association from the stylesheet to the implementation. I have to use saxon:script for Saxon, msxsl:script for MSXML, xalan:component/xalan:script for Xalan and so on, for all the processors that I can. But this makes the stylesheet a mess, and knowing me I'll miss out some implementation that actually does support user-defined functions in JavaScript but for which I didn't know the appropriate element. [xsl-editors@w3.org]

Lacking a standard way to write XSLT extension functions, the script in Figure 5 takes another tack. It uses Python functions to communicate with the environment -- in this case, Zope -- and then interpolates dynamically-written XSLT text into the stylesheet. Consider this form fragment:

<div>sessionDay: <select name="sessionDay">
%s
</select></div>

The %s is swapped for the result of the method call context.makeList('sessionDays',"@day"), and here is the body of that method:

list = context[property]
select = ''
for i in range (len (list)):
    select += '''<option selected="{%s='%s'}">
              %s
             </option>''' % ( selector, list[i], list[i])
return select

Figure 7. The makeList() method.

It's a Zope PythonScript, so the parameters -- property and selector -- are specified using the Zope management interface. The raw Python declaration would look like this:

def makeList ( property, selector ):
    ....

We can test the function from the URL-line, for any Zope folder containing a sessionDays property and an instance of a sessions file, like so:

http://SERVER/FOLDER/makeList?property=sessionDays&selector=@day

Here's the resulting XSLT fragment:

<option selected="{@day='1'}">1</option>
<option selected="{@day='2'}">2</option>
<option selected="{@day='3'}">3</option>
<option selected="{@day='4'}">4</option>
<option selected="{@day='5'}">5</option>

Figure 8. Template for an XSLT-driven SELECT widget.

When this XSLT fragment is executed in the context of Figure 5, a single session will be current, and only one of the expressions in Figure 8 will return true. For example,

<option selected="false">1</option>
<option selected="false">2</option>
<option selected="true">3</option>
<option selected="false">4</option>
<option selected="false">5</option>

Figure 9. Fragment from Figure 8 instantiated in the context of a current session.

That's close, but there's still some minor surgery required to make this work in the browser. The adjustSelectWidget() method shown in Figure 5 takes care of that, along with a few related warts:

select = select.replace('selected="true"','selected')
select = select.replace('selected="false"','')
select = select.replace('multiple="yes"','multiple')
return select

Figure 10. Cleaning up the XSLT-generated SELECT widget.

Procedural joins of XML data

The sessionSpeakers widget requires information from both data sources. Speakers assigned to a session appear in the session file as email addresses (see Figure 2), but the corresponding names appear in the speakers file.

The first thing to note about this arrangement is that, just like in the old dBase days, there's no referential integrity other than that which you enforce programmatically. However, the RDBMS vendors are catching up to this requirement. In Oracle 9i release 2, for example, it's possible to code triggers that keep multiple XML fragments consistent in the same way they can keep multiple SQL tables consistent.

Oracle and DB2 also embrace SQL/XML, the XML extensions in the proposed ISO/ANSI SQL:2003 (or, since we are running out of time this year, SQL:200n) standard. And everyone's hot on the trail of XQuery implementations, another way to join heterogeneously across SQL and XML data. So do-it-yourself programmatic joins across XML documents won't always be necessary. But for simple problems and for small quantities of data, it's handy to be able to do it.

To create the sessionSpeakers widget seen in Figure 3, I needed a way to get speaker and session data out of their respective XML files and into Python data structures where I could manipulate it. I tried several approaches. The first was Python's xml.dom.minidom. Figure 11 shows one way to use it to convert the speakers file to a Python dictionary.

from xml.dom import minidom

doc = minidom.parse('speakers.html')

divs = doc.getElementsByTagName('div')

dict = {}

for e in divs:

    if e.hasAttribute('email'):
        email = e.getAttribute('email')
        dict[email] = {}

    if e.hasAttribute('class'):
        attr = e.getAttribute('class')
        if attr in ['speakerName','speakerTitle']:
            for n in e.childNodes:
                val = n.nodeValue
                if val != '\n':
                    dict[email][attr] = val
print dict

Figure 11. Converting XML to Python using xml.dom.minidom.

Next, I tried the same thing using Aaron Swartz's xmltramp:

import xmltramp
toplist = xmltramp.load('speakers.html')
speakers = toplist[1]
dict = {}
for i in range(len(speakers)):
    speaker = speakers[i]
    email = speaker._attrs['email']
    speakerDict = {}
    speakerDict['name'] = s[0][0]
    speakerDict['title'] = s[1][0]
    dict[email] = speakerDict
print dict

Figure 12. Converting XML to Python using xmltramp.

Both approaches are workable, but in each case I found it hard to match up the code to the structure of the document. Here's an alternative XSLT-based approach: a PythonScript called speakersAsDict that's similar to the sessionsAsDict method seen in Figure 5.

datafile = 'speakers.html'
xsl = '''%s

<xsl:template match="//speakers">
{
<xsl:apply-templates select="*[@class='speaker']" />
}
</xsl:template>

<xsl:template match="*[@class='speaker']">
|||<xsl:value-of select="@email"/>||| : 
  { 
  'name'  : |||<xsl:value-of select="*[@class='speakerName']" />|||,
  'title' : |||<xsl:value-of select="*[@class='speakerTitle']" />|||
  }, 
</xsl:template>

<xsl:template match="text()" />

</xsl:stylesheet>'''

xsl = xsl % ( context.xsltPreamble() )

result = context.xslt(xsl, datafile)

result = result.replace('<?xml version="1.0"?>\n','')
result = result.replace ('|||', '"')

return context.safeEval(result)

Figure 13. Converting XML to Python using XSLT.

This strategy produces a textual representation of a Python dictionary, and then evaluates it to yield a reference to the dictionary. Clearly that's a security concern. I'm addressing it in two ways. First, the safeEval method is a PythonScript wrapper for an Zope external method that uses a neutered eval like so:

eval(s, {'__builtins__': {}})

As a result, safeEval doesn't get to call any functions -- in particular, it doesn't get to call the globals() method that gives access to OS-level functions.

The next line of defense is Zope's folder-level security. I'm only planning to locate safeEval in a restricted, access-controlled area of the site. Together these strategies add up to reasonable, if not iron-cladprotection. (It's always tricky to strike the right balance between the power of dynamic languages and the danger they can lead to. I'll be curious to see how Zope's RestrictedPython implementation, which is used to sandbox PythonScripts and Zope Page Templates, may evolve in the forthcoming Zope 3.)

All that said, I like the XSTL-oriented way of extracting Python data because

You can precisely select a subset of the data.
It's easier to visualize the structure of the document.
It's not language-dependent -- you could for example emit a Perl or Java hashtable instead.
It's not sensitive to the sequencing of elements in the document.

The techniques I've been exploring for the past few months are, admittedly, an unorthodox approach to building Web applications. The gymnastics required can be strenuous, and some of the integration is less than seamless. But the result is useful, and along the way I've deepened my understanding of XPath and XSLT.

Is it really advisable, or even possible, to make XML the primary abstraction for managing data? I'm still not sure, but I continue to think it's a strategy worth exploring.