StarOffice/5.2 (Linux) Adventures with Open Office and XML Matt Sergeant 2000-11-13T20:48:48 2001-01-23T13:31:07 Open Office Sun XML XPathScript en-US 173 P3DT7H23M34S Adventures with Open Office and XML Matt Sergeant At the Open Source conference in Monterey last year, Sun announced their plans to open source the current source code for Star Office, re-dubbed as OpenOffice. In October they followed up on their plans, releasing both the source code, and binaries for the current code (then OpenOffice build 605). One of the features added since Star Office 5.2 was the ability to Save As XML. Saving as XML makes OpenOffice truly open. Aside from being just open source, XML’s self documenting nature allows us to dive into the document format, without having to dive into C++. And more significantly, XML allows us to use simple, freely available tools to manipulate the documents themselves. In this article we will examine the structure of the format. We will not go into great detail, as Sun have already done so in a lengthy (~400 pages) specification, instead we will focus on using the XML to generate something of potential interest to web developers and content editors. As a sidenote it is important to say that OpenOffice is not really ready to be a usable word processor for most people for their every day work. Many components (such as printing and spell checking!) were removed in the migration to open source because these components did not belong to Sun. I expect they will be added back by the open source community as time goes by (and when Sun releases Star Office 6 I expect they will include the proprietary spell checker and print engine again). Also worth noting is that OpenOffice is relatively unstable at the moment. I experienced several crashes and times when my work just could not be loaded while working on this article. Thanks to Daniel Vogelheim of Sun for helping me out when that happened! XML Requirements Sun had a specific set of requirements when designing the XML format for OpenOffice. Having migrated the project to open source, Sun have done the right thing and also opened their development process, so all of their decisions are open to public discussion on the OpenOffice mailing lists. The short list of requirements can be found on the OpenOffice XML web site, at http://xml.openoffice.org, but we repeat the list here: Core Requirements (absolutely necessary): The file format must be capable of being used as an office programs native file format. The format must be "non-lossy" and must support (at least) the full capability of a StarOffice/OpenOffice document. The format is likely to be used for document interchange but that alone is not enough. Structured content should make use of XMLs structuring capabilities and be represented in terms of XML elements and attributes. The file format must be fully documented and have no "secret" features. OpenOffice must be the reference implementation for this file format. I have skipped the list of "Core Goals" here, but you can find them on the above web site. Sun plans for XML to become the default save format. This is not the case at the moment, with OpenOffice build 605 (note that OpenOffice build 613 is available but has many known problems), as I have to select "Save As" to export to XML, but when OpenOffice is "released" expect it to be the default save format. Packaging OpenOffice documents can be compound - that is they can contain multiple documents of different formats. Sun have examined the different ways of packaging up compound documents using XML and picked ZIP files as the format. Initially they surprised me by this choice, as I have always thought the most standard way of storing binary data in XML was to base64 encoding the data, however their decision is fully explained in detail on the page at http://xml.openoffice.org/package.html, where they talk about ZIPs indexing ability, and the importance of being able to load/save on demand. That page is worth a read to understand this. Ultimately it means that an OpenOffice file will be a Zip file, containing at least one XML file, along with other files of relevance to the document (such as images, and possibly other OpenOffice files). The format itself OK, so lets get down to the "nitty-gritty" of the XML format itself. The PDF document detailing the format is at http://xml.openoffice.org/xml_specification_draft.pdf, although this is a large document so we hope to distill some of that information here. Document Root <office:document> The document root element is <office:document> (note we will leave out namespaces URIs in this document, please see the above PDF file, or the format itself for namespace URIs, however I will note that OpenOffice appears to use namespaces to its advantage in a very clean manner, unlike some other office suites I could mention). According to the specification, this is a generic document root, meaning that all OpenOffice documents have this as a document root. A spreadsheet and a word processing file will have the same document root, allowing us to do some generic processing. Metadata <office:meta> <dc:title>Adventures with Open Office and XML</dc:title> Document metadata is perhaps one of the more interesting features of the OpenOffice XML format. OpenOffice metadata is enclosed in the <office:meta> tag at the top level of the document, immediately following the document root. Sun have chosen Dublin Core for the majority of thier metadata elements. Where Dublin Core did not have an available element for Sun to use, they have created elements in their "meta" namespace. These include the following tags: generator - the application that created this file initial-creator - the original author of the file (dc:creator is used for the person who last edited the file) creation-date - the date this file was first created (dc:date is used for the date of the last edit) keywords - you can edit the keywords in the document properties dialog A simple example of usage would be in creating a directory listing of files, and displaying their author. Here is a simple perl script that does just that: use XML::XPath;while (my $file = shift @ARGV) { next unless -f $file; eval { my $xp = XML::XPath->new(filename => $file); print $_, ": ", $xp->findvalue("//dc:creator"), "\n"; };} If we name this script dcdir, the results on a directory full of OpenOffice XML files might be: $ dcdir *.sxwtest.so.xml.sxw: Matt Sergeant This will work regardless of the type of OpenOffice file we are examining. With a little more work we can ensure that the file is an XML file of the OpenOffice format (at the moment, this script will crash when it comes across a non-XML file). See Kip Hampton’s regular column here for more details on using XML::XPath. Styles <office:styles> <style:style style:name="Source Code" style:family="paragraph" style:parent-style-name="Text body"> <style:properties fo:font-family="Courier" fo:margin-left="0.25inch" fo:font-size="11pt"/> OpenOffice formats text using text styles. This allows you to modify the styles and have the appearance of your entire document change with it. The XML format saves these styles along with the document. The list of defined styles is enclosed within the <office:styles> element. Each style, marked up with the <style:style> element, defines the following in attributes: a style name, a style family (for example a paragraph style or text (inline) style, equivalent to <div> or <span> in HTML), a parent style (because styles inherit their parent style’s attributes) and a class, which is used in the OpenOffice style dialog box to categorize styles. Within the style element itself are style properties. These are stored in the attributes of the <style:properties> empty element. As we already mentioned, the properties of a style are inherited from the ancestor styles, and only modified properties are stored (which saves space). The second interesting re-use of public XML schemas occurs here, in the use of XSL FO attributes (Formatting Objects, see last weeks XML.com for an article on XSL FO [EDD: Link]) to define style properties. Theoretically this means we should be able to do some formatting to produce an XSL FO document. Why would we want to do this when we can (or should ultimately be able to) print directly from OpenOffice though? Well I work in content management and application serving (see my previous article on AxKit), and some of my clients would like to be able to use an ordinary word processor to create content. By doing some pre-processing, and then passing the output to FOP or another XSL FO processor, we can generate PDF files automatically from content saved into the web hierarchy (note that this functionality is not available yet, but please contact me at AxKit.com if this sort of thing interests you!). It is again worth noting here that where XSL FO did not have an equivalent attribute to the internal implementation in OpenOffice, Sun have defined their own attributes in one of the OpenOffice namespaces. Automatic Styles <office:automatic-styles> <style:style style:name="P1" style:family="paragraph" style:parent-style-name="Title"> <style:properties fo:font-family="Arial" fo:font-style-name="" fo:font-size="18pt" fo:font-weight="bold" fo:text-align="end" style:justify-single-word="false" fo:margin-top="0.16inch" fo:margin-bottom="0.0835inch"/> How do you allow people to use styles, yet allow local modifications to the fonts, weight, and so on? OpenOffice does this by injecting an "Automatic style" between the text and the real style, so that rather than: "Some text" -> is of style -> "Title" We have: "Some text" -> is of auto style -> "P1" -> parent style -> "Title" So OpenOffice has a section <style:automatic-styles> following the main styles definitions. Within the main body of the document, only automatic styles are used. There are some changes going on in this area at the moment. While the above was true of OpenOffice build 605, the current CVS builds at Sun only use automatic styles when local modifications to the formatting have been made. For example, suppose we have a paragraph and we make some text bold within that paragraph, OpenOffice uses the automatic style to define that hard formatting with a span (see below for information about spans). The result might look something like this: <text:p text:style-name="Text body">Some text with a <text:span text:style-name="T1">bold</text:span> word in it.</text:p> Main Body Text <office:body> Finally we get onto the main body of the document! Of all the sections, this is probably the simplest to follow. We will address each of the major tags in turn. Unlike other sections of an OpenOffice XML file, the body text is free-form, so the following tags can appear anywhere within the <office:body> section. Headings <text:h text:style-name="P4" text:level="1">The format itself</text:h> Headings are defined using the <text:h> tag. Here we see an automatic style (build 605 export), whereas with later builds that will likely be: <text:h text:style-name="Heading 1">The format itself</text:h> Paragraphs <text:p text:style-name="P3">Some text</text:p> Paragraphs of text are defined using the <text:p> tag. We can now start to see how some of the tags in the body are similar to HTML, albeit in a different namespace. Spans <text:span text:style-name="T6">spanned text</text:span> Spans are exactly the same as spans in HTML. They delimit an inline section of a paragraph, applying alternate styling to that small section of text. Lists Lists are defined using tags of the same name as those used in DocBook to define lists. Specifically these are <text:ordered-list>, <text:unordered-list> and <text:list-item>. Graphics Vector graphics can be embedded directly into the document with OpenOffice, which is a nice feature, but you will be even more pleased to know that OpenOffice uses SVG as its native vector graphics format. And these vector graphics can just occur directly within the flow of the body document! However Daniel Vogelheim informed me that while mostly correct, the format is only "mostly SVG", because there are some things that OpenOffice can do with graphics that SVG does not define. So again they have extended the format using elements in their own namespace. Putting it all together You can find the source XML file for the original of this article (which was written in OpenOffice build 605, saved as XML, and then transformed using the techniques below) here [EDD: Please make it so!]. Now how can we put the XML generated here to good use? Well what we as XML geeks really want to see is a free WYSIWYG XML editor akin to XMetaL or Adept. And here it is. If we restrict ourselves (or our customers) to using defined styles, OpenOffice can truly be a structured XML editor - without ever knowing you are editing XML! By post processing the XML generated by OpenOffice, we can turn tags like <text:h text:style-name="P10"> into something that is significantly easier to work with from a styling point of view, like <Heading_3>. And for structured XML, we really don’t need all the font and page settings. However some of the style information may be of interest to us, for example <span> tags may point to XSL FO styles, which are almost identical to CSS styles, so these might be useful in trying to get a similar look if we translate the page to HTML. We could do this transformation with XSLT, however I prefer XPathScript [EDD: Make that a link to the old article], because it is much more natural to me, in its ability to use variables, define functions and pass parameters. Also worth noting is that the code below will only work on current releases of OpenOffice (and probably works best on files saved from build 605), due to the aforementioned changes in the automatic styles functionality. From Automatic Style to the Real Style First we need to find an XPath expression that will take us from the text’s style name (which will be an automatic style name like "P1") to the real style name. This is actually rather simple: /office:document/office:automatic-styles/style:style[@style:name="P1"]/@style:parent-style-name This finds the style:parent-style-name attribute of the automatic style. We will call this the "actual style". We can translate this "actual style" to a string we can use for an element name by removing spaces using XPath's translate() function. This will change, for example, "Heading 1" to "Heading_1". A name mapping Next we need to setup a name mapping, to allow us to translate style names to something we would prefer instead. For example, we translate "Text_body" to "para". Mappings are trivial in Perl (and hence XPathScript), we simply setup a hash: my %stylemap = ( Text_body => "para",); Adding the metadata Lets assume for now that we are only interested in Dublin Core metadata. To get this we use the simple XPath /office:document/office:meta/dc:*. Transformation Results You can access the full stylesheet here [EDD: link please]. The stylesheet can be run using the Perl module XML::XPathScript, which you can download from the CPAN, here [link to http://www.cpan.org/modules/by-module/XML]. This comes with a command line utility, xpathscript. The results of this transformation on a simple OpenOffice test document [EDD: link] is: <article> <artheader xmlns:dc="http://purl.org/dc/elements/1.1/> <dc:title>Test Example</dc:title> <dc:creator>Matt Sergeant</dc:creator> <dc:date>2000-11-13T21:00:01</dc:date> <dc:language>en-US</dc:language> </artheader> <body> <Heading_1>Test</Heading_1> <para>Here is some text</para> </body></article> Much simpler than the original, and we can easily work with this to transform to HTML using more XPathScript, or XSLT. Flat Structure One thing to note about the above is that the document format follows HTMLs style of headings followed by text. This is not my personal preference. I prefer DocBook which provides the document in a tree structure - sections are contained within a <sect1> tag, and sub-sections are contained within the parent section, rather than just occuring in the main flow of tags. A tree structure makes it easier to manipulate the document. For example, generating a table of contents is a simple recursive loop. But with the flat format in OpenOffice it becomes more complex as we have to maintain information about the current heading levels. An ideal would be to write the stylesheet above to generate a tree shaped document. So that is what I did. Doing this requires maintaining some sort of state information about the current heading level. Again this is something XPathScript makes easy, because it is just Perl. You can find a stylesheet that gets very close to generating DocBook from OpenOffice XML files here [EDD: link]. This is the stylesheet I used to in providing this article to XML.com, followed by another transformation to generate HTML. I can do this in one step using AxKit pipelines - just save the file into the web document root, and AxKit transforms it to HTML for me. OpenOffice for Content Management As I mentioned earlier, the aim, from my point of view, is to use OpenOffice as the editing component for a content management system (specifically, as an add-on for AxKit). The one thing that has thrown a spanner in the works for this in the future plans for OpenOffice is the packaging format. You cannot pass Zip files to an XML parser (except for gnome’s libxml which will accept gzipped XML files, but that is not the same thing). However because AxKit (and other XML application servers such as Cocoon) allow the XML provider to be overriden, we can even reach into those Zip files to extract the XML, before further processing with stylesheets. So what is the conclusion of this? In November at XML Dev Con in San Jose I gave a talk about the current state of XML applications for web developers in the open source world (plug: I am giving this talk again at XML Dev Con in London next month). My conclusion was that while the server side of XML processing is very much right up there (if not better than) the respective proprietary products, the client/editor side of things was a long way off. But this changes everything. Now you really can edit a richly formatted document in a WYSIWYG word processor and publish it direct to the web using XML to style it to fit your web site. That is a huge step in the right direction for the open source community. However there is more to it than my needs. Here are some other thoughts I have had that you could implement: A presentation file could be converted to Suns XML slide format, and onwards to SVG using their toolkit. Stylesheets could take other XML formats like docbook or XHTML (or the output from the transformation above) and generate OpenOffice XML format, which would allow a form of round-trip editing. The stylesheet could generate XHTML directly, rather than producing an interim format. More suggestions are welcome, and I look forward to feedback from this article. Bio Matt Sergeant is Director and CTO of AxKit.com Ltd, a Scotland based startup who produce an open source XML Application Server. AxKit.com specialise in getting content onto the web with a minimum of fuss and maximum efficiency. Matt regularly speaks at XML, Apache and Perl conferences, although he always misses his dogs when he goes away.