XForms, XML Schema, and ROX
"If I have an XML schema, is there any way that I can work with that schema to build forms for populating instances of that schema?"
Over the years, I've seen a number of variations on this same question, and generally for a pretty good reason. It takes a lot of work to create a schema in the first place, but when you're done, what you end up with, in general, is something that seems like it should be good to generate something; you have data type information, constraint information, enumerations, and enough other pieces that it would seem that making forms from them should be a cake walk.
However, the process is generally fraught with more land mines than you might expect. While the schema is generally a pretty good blueprint from which to start, it faces a number of problems that tend to limit the ability to build forms of any sort:
- Labels and localization: it is generally possible to use the names of elements as labels in a pinch, but such labels aren't necessarily ideal for professional looking forms. Moreover, shifting out of the language that the schema was originally written in can usually destroy any hope for simple text transformation functions.
- Non-schema interface elements: even the most basic form has a need for more than just fields of text. There needs to be a way of introducing additional content, from descriptions to containment boxes to the form in a consistent manner.
- Binding fields: you need to devise a consistent mechanism for tying in each element to an interface control, such that changes to the value of one controls the appearance of the other and vice versa.
- Presentation choices: a number of schema elements may have multiple ways of being expressed to retrieve the same data or they may be some form of a container that could be represented in different ways. Because of this choice, it's not always easy to determine from a single element declaration in a schema how best to express this content.
- Reordering and filtering: often, the order of elements in a schema may be less than ideal from an interface standpoint or elements may be displayed that would better be hidden in practice (such as record ID numbers).
Having laid out these objections, its also worth bringing up one point that I've heard expressed more than once: that a schema-generated output makes for uninteresting or awkward user interfaces. As it turns out, this particular argument is something of a red herring. Typically, schemas tend to be designed hierarchically, with the most significant portions of the schema appearing near the top of the schema.
Curiously enough, this structure is actually pretty typical of most form content, with the order of items appearing in a form usually correlating fairly closely with their significance. If you have a mechanism for providing some kind of exception handling when this is not the case in your forms generators, you can usually get the schema to do the heavy lifting for you.
Working with ROX
I've long been intrigued by XForms, in part because they represent a pure declarative approach to creating interactive forms, but also because they work against a common XML-based data model or, to put it other terms, an XML schema plus its associated instance data. Because of that, the idea that you can create an XForms document from a schema is not as absurd as it may seem on first reading, because that form is in fact working against the schema anyway and, as such, probably reflects the inner schema more readily than a more raw HTML format might.
Because of this, I set out some time ago to create a set of transformations that would perform such a mapping between schema and XForms content, and after a few false starts, settled upon the following architecture:
- Schema design: start with a schema that is Element Nominal (explained shortly), that explicitly defines cardinality (the minimum and maximum number of child elements in a parent element) and that uses an explicitly declared namespace. In general, such schemas can be transformed easily from Element Referential schemas.
- Base metastructure: using an XSLT Transformation, analyze the relationships between elements and assign them into metastructure categories (such as Category Sets, Collections, Records, Bags, etc., explained shortly). The result of this is a base metastructure document.
- Metastructure overlays: an overlay uses a similar schematic notation to define exceptions to the Base Metastructure. Such overlays work either by overriding the default characteristics of an element in the base metastructure (such as the order of its children) or by adding new elements that define additional behaviors or presentations (such as adding a style element for a given element). Overlays inherit from other internally referenced overlays, so that you can have one overlay that provides a French language version on top of an English language one, another that establishes XBL bindings on elements of certain types and yet another that contains help information, and these are then evaluated from the outermost overlay (as specified by a pointer in a configuration document) to the innermost (so that overlay two gets added into overlay one, and the result then gets added into the base metastructure).
- Editor and report generators: once the final metastructure is created, it in turn generates three things: an XForms Editor document, an XForms Report document, and an XML Default Instance used to populate newly created records. It also registers the newly created items in a configuration file (conf.xml) in an associated XML Database (in this particular case, the eXist XML database).
- XQuery (eXist) publishing system: the server-side code for handling the actual publishing of the various components uses the eXist XML database and eXist's XQuery system as its primary engine.
- Syndication feeds: one of the key aspects of this system is to recognize that the objects created should be usable by as large an audience as possible. For this reason, the objects so created in this process will be accessible via syndication feeds (so that you can get a list of most recent invoices, bibliographic entries, medical requests, etc., readable by standard news readers). This system also includes hooks for an additional summarization transformation that performs a summary on a given entry, with a default being created as part of the report generators. System managers can add views of this summary data as additional XSLT transformations, passed to the client as part of the syndication feed.
- Searches: similarly, structured data establishes a fairly high bar in terms of being able to search against a given database. The system defines a default OpenSearch template for performing base searches on objects of a given type, with the option of adding additional custom searches with more specific parameters for querying the system in more novel ways (and generating the output accordingly). The combination of Custom Search and Custom View makes it possible to customize the output in a wide variety of ways while at the same time keeping the base system simple to use.
- Validators and work-flow system: users can save XML files built from within the XForms without validating them (in essence working with draft copies), but once it becomes necessary to add it into the system, the XForms would run against server side validators that use XSD and Schematron (client-side validators are added in as part of the overlay layer). Once a document is validated, it then triggers on the server side a work-flow event that sets the flags necessary to move it to new syndication queues.
Now, if you've been following all of that, one thing that you may be thinking is that this system has suddenly jumped from simply generating XForms from schemas and is now essentially running as a full blown application. Yup, it is. Realistically, the XForms act essentially as mechanisms for creating valid and consistent XML, but without some kind of server-side component the created XForms are fairly worthless. With them, on the other hand, this system is rather disturbingly powerful.
In previous articles and blogs, I've mentioned an open source project that I've been working on, called the REST Objectified XML System, or ROX System. What I've outlined here is essentially the core of that system. The idea behind this is to marry the concepts inherent in data editing (which typically have been built one field at a time, usually with an implicitly defined data model) with document editing CMS systems, which is the basis for the full name.
As an aside, the acronym preceded the formal title of the project. Those of us working on this project, including David Baker of eVision, Mark Birbeck of X-Port, Dan McCreary and myself, happened to like the acronym better than any other we could come up with. What the acronym means, however, has shifted back and forth for a while, and includes such titles as REST on XML, REST Oriented XML, and Really Out-there XML, which is my own favorite, if somewhat irreverent, name for what we're trying to do. However, the jury's still out, if you can think of a more suitable meaning, by all means, let me know!
One of the key issues I faced while designing this system was attempting to deal with schematic information at an abstract level, what was essentially the meta-schema (or design patterns) of a given object. Such a meta-schema needed to be semantically neutral, because it needed to apply to any schema, not just a single given schema. Moreover, it couldn't be dependent on a particular naming convention, as different people and organizations tend to use different conventions, often for very good organizational reasons. Thus began my search for the metastructure of a given schema.
One of the things that seemed to emerge in this process was that there were a number of criteria that determined which pattern a given element fell into:
- Arity: the minOccurs and maxOccurs values for a given element.
- Containment: for a given container node, the number (and arities) of each child element.
- Identity and uniqueness: whether a given node had some form of identifier that uniquely differentiated that node from other nodes.
- Choice: whether one or an other element was considered to be in force at any given time.
- Atomic type: for simple types, this is the XSD primitive or atomic types such as string, number, date, and so forth.
These criteria open up a fairly wide possible matrix of choices (some of which are of course nonsensical), but in general what emerged from this were the following patterns:
- Category binder: an element that contains a set of unique (min=0/max=1) subcontainers. Category binders are often represented as tabs or menus.
- Collection: a container element that holds a set of zero or more (min=0/max=unbounded) schematically identical elements (known as records.
- Record: a container element that is both unique (it has a unique identifier in the context of the collection) and has either property elements, collections, category binders, or bags as children.
- Bag: an element that contains either property elements or other bags and serves primarily as a grouping mechanism. A good example of a bag is a
<name>element that may contain
- Property: an element that only contains a scalar value; in a given record or bag, such a property is typically unique (occurs once and only once), though it may not be unique within the entire schema.
- Switch: a container element holding two or more container elements, only one of which can be relevant at any given time. For instance, a switch element may hold an American address block, a Canadian address block, and a European address block, but only one of those blocks can be defined in any given schema.
Additionally, properties can have different expressions based on the schema type of the element; strings, dates, numbers, and so forth each may bind to different types of controls, while enumerants bind to selected enumerated properties, with the arity of the base type (one or more than one allowed value) defining whether the property displays as a single or multi-item selection control. This a given meta-structure binding may include both a pattern (Property) and a type (Enumerated Single).