XML and SAX Conformance Testing

David Brownell < db@post.harvard.edu>

September 8, 1999


What This Is

This package has a preliminary version of an XML processor conformance test driver. That driver uses the SAX 1.0 (and SAX 2.0 alpha) processor APIs, and is driven by the data in the OASIS/NIST XML conformance test suite.

It generates reports such as this one for a validating XML processor, and this one for a nonvalidating XML processor. Note that the first version of the OASIS/NIST test suite has some bugs which needed to be manually patched in order to get that output ... see the instructions later. Updates to the OASIS/NIST suite, probably coming no sooner than late August, should correct those problems.

It's built on top of my distribution of SAX2 and utilities, using JDK 1.1. Although that set of utilities uses Sun's "Technology Release 2" package, you can use the driver without needing Sun's software (which is unfortunately not open source).

This is not yet production level code. In particular, it does not have the 128,000 (or so) test cases needed to ensure that XML name characters and letters are handled correctly, and it hasn't been used for long enough to call it stable.

Why You Might Want It

Primarily, because you want to understand how conformant the various SAX parsers really are.

Alternatively, because you have an XML processor that you want to ensure is conformant with the XML specifications, and you want to demonstrate that using the OASIS/NIST test suite. This driver can be useful for folk wanting to see how to write such a driver in some other language (e.g. C, C++, PERL, Python, or some other favorite language).

This driver will also be useful for evaluating and debugging that test suite, which is currently in "first draft" status. At least some of those tests still require feedback from the W3C about what the XML specification means; a number of issues have been pending there for some time.

What Changed from Older Releases?

September 8, 1999

This release is a custom packaging of all the important driver components so it's available in one download. This is just for convenience; the primary distribution point (see below) still requires assembling some components. The OASIS tests still require a separate download.

September 3, 1999

This release removes the restriction that you use JDK 1.2; this means that you can use software such as the GNU Compiler for Java (GCJ) which is part of GCC 2.95, and generally work with the broad variety of system platforms which only support the JDK 1.1 release of Java. You can even work with Microsoft's JVM if you like.

This release also includes a workaround for a bug found in some parsers, where they don't close all of the files that they open. The fix doesn't close the files, but does ensure that the test report gets written (with enough "no more open files" messages that you'll know that the parser has such a bug).

Another change is that there is now an exception mechanism, where you can tell the driver fail certain tests without running them. This is used when parsers have certain kinds of problems, notably infinite loops triggered by selected test data.

The Makefile is set up to run tests against several more SAX parsers, and there is a file from which you may quickly see the results of your tests.

Licensing

This is distributed "as-is", with no warrantees of any kind.

The source code in this package is subject to the enclosed license, which is a variant of the "Q Public License 1.0".

Basically, it's an Open Source license which has the following restriction on any modifications you perform: if you change the interface to a Java package in this distribution, you're not allowed to keep using the original package name. Only the owner of a given package defines its interfaces; other interface changes call for a change to the package name.

If you'd to use this licence on your software, feel free. (Drop a line if you do!). When applied to your software, you may make changes you find necessary. Of course, you may not change the terms under which this software is distributed.


Updates, Bug Fixes, and All That

Other versions, or updated status, may be accessible through the current distribution point:

http://home.pacbell.net/david-b/xml/

Please report bugs to me through email, to db@post.harvard.edu; I may not be able to respond quickly, however.


Doing Developer Stuff

This should be pretty self-explanatory, and you have the source. For information about how to use this, have a look at the "Makefile", and use the source and/or the javadoc for reference.

This should work from both Win32 with the "Cygwin" shell environment ( http://sourceware.cygnus.com/cygwin), and from Red Hat Linux 6.0. It should work in both environments, with the same Makefile, using GNU make, and JDK 1.1 (or later).

This is one of three bundles of software you need, in addition to having a copy of the JDK (1.1 or later) and a SAX (or SAX2) parser. It's set up to run with them all being in subdirectories of one directory.

If you have all four of those packages (including Sun's parser) you should be able to just type "make" in the directory for this addon. See the Makefile for how to configure other SAX (and SAX2) parsers.

If you'd like to use this software with some other SAX parser(s), see the "extras" targets in the Makefile for some examples.

In case of trouble ... you've got the source, and my email address!


Patching the OASIS/NIST Suite

When you get this suite, it won't work "out of the box"; an upcoming revision should work more readily. Until that's ready, you'll need to patch the release a bit; try following these instructions.

Primarily, you just fixed all of the relative pathnames in the test database -- the URIs in the "TEST" element are relative to the file holding that element, but this first draft distribution didn't get that right (the focus was on test documentation rather than integrating that document with test operations).

You also provided a version of a missing DTD file.

After you apply those patches, there are still a few problems with that release, but none of them will be very severe. Try to run the tests. If you're interested, here are two other categories of known problems:

Of course, feel free to report any problems with these tests to the OASIS/NIST team. (Not me -- at this point, I have no role in maintaining the tests.) And keep in mind that the results from this test driver can only be as accurate as the data given to it from the test database ... so if there's a result you really don't agree with, the problem may easily reside in that database or in the XML specification itself.