OASIS XSLT/XPath Conformance
January 24, 2001
The XSLT/XPath Conformance Technical Committee (TC) was established by OASIS in April, 2000, to assemble, document, and develop a suite of test files and an associated XSLT/XPath Conformance Test Report. The report will be suitable to use in determining a processor's adherence to testable statements identified in the W3C XSLT and XPath Recommendations. The deliverables will include the material with which to produce human- and machine-legible renditions of the report and a package of associated test files, including the following items.
- a comprehensive test suite addressing the grammar productions, the prose, and the character set issues described in the XSLT 1.0 and XPath 1.0 recommendations. The test files that are correct XSLT/XPath must be accepted by an XSLT/XPath processor, and the test files that are incorrect XSLT/XPath must be rejected by the processor (within the bounds of functionality claimed by that processor).
- A human- and machine-readable report (in XML) describing all the tests, contributor, section(s) tested, and expected results.
- Normative conformance package content, which will include descriptions of
- alternative behavior as described by the specifications;
- ambiguous situations that may affect a test outcome; and
- error handling issues
The following diagram illustrates the planned XSLT/XPath Conformance Deliverables.
Submitters will provide test files, result files, and a catalog in XML. Lotus Development Corporation, Sun Microsystems, Microsoft, and others have contributed the tests.
XSLT Used to Create the Conformance Test Package
The Committee will then create the final conformance package by processing the submitted tests with XSLT. The TC provides an exclusion instance, an assembly control instance, and documentation instances from which the package is created. An exclusion instance is applied to the catalog instance to remove any tests the committee deems invalid. Information in the exclusion instance will dictate how many submitted files actually get used in the final product.
The committee will create XML files to document the conformance process, and how people work with the files . Each of these individual files comprises a documentation instance. A stylesheet will be used to create the documentation associated with the final package. The assembly control instance points to all of the documentation pieces, the exclusion instance, and the catalog files feeding the XSLT stylesheet with the location of all the information to be pulled together into the end result. The stylesheet needs to know the list of resources to pull together and gets that from the assembly control instance.
The final conformance package includes the entire set of test cases, result files, an XML catalog document, and the XSLT stylesheet. The set of test cases provides a complete set of tests for conformance testing processors, according to the XSLT and XPath specification and any subsequent errata, for all possible configurations of conformant processors.
The result files will be used to compare the processor being tested to the desired results, using either Canonical XML or XML Infoset comparisons.
Creating the Final Reports
A user applies a rendition control instance to the entire report package containing the suite of associated test files, result files, and XML catalog. The rendition control instance is a configuration document that qualifies all of the optional behaviors and features of an XSLT processor. The committee has defined a set of discretionary items defined by the specification. Processor designers should provide answers about their implementation of these discretionary items.
Running the report with the rendition control instance produces an informative (non-normative) rendition of the report tailored for a given XSLT processor, which is then used for processor testing, with all the tests selected for the anticipated behavior as described by the rendition control instance.
The Limits of Conformance Testing
Conformance testing determines whether an implementation satisfies the requirements and behavior of a specification. Conformance testing is bound in its scope to the specification against which it is testing. The XSLT/XPath Recommendations specifies conformance criteria as requirements and behaviors that an implementation or instance must meet. If the Recommendation does not specify it, it cannot be tested for conformance. Within this framework, a user of XSLT/XPath conformance testing could focus on the XSLT/XPath processor or XSLT/XPath instances, that is, a user may
- confirm a processor returns information to an application about a putative XSLT/XPath instance as defined by the standard; or
- confirm an XSLT/XPath instance produced by an application conforms to the production of the standard.
There are at least two distinct users of the XSLT/XPath conformance package. First, a processor vendor may be testing the conformance of their implementation; second, the end user of a processor may be testing its conformance. In either case, the report specifies on which tests the processor failed to produce the desired output.
The XSLT/XPath Conformance Committee maintains a public information web page and a public archive of committee discussion. The public information page contains information on the technical committee's policy on conformance issues, the guidelines for discretionary items, public sessions, documents of note for public review, and contributing members. The committee encourages comments and will monitor and respond to the public comment list.
G. Ken Holman, Crane Softwrights Ltd, is chairperson of the committee. Voting members include Crisman Cooley, Overdomain; John Evdemon, XML Solutions; Dr. John Robert Gardner, Sun Microsystems; Dr. Kirill Gavrylyuk, Microsoft Corporation; Tony Graham, Sun Microsystems; David Marston, Lotus Development Corporation; Carmelo Montanez, NIST; and Lynda Van Vleet, Class I.Q.