XML Processor Conformance Report:
fr.loria.xml.sax.SAXDriver

This document is the output of an XML test harness. It reports on the conformance of the following XML 1.0 processor configuration, as reported through the Java version of the SAX (Simple API for XML) parser interface. For SAX 2.0 (alpha) processors, the processing mode and entity handling values could be non-default values, if such a feature is supported by that processor. For SAX 1.0 processors, those values were as reported to the test harness by its operator.

The test harness and execution environment was:

An summary of test results follows. To know the actual test status, someone must examine the result of each passed negative test to make sure it failed for the right reason. That examination may cause the counts of failed tests to increase, changing a provisional "conforms" status to a "does not conform".

Sections of this report are: Explanation of Tables; Positive Tests, cases where this processor should report no errors; Negative Tests, documents for which this processor must report the known errors; and Informative Tests, documents with errors which processors are not required to report.

NOTE: The OASIS/NIST test suite is currently in draft state, and can't actually be used without modifications to the configuration file, which is used both to generate the test documentation published at the OASIS/NIST site and to operate this test harness. Accordingly, treat these results as preliminary.

Explanation of Tables

Sections presenting test results are composed largely of tables, with explanations focussing on exactly what those tables indicate. Diagnostics for failed tests are presented in italics, with a cherry colored background, to highlight the result. Diagnostics for succesful tests should as a rule only exist for negative tests. Diagnostics with parenthesized comments indicate the reporting path:

(fatal)

The diagnostic was reported as a fatal error. Such errors are primarily well-formedness errors, such as the violation of XML 1.0 syntax rules or of well formedness constraints.

(error)

The diagnostic was reported as a recoverable error. Such errors primarily used to report validation errors, which are all violations of validity constraints, but some other errors are also classed as nonfatal.

(warning)

The diagnostic was reported as a warning; warnings are purely informative and may be emitted in a number of cases identified by the XML 1.0 specification (as well as in other cases).

(thrown classname) ... abnormal

The named exception was directly thrown. If the exception is a SAXException (or a subclass thereof) this suggests an error in the parser (violating the SAX API specification) since it should normally have used the SAX ErrorHandler instead.

(odd classname) ... abnormal

After the identified exception was reported through the ErrorHandler, an exception of the named class was thrown directly. This suggests an error in the parser, since the parser either failed to continue after an error (or warning) which is required to be continuable, or else it did not pass the exception thrown by the application back to the application.

(EXCEPTION - DIRECTED FAILURE) ... abnormal

This test case was explicitly failed by the test operator; the test was not run. This may be done in the case of parsers with severe bugs which completely prevented handling the test case, typically because the parser seems to "hang" by entering an infinite loop.

In all cases, negative tests that appear to pass (diagnostics presented with a white background) must be individually examined in the report below. The diagnostic provided by the processor must correspond to the description of the test provided; if the processor does not report the matching error, the seeming "pass" is in fact an error of a type the test harness could not detect or report. That error is either a conformance bug, or an error in the diagnostic being produced, or sometimes both.

Nonvalidating processors may skip some tests if the tests require processing a class of external entities (general, parameter, or both) which that processor is known not to handle. If processor handling of entities is not known, all such tests are skipped, in order to prevent misreporting.

Note also that some SAX processors (or drivers for non-SAX processors) appear to have a bug wherein they don't close all of the files which they open. When running this test harness with large numbers of tests, failures at the end of this report could reflect such bugs in SAX conformance more than bugs in XML processor conformance.

Positive Tests

All conformant XML 1.0 processors must accept "valid" input documents without reporting any errors, and moreover must report the correct output data to the application when processing those documents. Nonvalidating processors (such as this one) must also accept "invalid" input documents without reporting any errors. These are called "Positive Tests" because they ensure that the processor just "does the right thing" without reporting any problems.

In the interest of brevity, the only tests listed here are those which produce diagnostics of some kind, such as test failures. In some cases, warnings may be reported when processing these documents, but these do not indicate failures.

No interpretation of these results is necessary; every "error" or "fatal" message presented here is an XML conformance failure. Maintainers of an XML processor will generally want to fix their software so that it conforms fully to the XML specification.

Valid Documents

All XML processors must accept all valid documents. This group of tests must accordingly produce no test failures.

Output Tests

The XML specification places requirements on the data which is reported by XML processors to applications. This data flows through the SAX API (with one minor omission: reporting external entities which are not included). These output tests verify conformance with the specification by recording that data and comparing it with what is required for conformance with the XML 1.0 specification.

Note that output tests automatically fail in cases where the processor failed to parse the (valid) input document used to generate the output data.

Invalid Documents

As noted above, nonvalidating processors must accept all documents which are well formed, but invalid. This same behavior would be delivered by a validating processor, if the application chose to continue processing after receiving each report of a validity error, and not report such validity errors. (These tests are run as "negative" tests for validating processors, since in those cases it is important that the correct validity errors be reported and that they be fully continuable.)

Negative Tests

All conformant XML 1.0 processors must reject documents which are not well-formed. In addition, validating processors must report the validity errors for invalid documents. These are called Negative Tests because the test is intended to establish that errors are reported when they should be.

Moreover, the processor must both fail for the appropriate reason (given by the parser diagnostic) and must report an error at the right level ("error" or "fatal"). If both criteria were not considered, a processor which failed frequently (such as by failing to parse any document at all) would appear to pass a large number of conformance tests Unfortunately, the test driver can only tell whether the error was reported at the right level. It can't determine whether the processor failed for the right reason.

That's where a person to interpret these test results is critical. Such a person analyses the diagnostics, reported here, for negative tests not already known to be failures (for not reporting an error, or reporting one at the wrong level). If the diagnostic reported for such tests doesn't match the failure from the test description, there is an error in the diagnostic or in the processor's XML conformance (or sometimes in both).

For this processor, 431 diagnostics must be examined to get an accurate evaluation of its negative test status.

Documents which are not Well-Formed

All XML processors must correctly reject (with a "fatal" error) all XML documents which are not well-formed. (Nonvalidating processors may skip some of these tests, if they require handling of a type of external entity which the processor ignores. Such skipped tests are not reported.)