XML Processor Conformance Report:
org.brownell.xml.OracleDriver

This document is the output of an XML test harness. It reports on the conformance of the following XML 1.0 processor configuration, as reported through the Java version of the SAX (Simple API for XML) parser interface. For SAX2beta processors, the processing mode and entity handling values could be non-default values, if such a feature is supported by that processor. SAX1 processors were wrapped in an adapter, and those values were as reported to the test harness by its operator.

The test harness and execution environment was:

An summary of test results follows. To know the actual test status, someone must examine the result of each passed negative test to make sure it failed for the right reason. That examination may cause the counts of failed tests to increase, changing a provisional "conforms" status to a "does not conform".

Sections of this report are: Explanation of Tables; Positive Tests, cases where this processor should report no errors; Negative Tests, documents for which this processor must report the known errors; and Informative Tests, documents with errors which processors are not required to report.

NOTE: The OASIS/NIST test suite is currently in draft state, and can't actually be used without modifications to the configuration file, which is used both to generate the test documentation published at the OASIS/NIST site and to operate this test harness. Accordingly, treat these results as preliminary.

Explanation of Tables

Sections presenting test results are composed largely of tables, with explanations focussing on exactly what those tables indicate. Diagnostics for failed tests are presented in italics, with a cherry colored background, to highlight the result. Diagnostics for succesful tests should as a rule only exist for negative tests. Diagnostics with parenthesized comments indicate the reporting path:

(fatal)

The diagnostic was reported as a fatal error. Such errors are primarily well-formedness errors, such as the violation of XML 1.0 syntax rules or of well formedness constraints.

(error)

The diagnostic was reported as a recoverable error. Such errors primarily used to report validation errors, which are all violations of validity constraints, but some other errors are also classed as nonfatal.

(warning)

The diagnostic was reported as a warning; warnings are purely informative and may be emitted in a number of cases identified by the XML 1.0 specification (as well as in other cases).

(thrown classname) ... abnormal

The named exception was directly thrown. If the exception is a SAXException (or a subclass thereof) this suggests an error in the parser (violating the SAX API specification) since it should normally have used the SAX ErrorHandler instead.

(odd classname) ... abnormal

After the identified exception was reported through the ErrorHandler, an exception of the named class was thrown directly. This suggests an error in the parser, since the parser either failed to continue after an error (or warning) which is required to be continuable, or else it did not pass the exception thrown by the application back to the application.

(EXCEPTION - DIRECTED FAILURE) ... abnormal

This test case was explicitly failed by the test operator; the test was not run. This may be done in the case of parsers with severe bugs which completely prevented handling the test case, typically because the parser seems to "hang" by entering an infinite loop.

In all cases, negative tests that appear to pass (diagnostics presented with a white background) must be individually examined in the report below. The diagnostic provided by the processor must correspond to the description of the test provided; if the processor does not report the matching error, the seeming "pass" is in fact an error of a type the test harness could not detect or report. That error is either a conformance bug, or an error in the diagnostic being produced, or sometimes both.

Note also that some SAX processors (or drivers for non-SAX processors) appear to have a bug wherein they don't close all of the files which they open. When running this test harness with large numbers of tests, failures at the end of this report could reflect such bugs in SAX conformance more than bugs in XML processor conformance.

Positive Tests

All conformant XML 1.0 processors must accept "valid" input documents without reporting any errors, and moreover must report the correct output data to the application when processing those documents. Nonvalidating processors must also accept "invalid" input documents without reporting any errors. These are called "Positive Tests" because they ensure that the processor just "does the right thing" without reporting any problems.

In the interest of brevity, the only tests listed here are those which produce diagnostics of some kind, such as test failures. In some cases, warnings may be reported when processing these documents, but these do not indicate failures.

No interpretation of these results is necessary; every "error" or "fatal" message presented here is an XML conformance failure. Maintainers of an XML processor will generally want to fix their software so that it conforms fully to the XML specification.

Valid Documents

All XML processors must accept all valid documents. This group of tests must accordingly produce no test failures.

Output Tests

The XML specification places requirements on the data which is reported by XML processors to applications. This data flows through the SAX API (with one minor omission: reporting external entities which are not included). These output tests verify conformance with the specification by recording that data and comparing it with what is required for conformance with the XML 1.0 specification.

Note that output tests automatically fail in cases where the processor failed to parse the (valid) input document used to generate the output data.

As of this writing, no output tests address the additional requirements which validating processors must satisfy. That is, reporting which whitespace is ignorable, and reporting declarations of unparsed entities.

Negative Tests

All conformant XML 1.0 processors must reject documents which are not well-formed. In addition, validating processors (such as this one) must report the validity errors for invalid documents. These are called Negative Tests because the test is intended to establish that errors are reported when they should be.

Moreover, the processor must both fail for the appropriate reason (given by the parser diagnostic) and must report an error at the right level ("error" or "fatal"). If both criteria were not considered, a processor which failed frequently (such as by failing to parse any document at all) would appear to pass a large number of conformance tests Unfortunately, the test driver can only tell whether the error was reported at the right level. It can't determine whether the processor failed for the right reason.

That's where a person to interpret these test results is critical. Such a person analyses the diagnostics, reported here, for negative tests not already known to be failures (for not reporting an error, or reporting one at the wrong level). If the diagnostic reported for such tests doesn't match the failure from the test description, there is an error in the diagnostic or in the processor's XML conformance (or sometimes in both).

For this processor, 463 diagnostics must be examined to get an accurate evaluation of its negative test status. These have been reported as provisionally passed tests.

Invalid Documents

Validating processors must correctly report "error" diagnostics for all documents which are well formed but invalid. Such errors must also be, "at user option", recoverable so that the validating parser may be used in a nonvalidating mode by ignoring all validity errors.

Documents which are not Well-Formed

All XML processors must correctly reject (with a "fatal" error) all XML documents which are not well-formed.