XML Processor Conformance Report:
MSXML3 SAX from VB

This is a report on the conformance of the following XML 1.0 processor configuration:

The results were as reported through the parser's API to this particular test harness and execution environment:

A summary of test results follows. To know the actual test status, someone must examine the result of each passed negative test (and informative test) to make sure it failed for the right reason. That examination may cause the counts of failed tests to increase (and passed tests to decrease), changing a provisional "conforms" status to a "does not conform".

Sections of this report are: Explanation of Tables; Positive Tests, cases where this processor should report no errors; Negative Tests, documents for which this processor must report the known errors; and Informative Tests, documents with errors which processors are not required to report.

NOTE: The OASIS/NIST test suite is currently in draft state, and can't actually be used without modifications to the configuration file, which is used both to generate the test documentation published at the OASIS/NIST site and to operate this test harness. In some cases, test cases may need to be reclassified; this would affect results attributed to parsers. Accordingly, treat these results as preliminary.

Explanation of Tables

Sections presenting test results are composed largely of tables, with explanations focussing on exactly what those tables indicate. Diagnostics for failed tests are presented in italics, with a cherry colored background, to highlight the result. Diagnostics for succesful tests should as a rule only exist for negative tests. Initial parenthesized comments typically come from the test harness.

Some such comments indicate the reporting category defined in the XML specification. Some low-fidelity processor APIs don't expose recoverable errors, which can make validation work awkward.

(fatal)

The diagnostic was reported as a fatal error. Such errors are primarily well-formedness errors, such as the violation of XML 1.0 syntax rules or of well formedness constraints. In some underfeatured parser APIs, this may be the only kind of error that gets reported.

(error)

The diagnostic was reported as a recoverable error. Such errors primarily used to report validation errors, which are all violations of validity constraints, but some other errors are also classed as nonfatal.

(warning)

The diagnostic was reported as a warning; warnings are purely informative and may be emitted in a number of cases identified by the XML 1.0 specification (as well as in other cases).

Other such comments may indicate other categories of conformance issue. For example, some errors relate to problematic implementation of SAX; and in exceptional cases, the harness can be forced to report a failure on some test.

(thrown classname) ... abnormal

The named exception was directly thrown. If the exception is a SAXException (or a subclass thereof) this suggests an error in the parser (violating the SAX API specification) since it should normally have used the SAX ErrorHandler instead.

(odd classname) ... abnormal

After the identified exception was reported through the ErrorHandler, an exception of the named class was thrown directly. This suggests an error in the parser, since the parser either failed to continue after an error (or warning) which is required to be continuable, or else it did not pass the exception thrown by the application back to the application.

(EXCEPTION - DIRECTED FAILURE) ... abnormal

This test case was explicitly failed by the test operator; the test was not run. This may be done in the case of parsers with severe bugs which completely prevented handling the test case, typically because the parser seems to "hang" by entering an infinite loop.

In all cases, negative tests that appear to pass (diagnostics presented with a white background) must be individually examined in the report below. The diagnostic provided by the processor must correspond to the description of the test provided; if the processor does not report the matching error, the seeming "pass" is in fact an error of a type the test harness could not detect or report. That error is either a conformance bug, or an error in the diagnostic being produced; or, rarely, both.

Nonvalidating processors may skip some tests if the tests require processing a class of external entities (general, parameter, or both) which that processor is known not to handle. If processor handling of entities is not known, all such tests are skipped, in order to prevent misreporting.

Positive Tests

All conformant XML 1.0 processors must accept "valid" input documents without reporting any errors, and moreover must report the correct output data to the application when processing those documents. Nonvalidating processors (such as this one) must also accept "invalid" input documents without reporting any errors. These are called "Positive Tests" because they ensure that the processor just "does the right thing" without reporting any problems.

In the interest of brevity, the only tests listed here are those which produce diagnostics of some kind, such as test failures. In some cases, warnings may be reported when processing these documents, but these do not indicate failures.

No interpretation of these results is necessary; every "error" or "fatal" message presented here is an XML conformance failure. Maintainers of an XML processor will generally want to fix their software so that it conforms fully to the XML specification.

Valid Documents

All XML processors must accept all valid documents. This group of tests must accordingly produce no test failures.

Output Tests

The XML specification places requirements on the data which is reported by XML processors to applications. This data flows through the processor API ... or it is not available, so the processor is in those respects nonconformant. For example, SAX1 did not report external entities which were not included; but SAX2 does. These output tests verify conformance with the specification by recording that data and comparing it with what is required for conformance with the XML 1.0 specification.

At this writing, the OASIS output tests have several categories of known omissions (or weak output test coverage). These include:

• No output tests address the additional requirements which validating processors must satisfy. That is, reporting which whitespace is ignorable, and reporting declarations of unparsed entities.
• Only a few output tests have been provided which address the requirement to report NOTATION declarations, and some of those appear to be missing.
• No tests address the requirement to report external entities which were not included.

Note that output tests automatically fail in cases where the processor failed to parse the (valid) input document used to generate the output data.

In some test harnessses, the output tests are unreliable because they can't directly compare the parser output against reference data. Such issues should be noted in the documentation for that harness.

Also, and not a bug, in some cases these diagnostics may seem like they say two equivalent results are not equal. The issue is that some differences, often those in reported whitespace, aren't easily visible in this report. HTML hides many such differences (because it normalizes whitespace before displaying it), and the method used to display the differing results may also mask some issues.

Invalid Documents

As noted above, nonvalidating processors must accept all documents which are well formed, but invalid. This same behavior would be delivered by a validating processor, if the application chose to continue processing after receiving each report of a validity error, and not report such validity errors. (These tests are run as "negative" tests for validating processors, since in those cases it is important that the correct validity errors be reported and that they be reported at the correct level.)

Negative Tests

All conformant XML 1.0 processors must reject documents which are not well-formed. In addition, validating processors must report the validity errors for invalid documents. These are called Negative Tests because the test is intended to establish that errors are reported when they should be.

Moreover, the processor must both fail for the appropriate reason (given by the parser diagnostic) and must report an error at the right level ("error" or "fatal"). If both criteria were not considered, a processor which failed frequently (such as by failing to parse any document at all) would appear to pass a large number of conformance tests Unfortunately, the test driver can only tell whether the error was reported at the right level. It can't determine whether the processor failed for the right reason.

That's where a person to interpret these test results is critical. Such a person analyses the diagnostics, reported here, for negative tests not already known to be failures (for not reporting an error, or reporting one at the wrong level). If the diagnostic reported for such tests doesn't match the failure from the test description, there is an error in the diagnostic or in the processor's XML conformance (or sometimes in both).

For this processor, 499 diagnostics must be examined to get an accurate evaluation of its negative test status.

Documents which are not Well-Formed

All XML processors must correctly reject (with a "fatal" error) all XML documents which are not well-formed. (Nonvalidating processors may skip some of these tests, if they require handling of a type of external entity which the processor ignores. Such skipped tests are not reported.)

Informative Tests

Certain XML documents are specified to be errors, but the handling of those documents is not fully determined by the XML 1.0 specification. As a rule, these errors may be reported in any manner whatsoever, or completely ignored, without consequence in terms of conformance to the XML 1.0 specification. And some of these documents don't have errors; documents in encodings other than UTF-8 and UTF-16 are legal, but not all processors are required to parse them.

Such "optional" errors are listed here for informational purposes, since processors which ignore such errors may cause document creators to create documents which are not accepted by all conformant XML 1.0 processors. (And of course, processors which produce incorrect diagnostics for such cases should be avoided.)

This report was produced by Free Software from http://xmlconf.sourceforge.net, and you should be able to reproduce these results yourself.