There was a lot of passionate debate around this point. Many people feel very strongly that if you're in the Web environment and there is an HTTP header, that's the only thing that should be used to determine character encoding; that the BOMs and encoding declarations and so on are just extras in those cases. They felt this so strongly that they wanted to write that rule right into the main body of the spec.
The majority felt, though, that XML was probably going to be used regularly in non-Web applications, so there was really no place for this kind of language in the main body of the spec. This text in an appendix represents the compromise. Lesson: the best experts in this area really think you ought to use external headers, when you have them. They're probably right.
Copyright © 1998, Tim Bray. All rights reserved.