XML and EDI Lessons Learned and Baggage to Leave Behind
August 4, 1999
I first sensed Electronic Data Interchange (EDI) had a less than glorious future when I visited a printing plant in Des Moines, Iowa a few years ago. This company, with about 120 employees, was one of the most innovative I have ever seen. It implemented the Deming continuous process improvement methods seriously, tracked its jobs meticulously, looked after the welfare of its staff, and grew quickly as a result. Yet the company had only one information systems person on staff and would not even consider implementing EDI. Why not? EDI took too long to implement, required expensive networks and software, and provided little in the way of benefits for a company this size.
This company's experience is hardly unique. Many smaller companies using EDI today do so only because their bigger customers demand it, a business model called hub-and spoke. If your company is the hub, the system works fine. But if you are the spoke it presents an enormous cost of doing business.
|"The XML community has a chance to fashion new forms of business data exchange where even the smallest companies can benefit."|
As a result, XML has an opportunity to fill the gap for companies like this printer and millions of others like it. The XML community has a chance to fashion new forms of business data exchange where even the smallest companies can benefit, and XML is the key. Charles Goldfarb and Paul Prescod, in The XML Handbook (Prentice-Hall, 1998) outline the problem with traditional EDI and the opportunity for a new EDI based on XML (page 103)…
"Traditional" EDI is based on outdated principles that will cause it to fade into technological obscurity, as it becomes embraced and replaced by the "New" EDI. Traditional EDI refers to the use of rigid transaction sets with business rules embedded in them. This model simply does not work in today's rapidly changing business environment.
This problem is compounded by the fact that companies have chosen to interpret these transaction set standards in ways that suit their unique business requirements. As a result, vendors who engage in EDI with multiple customers typically must create a unique solution to handle the transaction sets for each company. This makes the implementation of EDI far too expensive, especially for [Small and Medium size Enterprises] SMEs.
While EDI may deserve these criticisms, the XML world may want to build on the 30 years of EDI rather than tearing it all down. XML needs a lot of the EDI experience, for all of its warts, to make business data exchanges and the benefits derived from them a reality. If companies want to exchange business data to help reduce inventories, get products faster to market, create closer coordination between manufacturing and distribution, and provide more choices for consumers, EDI has lessons to help get there.
EDI, Warts and All
The Data Interchange Standards Association (DISA), the not-for-profit group that oversees the development of EDI standards in the United States, defines EDI as:
the computer-to-computer exchange of business data in standard formats. In EDI, information is organized according to a specified format set by both parties, allowing a "hands off" computer transaction that requires no human intervention or rekeying on either end. The information contained in an EDI transaction set is, for the most part, the same as on a conventionally printed document.
This definition helps explain why EDI has had success, at least with Fortune 1000 companies, but also has the seeds of many of the problems it faces . First, consider a little history of how and why EDI developed.
|"In EDI, information is organized according to a specified format set by both parties, allowing a "hands off" computer transaction that requires no human intervention or rekeying on either end."|
A dedicated group of visionaries led by the late Ed Guilbert of Washington, DC saw the need to reduce the growing burden of paper documents needed to ship goods from one place to another. Guilbert came out of the U.S. Air Force logistics command and played a key role in organizing the Berlin Airlift in 1948. He formed the Transportation Data Coordinating Committee (TDCC) in the early 1970s to develop voluntary standards for electronic formats to replace the growing pile of hard copy documents needed for shippers, transportation companies, customs authorities, warehouses, and receiving companies.
TDCC standards defined file formats and data elements for freight shipments, and continued until the early 1990s, when ANSI's Accredited Standards Committee X12 incorporated them into the overall North American EDI standards. The X12 standards identify a series of basic electronic documents called transaction sets. These transaction sets cover common business documents such as purchase orders, invoices, and manifests but also specialized single-industry documents such as those for government or health care use. EDI for health care has grown significantly in recent years; newer versions of the X12 standard read almost like medical textbooks.
The X12 transaction sets break down into data segments, consisting of collections of related data elements. The date/time data segment for example consists of data elements for:
- Date/time qualifier, a code indicating the type of date or time, such as date of the transaction or time of delivery
- Date, in CCYYMMDD format
- Time, in 24 hour format
- Time code or zones
- A date/time period format qualifier, indicating the date and time format -- various combinations of century, year, month, date, hours, minutes, and seconds
- Date/time period, an expression of one or a range of dates and times
The X12 standard is modeled on a relational database and, to its credit, makes many of the data segments and elements interchangeable. Dates, for example, appear in every transaction set, and use the same date/time segment shown above. Where a segment needs a simple date or time, it borrows those specific data elements without all of the qualifiers in the date/time segment.
|"EDI has enabled many companies to realize substantial savings on processing hard copy documents."|
For all the criticism heaped on EDI, the work of thousands of companies would look very different and much uglier if it did not exist. EDI has enabled many companies to realize substantial savings on processing hard copy documents, but that is only the beginning. Business practices like vendor-managed inventories, where suppliers keep close track of customer inventories and replenish as needed, could never happen without electronic product activity reports coming from the customer. Evaluated receipts settlements, a practice that literally removes invoices and the accounting they entail from the business process, can happen only if companies exchange purchasing, shipping, and receiving data electronically. As a result, many enterprises using EDI have streamlined their business processes and improved productivity markedly.
EDI has generated other very real and tangible results. Because organizations exchange data electronically, they can track individual units of production, raw materials for example, where before the volume of data would swamp human capacities. Graphic Communications Association (GCA, my employer) established a standard EDI transaction for transmitting paper performance data from printing plants to paper mills, where the mills can plug the data directly into their statistical quality programs. The result is a higher quality product and less downtime by the customer.
|"Rigid transaction set formats are one of the problems of EDI, but th at is also one of its strengths."|
Organizations exchanging EDI transactions can set up their systems for hands-off processing, as the DISA definition notes, because the standards provide for a predictable structure to the data stream. Goldfarb and Prescod, in The XML Handbook (Prentice-Hall, 1998, p. 103) point out rigid transaction set formats as one of the problems of EDI, but in fact, that is one of its strengths. Rigidity can also mean stability, and if a company plans to quickly process thousands of detailed transactions or line items within these transactions, they must have predictability in the incoming data stream.
While EDI deserves a lot of the credit for these improvements in business processes, it also needs to recognize its shortcomings. EDI standards are large, complex, difficult to implement, and often have high price tags attached for software, networks, and consultants. A look at the date/time segment above provides a glimpse of its complexity. The X12 standard addresses the needs of nearly all industries and businesses, and therefore needs to cover nearly all contingencies. As a result, defining something as seemingly simple as date and time requires a definitional exercise befitting a Talmudic scholar, a process repeated for almost all data that organizations want to send or receive. It can take months or even years for an industry to define an implementation of an X12 transaction set.
|"Defining something as seemingly simple as date and time requires a definitional exercise befitting a Talmudic scholar."|
People often talk of EDI standards in the plural, for good reason. While North America may have its X12 standard, the rest of the world uses the UN/EDIFACT standard for EDI. UN/EDIFACT resembles X12 in many ways but still has many differences that require companies doing business internationally to carry at least two sets of electronic formats for each transaction. X12 has implemented some of the features of the UN/EDIFACT system, but they are still different standards.
To make matters worse, each industry defines its implementation guidelines for the X12 standard differently. In many respects, one cannot avoid this situation since each industry has its own set of business rules and practices. But unless the respective industry groups developing the transactions make a special effort to agree on common definitions, interoperability depends as much on sweet good fortune as anything else. Companies serving many industries, such as warehousing or logistics services, need to carry each industry's EDI implementations in their systems. At the same time, companies wanting to move their management services to EDI will find different implementations for banking, insurance, telecommunications, and health care for example.
|"In addition to different EDI standards around the world and in different industries, the X12 standards change every year."|
In addition to different EDI standards around the world and in different industries, the X12 standards change every year. The most basic change is the addition of new transaction sets. In 1992, the X12 standard had 105 transaction sets in version 3020 of the standard. By 1998, the number had ballooned to 297 transaction sets in version 4010. Besides the new transaction sets, the technical working groups that maintain the standard will change the data segments, elements, and codes as well. For example, GCA's EDI Committee recently upgraded its transaction sets for paper inventory control. In doing this task, it found that the code for short tons (2000 pounds, as opposed to metric tons or 1000 kilograms), the most common measure of roll paper volume in North America, had changed in the X12 standard. Every company using EDI for paper purchases and shipments in the industry had to change their code tables as a result.
Using EDI day-to-day gets pricey. Translator software that takes data from legacy systems and formats them in the X12 syntax and back again, needs to change with the growing and ever changing X12 standard. Therefore it often has a high initial price tag and maintenance costs. Enterprise management software (e.g., SAP, Peoplesoft) often has EDI modules that make implementation a little easier, but they still need to keep up with changes in the EDI standards.
Until recently, organizations sent just about all EDI transactions over value-added networks that also had high price tags both for startup and ongoing operations, often charging per thousand characters transmitted. The Internet pricing model of flat monthly rates has forced most of the networks to change their pricing structures.
XML to the Rescue?
XML, with its ability to transfer structured data over the Web, immediately attracted the attention of people and organizations struggling with traditional EDI. In August 1997, as XML had begun to emerge, David Webber and Bruce Peat, two of the founders of the XML/EDI Group (which later became a special membership organization of the GCA Research Institute) explained how XML could transmit structured data like those carried in EDI:
An EDI application for XML provides the structural complexity that supports and parallels today's EDI transaction sets. XML provides a rich document structure that can be nested to any level of complexity. With XML, our documents are like chameleons, capable of being processed by different components, delivered by different mechanisms, and displayed to the user in different ways.
By using the XML extensible tag set, EDI "objects" can be either passed or dynamically referenced to objects stored in repositories. The XML/EDI Group is proposing the use of XML as a "carrier" for the document information so that the transaction can carry not only data (like trad-edi), but also code (at each level in the transaction tree). With an element having data properties and code methods, this allows the business elements to be manipulated as "objects". [Note: object inheritance is slated to be incorporated in a later version of XML.] In XML, each document is an object and each element of the document is an object.
The logical structure of the document and tag set can be specified in a Document Type Definition or DTD. (The best-known example of a DTD is HTML, which is defined by a DTD describing the structure of HTML documents.) In a DTD, sets of elements and their attributes are defined; the names that are used as tags are assigned; and the element relationships or transaction is defined. If a DTD is used then programs can validate the transaction's structure. You can validate the structure of an XML/EDI document automatically. If this sounds complex, it is not. Defining your own markup language (DTD) with XML is actually surprisingly simple. (XML/EDI Group web site, www.xmledi.com/)
|"The business model of the Web stresses inexpensive and even free tools, so the economics of XML held out the promise of extending the benefits of EDI to smaller companies."|
Moreover, the business model of the Web stresses inexpensive and even free tools, so the economics of XML held out the promise at least of extending the benefits of EDI to smaller companies with considerably less pain and expense of traditional EDI. At this point, the economics of XML appear to be moving in that direction. The W3C and groups developing specifications for XML applications make these documents available free of charge over the Web. One can find XML software tools for free or nominal costs from several vendors. Leading web browsers available now (or soon) for free or nominal cost support XML. Internet service providers offer connections for low monthly rates, and stiff competition in the field promises to keep rates low, even as bandwidth expands over cable and fiber-optic lines.
XML appears to solve the problem of differing North American vs. international standards found in EDI. The ubiquity of the Web and relative ease of connections worldwide has made development of XML and its various applications international exercises. The XML/EDI Group and CommerceNet, two of the standards groups working on XML/EDI guidelines, have active European and Asian chapters as well as North American participation. IBM, Microsoft, iPlanet (the Sun/Netscape alliance), and most other leading vendors are truly international companies with a stake in preventing national or regional conflicts in standards.
XML also appears to better handle the problem of integrating data from EDI transactions into corporate systems. EDI transactions move door-to-door, that is from the sender’s mailbox to the receiver’s. When a transaction arrives, EDI translator software strips away the X12 syntax and presents a flat file that the receiver’s systems need to parse, check for accuracy, edit, and distribute. Enterprise management software will often include these functions with the package, but most of these packages are priced well beyond the resources of small and medium-sized companies. For smaller organizations, they either need to write custom handlers or print-and-rekey, two less desirable options.
Using XML, enterprises have more options for the display and processing of incoming data. The Extensible Stylesheet Language (XSL) allows for the visual display of incoming data and formatting of those same data for further processing by corporate systems. In addition, more end-user applications packages already or plan to support XML that enables the recipients to capture and process the incoming data directly. Even with legacy systems, industry groups can specify standard scripting language or Java code that reflects industry rules, to provide for greater mapping and integration of data exchanged over the Web with XML.
EDI, Take It and Leave It
XML can learn valuable lessons from the EDI experience, as well as leave some its baggage behind. First, the lessons:
Data exchanges need predictability. Enterprises receiving data from trading partners need to know the format of those data coming down the line. EDI presents a predictable format that enables trading partners to know with certainty the contents of the data stream. In the X12 manifest/ship notice transaction set, for example, the detail segments begin with a hierarchical level (HL) segment, followed by an item identification (LIN) segment, and so on. Exchanges using XML need no less certainty.
|"When companies go from hard copy to electronic transactions, they lose one of the basic means of tracking business activity—through human eyes."|
Identify everything. With EDI, everything is identified uniquely -- each trading partner, each location, each transmission, each product, each shipment, and sometimes each unit load within a shipment. It seems obvious, but when companies go from hard copy to electronic transactions, they lose one of the basic means of tracking business activity -- through human eyes. As a result, all of the tracking of transactions relies on the same systems used to send, receive, and process the transactions. Therefore, one needs means of tracking the transactions, and elements of the transactions, that are independent of the sending and receiving systems.
For example, one of the best practices employed with EDI is the use of the manufacturers' product numbers or stock-keeping units (SKUs) by all of the companies in the supply chain. This practice provides a straightforward way of tracking inventories throughout the chain, rather than interpreting different codes assigned to the same products as they flow from manufacturer to distributor to retailer to consumer. Unique identification also allows for accumulating data at a fine level of detail, which provides valuable raw material for forecasting future requirements.
Another example is the use of Data Uniform Numbering System (DUNS) numbers assigned by Dun and Bradstreet to identify trading partners. Most every company has a unique DUNS number for financial reporting and by using that number, trading partners have a common means of identifying each other. Some industry groups have company identifiers for other purposes, such as the paper industry's two-character company codes used in bar codes that become part of a unique unit number assigned to each roll of paper manufactured in North America. EDI transactions in the paper industry use this North American Roll Identifier to identify each roll of paper shipped from paper mills and used in printing plants.
Acknowledge everything. No good EDI transaction or XML transfer should go unacknowledged. Nor should any bad transaction or transfer either. Good X12 practice encourages use of a functional acknowledgment with each transmission. A functional acknowledgment that uses the transaction set number 997 tells the sender when the trading partner retrieved the message and indicates any X12 syntax errors.
|"To explain the difference between functional and substantive acknowledgments, consider a functional acknowledgment as the equivalent of mimicry and substantive acknowledgment as conversation."|
Good EDI practice also encourages use of substantive as well as functional acknowledgments. To explain the difference between functional and substantive acknowledgments, consider a functional acknowledgment as the equivalent of mimicry and substantive acknowledgment as conversation. Both acknowledgments send a message back to the sender, but one contains vastly more value than the other. An example of a substantive acknowledgment is the purchase order acknowledgment sent upon receipt of a purchase order. This transaction tells the supplier's understanding of the order and lists any exceptions to its terms or conditions. Another example is the receiving advice that lists in detail the items received, and any differences between the shipping manifest and the items unloaded at the receiving dock.
One XML vocabulary that puts a premium on identification and acknowledgment is the Information and Content Exchange (ICE) protocol used for syndicating content over the Web. While first considered a publishing application, ICE has found significant business uses with manufacturing parts catalogs and financial services. The ICE authoring group, an organization under the GCA Research Institute umbrella, takes great pains to identify the different parties in these exchanges. ICE also uses a request-and-response model so no transmission in the process goes without an acknowledgment.
|"The real benefits from EDI have occurred when trading partners looked at the data they needed to exchange and stopped thinking of documents."|
Think data not documents. Despite the idea of EDI transaction sets being modeled after hard copy documents, the real benefits from EDI have occurred when trading partners looked at the data they needed to exchange and stopped thinking of documents. Recent supply chain management initiatives use EDI transactions for exchanging much of the data needed for forecasting and replenishment of inventories, but they are based on an analysis of product and information flows not on current hard copy documents or EDI transactions for that matter.
When evaluating the flood of new XML vocabularies ask yourself if they have any basic data modeling behind them, or are they emulating paper documents? I find it interesting that one benefit of the new supply chain management experiments is the reduction and almost elimination of the purchase order. Yet, look at all the white papers on XML that use a purchase order as an example. Is this how XML plans to save the world?
EDI's other legacy. EDI may have some achievements to crow about, but it needs to eat a little crow as well.
|"Organizations have little incentive to develop systems if the standards on which they are based change constantly."|
The idea of EDI standards changing every year challenges the whole idea of standards. Organizations have little incentive to develop systems if the standards on which they are based change constantly. Standards should provide an element of stability and change only with the greatest reluctance. It requires standards groups to take the time to get consensus among the industries they serve and test the specifications before issuing the documents. You can fix bugs, but once you issue a standard stick to it, at least for a while.
XML will enable transactions to go room-to-room, not just door-to-door. The extended enterprise, as futurist Don Tapscott calls it, really exists today. The manufacturer of a product for example has almost as much stake in the customer's success as the customer itself. Why stop and start data transfers therefore at the respective company mailboxes? EDI may send data from one company to another, but the data flow starts much deeper in the organization and needs to go directly to where the trading partner uses the data. With XML, data can flow from an application in one company's system to its equivalent application with the trading partner.
|"If any legacy of EDI needs erasing, it is the X12/EDIFACT dichotomy, which hangs like an anvil on the development of EDI systems."|
An encouraging sign with XML is the international nature of the specifications under development. If any legacy of EDI needs erasing, it is the X12/EDIFACT dichotomy, which hangs like an anvil on the development of EDI systems. Companies in Kuala Lumpur or Kalamazoo should use the same set of standards for their business data exchanges. Period.
Going Where No Business Data Have Gone Before
Developing a new paradigm for business data exchanges means thinking both in new ways and old, combining the promise of XML with the lessons from EDI. Here are a few ways to encourage it happening:
Aim to improve the business process. EDI has taught the business world that the real payoff comes from the improvement in the business process, not just automating the status quo. Innovations like evaluated receipts settlements or vendor-managed inventories happened when trading partners used the data sent electronically to streamline operations, get products to market faster, or better serve the customers. XML applications should begin with a business and data model that tries to improve the ways of doing business.
Perhaps the best example of XML applied this way is the RosettaNet project undertaken by the information technology industry. The RosettaNet project first identifies a series of partner interface processes or PIPs that directly attack inventory or customer service problems identified in the business. These PIPs then drive the development of XML exchanges. RosettaNet provides a good model for other industry and standards groups to follow.
|"RosettaNet provides a good model for other industry and standards groups to follow."|
Sweat the details. We have seen far too many announcements of XML 'frameworks' in the past several months. It is time for details to start emerging. By details I mean individual well-defined data elements, with the relationship among the elements clearly spelled out. Details also mean a universal system for identifying trading partners, uniquely identified transactions, and fine levels of granularity for the contents of the transactions. Suppliers for example may identify shipments or even individual pallets in a shipment, but if one can identify individual cartons, why not do it? The more detail one can provide, the more information one has about the shipment, and thus more control one has over the inventory.
Aim for interoperability. One of the buzzwords in our business is interoperability, but in some key sectors it has become critical. With more use of just-in-time deliveries, vendor-managed inventories, and supply chain integration, we have become more reliant on transportation and warehousing services. Yet these vital services need to carry products from many different businesses, so either they reconcile different industry vocabularies or impose their own standards on the trading partners.
|"With the increasingly fluid nature of mergers, acquisitions, and alliances, one needs ways of quickly realigning ones business terms and rules, and Bizcodes offer a means of meeting that test."|
A better approach might be for XML applications to develop glossaries that link their data elements and tags to a neutral set of terms, known as Bizcodes or basic semantic repositories. While it adds a little to the development process, it pays off much later down the road. Also, with the increasingly fluid nature of mergers, acquisitions, and alliances, one needs ways of quickly realigning ones business terms and rules, and Bizcodes offer a means of meeting that test.
Link to other technologies. While we may marvel at XML, there are other technologies out there that also offer opportunities for significant business process improvements. Many of these exciting new technologies involve automatic identification, such as smart cards or radio frequency identification (RFID). Smart cards, that allow for replenishment of monetary value for example, have had a mixed record in the United States but have gained much more acceptance overseas. RFID however has become a hot technology at gas pumps and toll booths as it eliminates the need for small cash exchanges.
|"If we can meet these privacy concerns, the combination of XML and smart cards can encourage more personalized deliveries of goods and services."|
Smart cards and RFID allow for the tracking of individuals, in much the same way as bar codes allow for the tracking of things. Using these technologies in the tracking of people however raises basic privacy questions that need to be addressed. If we can meet these privacy concerns, the combination of XML and smart cards for example, can encourage more personalized deliveries of goods and services, much like push technology on the web provides for personalized information deliveries.
Develop a core set of standards for small business. When Volkswagen had its original beetle in the 1960s, it had a print ad campaign around the theme 'small is beautiful.' Small business nowadays is also beautiful, and it offers the largest potential market for XML business data exchanges. For many of these companies, the business model consists of a passion for the work they perform and a deep caring for the customers. This group, perhaps more than any other, deserves the benefits of XML data exchanges that improve the way business is conducted. But it can only happen if XML becomes part of the $100 accounting package and the $25 browser, or downloaded from the industry XML repository. Until these businesses start reaping the benefits of XML, our work is not done.