Jim is a researcher at the University of California Santa Cruz. He can be contacted at [email protected] .ucsc.edu.
XML is an increasingly popular format for exchanging data. It handles optional fields and hierarchical data structures well, is readable by humans as well as computers, is portable across a wide variety of platforms, and it can even handle recursive data structures. In my own field of bioinformatics, XML has become almost as common as the venerable tab-separated file for exchanging information between databases. Still, integrating data from XML sources into our relational databases and our largely C code base sometimes seemed to involve more work than it should. Consequently, I was motivated to write the four toolsautoXml, AutoDtd, sqlToXml, and xmlToSqlpresented in this article. The complete source code for the programs and libraries is available electronically from DDJ (see www.ddj.com/code/) and from me (www.soe.ucsc.edu/~kent).
XML files contain structured data, and the C language, of course, can represent structured data as well. However, to get from the XML to C representations is not a simple task. At a minimum, you must write several lines of code for each field of each data structure. Converting the parent/child relationships from XML to C is especially problematic. Using stream-oriented parsers requires that your callback procedures maintain some sort of context stack so that the child can be linked into the parent. With tree-oriented parsers (such as DOM), the relationships are reflected in the parsed tree. Unfortunately, the whole tree must fit in memory, which is not possible with larger databases.
The autoXml program automates mapping between C and XML data structures. Its input is an XML document type definition (DTD), and it outputs the equivalent data structures in C. The code converts these structures to and from XML. The full tree can be loaded, or just subtrees corresponding to a particular tag and its children. The C structure can contain numeric as well as string fields.
AutoDtd creates a DTD from an example XML file. This comes in handy because not all XML files have a DTD (autoXml requires one). AutoDtd figures out whether elements are required or optional; are lists or nonempty lists; and whether the fields are integer, floating point, or string, just by examining the example file.
Another common task is saving data stored in a relational database in XML format. The sqlToXml tool produces descriptive and readable XML guided by a simple tree definition that describes the parent/child relationships within the database. Currently, sqlToXml only works with MySQL, but it shouldn't be too much trouble to adapt it to other relational databases.
Finally, xmlToSql converts an XML document into a reasonably normalized relational database. The output is a directory full of tab-separated files and SQL table creation statements, so it should work with any relational database. The normalized relational representation is typically an order of magnitude smaller than the XML representation of the data.
All four programs are built using the GCC compiler linking in with the jkweb.a library. The programs, which you can incorporate into commercial or open-source tools without licenses, are regularly built on Linux and Mac OS X, and should port with a minimum of effort to other UNIX-like platforms. You'll need to have MySQL installed to build the sqlToXml program.
The programs run from the command line and print a usage summary if run with no parameters. You can use stdin/stdout as filename parameters if you wish to use them as part of a pipeline.