How many computing platforms do you have in your home? Now add in your office. How many platforms does your workplace support: Windows, Linux, Mac OS X, AIX, Solaris, QNX, HP/UX, FreeBSD, VxWorks? Probably more.
This list does not begin to reflect the whole array of platforms available to developers. One can count dozens of operating systems running on everything from tiny embedded systems to home computers, servers, and mainframes. Many of these operating systems (Linux and VxWorks, among others) support multiple hardware platforms, including different processors. On top of that literally running on top of the operating system you can find dozens of software frameworks such as Java, .NET, VCL/DBX, etc.
In this article, I examine how a layered architecture enables an application to keep pace with an ever-changing world of APIs and frameworks. Applying these techniques to your products will help you keep pace with the fast-changing landscape by limiting portability issues and providing product flexibility.
We have all been taught the basics of programming with cross-platform portability in mind: Isolate platform-specific code, use
typedef for data types, pay attention to size and alignment of data objects, and use function wrappers to abstract specifics of a user interface from the core logic. As it turns out, there is more value in these techniques than simply reducing the time required to port to new platforms. By adhering to these best practices, companies can expand their product offerings in new directions and on new operating systems, without affecting the core technology.
While this has always been true, it is so often forgotten. Developers begin to write an application with a particular platform or device in mind (for example, iOS), then later decide that they want the application to run on another platform (Android). Without careful and intentional consideration up front about the overall architecture and placement of your platform-specific code, you can wind up rearchitecting the entire application simply to support the additional platform.
The software industry has been facing these same issues since its beginning. As evidence to this statement, a colleague and I described the techniques used for portability across operating systems (function wrappers, creating portable data types using
typedef, etc.) in Dr. Dobb's Journal in March 1995.
APIs, APIs, and More APIs
Just as there has been a proliferation of platforms, the number of programming APIs has grown considerably. APIs can be supported portably by using a layering approach, rather than the isolation approach (isolating all of the platform-specific code) discussed previously.
To illustrate, allow me to explore how we implemented these approaches at FairCom with c-treeACE, our database product that has been ported to more than 100 platforms and, at present count, supports 17 APIs and frameworks. A layered architecture has allowed us to keep pace with the new APIs and frameworks. Applying similar techniques to your own products will help you keep up as well.
An API Stack
At their heart, most database products implement fundamental concepts of data storage and retrieval. At the end of the day, the purpose is the same put data reliably on a storage medium (hard disk, solid state memory, RAM), and quickly find and retrieve it when needed. Most relational-only databases fully abstract their SQL interface from the internal routines required for record-oriented, indexed storage, although most have some form of this logic embedded deep within their core at some level.
An alternative approach is to separate the functionality. We have done this with our database architecture, for example, which makes a clear distinction between the NoSQL and SQL routines. Instead of a single monolithic block of code, we use the layered architecture illustrated in Figure 1:
Figure 1: c-treeACE API/Framework hierarchy. Blue boxes = NoSQL APIs; Orange boxes = SQL APIs; Green boxes = NoSQL APIs available in c-treeRTG.
The key point to note from Figure 1 is the way the APIs/Frameworks (which, from hereon, I'll refer to as just APIs) are layered. At the core layer, there is a mix of platform-generic code and platform-specific code that together form the foundation of the database engine. Operating system calls (such as seeks, reads, writes, and so on) are found in this non-exposed layer.
Moving up a layer, the File System API (or Low-Level API) is the first level with an exposed set of API functions that begin to provide the foundation for a database actions such as create a data file, create an index, add data to a data file, add a key value, find a key value, first key, next key, previous key, etc. This layer also begins to provide the foundation for data integrity through record-level and key-level locking. A developer using this API is responsible for making API calls to update each index defined for a given data file. So if a data file has five indexes, the developer will make one "low-level" function call to update the data file, then five additional calls to update each index. This API is useful for instances where you might not need an index, or might not need a data file, and are choosing to store all of your information within an index.
Moving up through the layers, we come to the NoSQL API (or ISAM API) layer where we begin to relieve some of the overhead from the developer by automatically maintaining all the indexes for a given data file. Atomic operations are also implemented within this layer, providing an immediately consistent ACID-compliant transaction engine. Implementing the ACID-compliant transaction engine at this level is one of the keys to the product's ability to mix and match any combination of APIs over the same data.
Next we add an abstraction layer on top called "c-treeDB C API," which begins to establish more relational concepts, such as creating a database to house related tables, establishing a session, data dictionaries, and so on, and eliminating the need to reinvent these components within each SQL API.
The progression then continues to an array of NoSQL APIs. These include a Java database interface, a C++ interface, a VCL interface for Delphi and C++ Builder environments, and a .NET interface. Also on this same c-treeDB C API is a set of SQL APIs. These include ODBC, JDBC, Direct SQL, Interactive SQL, ADO.NET, and a DBX driver for Delphi and C++ Builder environments, PHP, and Python.
Stacking APIs in this manner has allowed us not only to easily introduce additional APIs, but also to more easily introduce new products.
An Extensible Solution
By grouping functionality into layers, it is possible to efficiently "plug in" any new layer. In our case, database fundamentals such as record and key level locking, deadlock detection, and an ACID-compliant transaction engine are all implemented in the first two exposed layers (File System API and NoSQL Key Value Store) of the API stack. The c-treeDB layer then makes the suite of relational APIs on top of the NoSQL core possible. This provides portability benefits because platform-specific issues tend to be confined to the lowest layers. So, when moving to a new platform, we can focus portability efforts on those two layers, knowing that once they're complete, the layers above will generally move over with no difficulty.
This layered approach also provides the ability to mix and match APIs over the same occurrence of data. Again, because many database fundamentals are implemented at a lower API level within the stack, a developer can safely mix and match any combination of the NoSQL and SQL APIs over the same data, without experiencing data integrity concerns.
Furthermore, because a new interface need only call the corresponding functions in an already existing lower layer, operation remains consistent through the various interfaces. Testing and fixes are easily accomplished because of the consistency between interfaces. In most cases, a problem only has to be fixed in one place: It is either in the underlying layer (and, therefore, common to all interfaces) or it is in the new interface (and, therefore, unique to the new interface).
By using this layered approach, we've been able to efficiently and safely provide customers with access to 17 database APIs, with undoubtedly many more to come as the technology wave continues to evolve in the decades ahead. Using a similar approach with your application will provide an extensible framework that will allow you to adapt quickly and efficiently to the changing technologies.
Randal Hoff is the VP of Engineering Services at FairCom, a company that makes database products for a wide variety of platforms.