How do you balance the enterprise's long-term needs with the short-term needs of application development teams? This fairly straightforward goal remains elusive in practice. Too often, project teams focus solely on their piece of the pie, building new stovepipe systems that don't fit into the existing environment, don't reuse or share services or components, and prove overly expensive to develop and maintain. It's not uncommon to discover, in the same organization, enterprise data groups, typically comprising experienced data architects and data administrators, railing about the horrendous state of application development. These folks usually have a vision to address this situation, but because they haven't actually coded in years, they often propose a data-oriented approach instead of a more contemporary object- or service-oriented architecture.
Avoiding the Road to Hell
This mismatch of vision and solution causes development teams to ignore the
data groups from which they desperately require aid.
Sound like a familiar problem? It should—I see it in almost every company that I work for, and I get around.
Your enterprise data group is likely the protector of corporate data, commited to ensuring and enhancing its quality and availability. It should also support development teams; although in many organizations, this support often proves to be more of a hindrance than a benefit. Enterprise data groups are often responsible for developing naming conventions and access techniques, maintaining meta-data about your corporate systems (see David Marco's Building and Managing the Meta Data Repository [Wiley, 2000]), and conceptual data modeling of your enterprise to identify and promote the "big picture." Prescriptive, top-down approaches for implementing data group activities have not been very effective, so it's time to consider a more agile tack. As Jim Highsmith and Martin Fowler indicated in "The Agile Manifesto" (Aug. 2001), the Agile Alliance (www.agilealliance.org) defines four values, or preferences, of agile software developers:
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
Let's explore how your enterprise data group can become more effective by adopting these values.
Beyond Tool Use
Although organization-wide views of enterprise data groups are important, few
developers are interested in them. Therefore, it's up to you to ensure that
organizational issues are addressed.
Processes and tools are still important to your success, but to be effective, they must be in sync with the needs of your environment. The world has changed dramatically since the 1970s, when most data-oriented techniques and processes were first introduced for Cobol mainframe development. Today's environment is dominated by object-oriented technologies and components, which are used to develop Internet-based applications.
This discrepancy is often revealed in a data group's insistence that development teams produce logical data models that must be onerously reviewed early in the project. What project teams actually need is ongoing and active help to understand legacy data sources as well as input into the development of their UML class models—not reviews of make—work efforts created to conform to a software process. When your data group isn't in sync with your developers, there's little hope that the two groups will interact effectively.
Disparate Goals
A significant element of the current split between enterprise data groups and
development teams arises from their disparate goals: respectively, the creation
of an accurately documented view of the enterprise versus the successful development
of a working system.
Which view is more important? I'm convinced that the focus on building systems is most critical to your project stakeholders. This is proven time and again by their willingness to hire outside vendors to build systems when they perceive that their IT department isn't up the task—even though they know they'll build stovepipes. To succeed, your IT department must be able to deliver working software and ensure that it meets the overall needs of your organization—priorities reflected in the Agile Modeling (www.agilemodeling.com) principles of Working Software Is Your Primary Goal and Enabling the Next Effort Is Your Secondary Goal. This change in viewpoint can be a major shift for data professionals who consider themselves "above the fray" of software development.
Expanding Horizons
When data groups reorganize to support the development of working software,
in addition to fulfilling their enterprise responsibilities, they quickly discover
that they need to expand their horizons because their focus on data isn't sufficient.
When all you have is a data model, everything looks like an entity. You must
look beyond data issues to truly understand an enterprise, let alone support
the complex environment faced by modern development teams.
Agile Modeling's Multiple Models principle recommends keeping several modeling techniques in your intellectual toolbox—one of which should be data modeling, so you can create the right set of artifacts. The concept of developing multiple models isn't new; in fact, it's an important aspect of the Zachman Framework (www.zifa.com) for enterprise architecture. You'll need to instantiate the Zachman Framework to meet your organization's actual needs, and, although the framework suggests using data models in its first column, a combination of UML component diagrams and UML class diagrams may be far more effective.
The Agile DBA
Enterprise data groups have two major customers: the business community and
the developers. Enterprise modelers will work with the business community, either
in strategic situations in which senior managers and executives formulate their
vision for your organization, or in project-specific situations in which they
help to explore the application's requirements.
Enterprise data groups must also support development teams. For example, an "agile database administrator" works directly with developers to perform traditional DBA activities such as building and evolving the database over time. Agile DBAs differ from their predecessors in that they work iteratively and incrementally, refactoring their data schema as needed (see "Refactoring for Fitness," The Agile Edge, Feb. 2002) instead of taking a big design up-front (BDUF) approach that insists on extensive data models early in the project. In all aspects, the agile DBA's work complements that of agile developers. Furthermore, an agile DBA accounts for enterprise issues by following corporate data standards and guidelines, and by making sure that the realities of legacy data sources and the overall environment are considered—for example, by contributing effective data migration scripts and associated tests, and ensuring that the best source data is used. An agile DBA will also collect appropriate meta-data, an activity that is often of little interest to individual developers and is typically seen as unnecessary overhead.
Traditional enterprise architects have a tough time responding to changes in technology because they have little opportunity to obtain concrete feedback. On the other hand, agile architects will refocus their efforts as required. Sometimes they'll work on evolving the enterprise models and artifacts; at other times they'll actively work with development teams, helping them to design and build systems. Patterns expert and Bell Laboratories guru Jim Coplien describes a process pattern called Architect Also Implements (www.bell-labs.com/user/cope/Patterns/Process/section16.html) that implores architects to get their hands dirty building systems—a best practice captured by Agile Modeling's maxim, Prove It With Code. This "in the trenches" work grounds architects because they quickly discover whether their ideas actually work. It also helps gain respect from developers who realize that the architects aren't simply ivory-tower academics. This approach also benefits overall enterprise efforts because the architects feed their experiences back into the enterprise architectural models, evolving them over time. Modern businesses are grown, not engineered; hence, an evolutionary approach to enterprise architecture has a far greater chance of success than a traditional BDUF engineering strategy.
Agile architects must also be multidisciplinary, not just "agile data architects" or "agile object architects." Development teams not only need help understanding the enterprise conceptual model, they must learn about the underlying business processes, the existing and proposed technical infrastructure, pertinent business rules, and any business components or services available for reuse.
Agile—or Fragile
The decision to reengineer your enterprise data group to become an agile enterprise
group is a difficult one. Data professionals must recognize new ways to develop
systems that they'll need to understand and support. They must evolve into an
enterprise group that focuses on a myriad of issues, including but not limited
to data. Agile enterprise professionals don't become bottlenecks by making others
follow onerous procedures; instead, they empower others by documenting and publicly
displaying pertinent standards, guidelines, models and meta-data, and then supporting
development efforts through active involvement on project teams.
I'd like to thank Charles T. Betz, Scott Fleming, Michael M. Gorman, Alan
Inglis, Terry Moriarty, Susan K. Peekna and Dwight W. Seeley for their input
on the Agile Modeling and DAMA mailing lists.
