Data management is a critical success factor in all IT organizations, yet as an industry, we really don't have good figures indicating how effective we are at it. In July 2006, DDJ ran a survey that explored the current state of data management within organizations. The results likely aren't a surprise to youwe're clearly facing some serious challenges. Because the first step in addressing a problem is recognizing that you've got it, I believe that this survey reveals some important deficiencies that organizations need to address.
We're Not a Big Happy Family
In total, 60 percent of organizations had a data group, although that figure rises to 80 percent for organizations with 50 or more IT professionals. In organizations with a data group, two-thirds of respondents indicate that sometimes developers go around the data group and address data issues on their own. Why is this a problem? Because on their own, developers do a less-than-perfect job of database design. The "command-and-control" solution to this problem is to put processes and organizational structures in place to force developers to work with their data groups. Sounds great in theory, but it doesn't seem to be working well in practice. A more practical approach would be to recognize that if developers are doing database design, then we should help them gain those skills through training, mentoring, and pair development. Sadly, in the organizations with a data group and where developers sometimes choose to go around the data group, only 34.2 percent provide data-oriented training to developers.
Figure 1 summarizes the motivations of development teams that go around their data groups. The good news is that roughly 25 percent of the problem can be easily fixed through education of developers: 8 percent don't know that the data group exists and 17 percent don't know that they're supposed to be working with the data group. The bad news is that the other problems aren't so easy to address: 20 percent of developers find data professionals too difficult to work with (although to be fair, many data professionals find developers difficult to work with as well); 36 percent of developers believe that data groups are too slow to respond; and 19 percent believe that data groups offer little value to them.
One way to address these problems is to promote greater understanding between the groups: If developers understand the basics of data management, then they would likely recognize the value that data professionals have to offer and would more likely be able to find ways to work with them effectively. Similarly, if data professionals understood modern development techniques such as refactoring and agile modeling, and modern methodologies such as Extreme Programming (XP) and Open Unified Process (Open UP), then they would be in a better position to work in a more responsive manner. The survey indicated that many organizations can benefit from these strategies. Currently, only 36.4 percent provide data training to developers and 44.2 percent provide development training to data professionals.