Channels ▼


Whence Data Management?

Quality Concerns

It should be no surprise that 95.7 percent of organizations considered data to be a corporate asset (although it is surprising that 4.3 percent don't). If data is a corporate asset, doesn't it make sense that you have a test suite in place to validate it? Apparently not, because only 40.3 percent of respondents indicated that they do. Worse yet, of those organizations, only 63.3 percent let developers run this test suite whenever they needed to, hampering their ability to detect whether their development efforts would inject defects into the database. Of the organizations that didn't have a test suite, only 31.6 percent had discussed putting one in place, implying that 40.8 percent (59.7 percent * 68.4 percent) of organizations are seriously challenged with respect to ensuring data quality.

The problem just gets worse. 63.7 percent of respondents indicated that their organizations implement mission-critical functionality in the database, yet only 46 percent of those had a regression test suite in place. If functionality is mission critical, or even if it isn't for that matter, shouldn't you test it? Similar to data quality testing, only 66.3 percent of respondents work in organizations where developers can run this test suite whenever they need to, and in organizations without such a test suite, only 38.6 percent had discussed putting one in place.

Although these numbers sound bad, and they are, I suspect that they're optimistic. The survey didn't distinguish between traditional regression testing where the majority of testing is done late in the lifecycle and the more agile test-driven development (TDD) approaches where testing is done throughout development on a continuous basis. A survey being run in September addresses this issue, and more, and will be summarized in early 2007.

Have We Given Up?

61.9 percent of respondents indicate that their organizations have problems with their existing production data. Although this number is arguably low, very few data sources are perfect, and we can often live with minor data problems. However, considering that most organizations consider data to be a corporate asset, shouldn't we be doing something to fix it? As Figure 2 reveals, many organizations seem to be struggling with addressing legacy data problems.

Figure 2: Strategies for addressing production data problems.

Of the respondents working in organizations with data problems, 18 percent report that there is no strategy in place to address the problems and 33 percent have strategies not to make things worse. In my opinion, these two strategies will both eventually lead to failure: With developers commonly going around data groups and often doing a questionable job of database design as a result, and with business users using existing applications to do new things that weren't considered in the original data design, things are bound to get worse. 8 percent of organizations indicate that they intend to rewrite everything at once, a strategy that I suppose could work for smaller organizations. The good news is that 33 percent indicated that their organizations are taking an evolutionary approach to fixing data sources, which in my opinion is the most viable approach.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.