Channels ▼

Measure Me Wisely

The primary measure of a software development project's success is the delivery of working software. At the end of each iteration, agile developers deliver working software, at least into a demo or testing environment, so their stakeholders can clearly determine whether progress is being made and money is being spent wisely.

However, although working software answers the question, "How well is the team progressing?" it doesn't deal with financial soundness, deadline or quality issues. For these, you need other measures.

The Break-Even Point

[click for larger image]
The Break-Even Point for a Traditional Project
The break-even point metric can bolster your argument for an agile approach. Here is the cost-benefit curve for a fictional 16-month project with a single release. With this approach, you incur development costs for 16 months, at which point you start to recoup those costs until you reach the break-even point at 26 months. Compare this with the same system delivered by an agile team taking an incremental approach to delivery.
One of the most important measurements you can make is an estimate of your project's financial break-even point—the moment when the total revenue or savings generated by your system matches the cost of developing and then running it. The sooner you reach the break-even point, the lower your financial risk. This is something that stakeholders understand and are keenly interested in—after all, it's their money you're spending.

On the Plate

Another critical metric involves the amount of work remaining (see down.php for a sample work burn-down chart). In agile projects, requirements are managed as a prioritized stack, and each item in the stack includes an up-to-date estimate of the amount of effort necessary for implementation to occur. An effective burn-down chart tracks both the amount of work required to implement a particular iteration and the work needed to complete the entire project. With these metrics, you can better determine a more accurate estimate of your project's end date.

Quality Counts

You can't deliver a system into production unless it's stable, and a critical determinant of stability is the number of defects remaining in the system. Both traditional and agile project managers commonly gauge stability by tracking the reported defects from system- or acceptance-testing efforts. "Defect Trends Analysis" shows how high-, medium- and low-priority defects are tracked over time. You shouldn't deploy your system until you see a clear downward trend in defects, indicating that system quality is improving over time; nor should you deploy until the overall number of defects has fallen to an acceptable level.

Agile Measurement

[click for larger image]
The Break-Even Point for an Agile Project
The agile team in this scenario delivers the first version of the system at eight months, implementing the highest-priority requirements up to that point. Because this system is released earlier, it starts earning revenue earlier, even though development is still in progress. The result? The agile team breaks even after 14 months, a full year earlier than the traditional team.
How can you collect and report metrics in an agile manner? First, keep the process as simple as possible. Sharon Fay of Flashline describes the concept of "good enough" metrics (GEMs) in her recent paper. According to Fay, you don't need perfect measurements, so don't waste hours collecting minutely accurate data. Instead, spend just a few minutes gathering information that's good enough. Automate metrics collection as much as possible; if metrics are difficult to collect, people simply won't invest the effort to do so. Consider using industry statistics (assuming you can find anything relevant) until you have your own values to work with.

Second, recognize that metrics can kick-start verbal communication. For example, a metrics report could reveal 49 outstanding defects in your system, down from 55 in the previous week. Yes, the defect trend is headed in the right direction, but you also need to know how those defects are affecting the system in practice and why you have them in the first place. A report can't tell you this information, but the people involved can.

Third, be aware that feedback must be rapid. Forget the belated suggestions to improve a feature that had subsequently been scuttled. The faster you get the information you need, the faster you can act on it.

Measured Judgment

Traditionalists may tell you that you shouldn't use metrics to judge people, but I suspect they're protecting their metrics program, not their people. When you're doing a good job, you want to be judged, don't you? You may receive an annual bonus based on a metric that takes your personal performance, your division's performance and your organization's overall performance into account. Furthermore, many people are paid on an hourly basis, so the "hours worked" metric is extremely important to them.

Be open and honest about the metrics you're collecting and what you're doing with them. If staffers understand that their timesheets are used to bill clients or help track project costs, they'll be more likely to input their hours accurately. However, if they think they'll be punished if they don't input what management wants to see, they're virtually guaranteed to offer false information.

[click for larger image]
Defect Trends Analysis
You shouldn't deploy your system until you see a clear downward trend in defects.
The true challenge of metrics? You must be willing to accept answers you don't like and act on them accordingly. For example, if your work burn-down chart indicates that your team isn't going to deliver on time, you should use that information to motivate people to find ways to work more effectively. Having your team drawn and quartered isn't an appropriate response.

A metrics program is like a hammer: You can use it to create—or destroy. In my experience, many metrics efforts seem more destructive than constructive, but with careful planning, quick feedback and attention to the human factor, an agile approach to metrics can serve as a springboard to more effective communication and project success.

Five Traditional Tips

Here is some traditional metrics advice that still makes sense for agile teams:

  1. Trends—are we improving?—are far more important than exact values.
  2. Metrics don't need to be perfect, just good enough.
  3. Combine metrics to reveal different facets of a project.
  4. You can't manage by numbers alone; you still need to interact with team members.
  5. Automate metrics gathering as much as possible.

Nonmeasures of Progress

Because traditional software development teams take a mostly serial approach to development, resulting in software being delivered late in the lifecycle, many traditional metrics programs are forced into using questionable progress measurements. You know you're in trouble when progress measures are based on documentation. Systems have been known to fail even when they have detailed requirements documents, architecture documents or project plans.


Scott W. Ambler is a Canada-based software process improvement (SPI) consultant, mentor and trainer with AmbySoft Inc. He has written several books and is a regular speaker at software conferences worldwide.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.