Measure Me Wisely

The primary measure of a software development project's success is the delivery of working software. At the end of each iteration, agile developers deliver working software, at least into a demo or testing environment, so their stakeholders can clearly determine whether progress is being made and money is being spent wisely.

However, although working software answers the question, "How well is the team progressing?" it doesn't deal with financial soundness, deadline or quality issues. For these, you need other measures.

The Break-Even Point

[click for larger image] The Break-Even Point for a Traditional Project The break-even point metric can bolster your argument for an agile approach. Here is the cost-benefit curve for a fictional 16-month project with a single release. With this approach, you incur development costs for 16 months, at which point you start to recoup those costs until you reach the break-even point at 26 months. Compare this with the same system delivered by an agile team taking an incremental approach to delivery.

On the Plate

Another critical metric involves the amount of work remaining (see www.controlchaos.com/about/burn down.php for a sample work burn-down chart). In agile projects, requirements are managed as a prioritized stack, and each item in the stack includes an up-to-date estimate of the amount of effort necessary for implementation to occur. An effective burn-down chart tracks both the amount of work required to implement a particular iteration and the work needed to complete the entire project. With these metrics, you can better determine a more accurate estimate of your project's end date.

Quality Counts

You can't deliver a system into production unless it's stable, and a critical determinant of stability is the number of defects remaining in the system. Both traditional and agile project managers commonly gauge stability by tracking the reported defects from system- or acceptance-testing efforts. "Defect Trends Analysis" shows how high-, medium- and low-priority defects are tracked over time. You shouldn't deploy your system until you see a clear downward trend in defects, indicating that system quality is improving over time; nor should you deploy until the overall number of defects has fallen to an acceptable level.

Agile Measurement

[click for larger image] The Break-Even Point for an Agile Project The agile team in this scenario delivers the first version of the system at eight months, implementing the highest-priority requirements up to that point. Because this system is released earlier, it starts earning revenue earlier, even though development is still in progress. The result? The agile team breaks even after 14 months, a full year earlier than the traditional team.

Second, recognize that metrics can kick-start verbal communication. For example, a metrics report could reveal 49 outstanding defects in your system, down from 55 in the previous week. Yes, the defect trend is headed in the right direction, but you also need to know how those defects are affecting the system in practice and why you have them in the first place. A report can't tell you this information, but the people involved can.

Third, be aware that feedback must be rapid. Forget the belated suggestions to improve a feature that had subsequently been scuttled. The faster you get the information you need, the faster you can act on it.

Measured Judgment

Traditionalists may tell you that you shouldn't use metrics to judge people, but I suspect they're protecting their metrics program, not their people. When you're doing a good job, you want to be judged, don't you? You may receive an annual bonus based on a metric that takes your personal performance, your division's performance and your organization's overall performance into account. Furthermore, many people are paid on an hourly basis, so the "hours worked" metric is extremely important to them.

Be open and honest about the metrics you're collecting and what you're doing with them. If staffers understand that their timesheets are used to bill clients or help track project costs, they'll be more likely to input their hours accurately. However, if they think they'll be punished if they don't input what management wants to see, they're virtually guaranteed to offer false information.

[click for larger image] Defect Trends Analysis You shouldn't deploy your system until you see a clear downward trend in defects.

A metrics program is like a hammer: You can use it to create—or destroy. In my experience, many metrics efforts seem more destructive than constructive, but with careful planning, quick feedback and attention to the human factor, an agile approach to metrics can serve as a springboard to more effective communication and project success.