Channels ▼


We Need More Agile IT Now!

There is more to IT than software development. Not only do we build software-based solutions for our stakeholders, we also release them into production and we operate and support them once there. There are also cross-solution organizational functions such as enterprise architecture, portfolio management, data management, and others that occur in parallel to software development and operations. Recently, I decided to research the question: How effective are these IT functions? I found that we have a lot of room for improvement.

In September through November of 2014, I ran an online survey that explored the effectiveness of several common IT functions. The survey was advertised in my Dr. Dobb's article Software Development is Very, Very Hard, via several Twitter posts, and at my IT Surveys home page. The survey had 117 respondents, almost half of whom had twenty or more years of experience in IT (88% had 10 or more years). The survey explored two fundamental issues: How easy is it to work with the people fulfilling various IT functions and how much value does that function provides to the organization. Table 1 summarizes the results for the six IT functions that the survey explored. The ratings are on a scale of +10 to -10, where +10 indicates that a group is very easy to work with or provides great value and -10 indicates that a group is very difficult to work with or provides little value to the organization.


Perceived Ease of Collaboration

Perceived Value

Data Management



Enterprise Architecture



IT Governance






Portfolio Management



Release Management



Table 1. How various IT functions rate. (Scale -10 to +10. Higher is better.)

As you can see in Table 1, the average scores were all near the center — as they would say in the UK: on average, the groups are muddling along. For every IT function, there were some respondents indicating that the groups providing that function were very easy to work with and some indicating that the group was very hard to work with. Similarly, each function had people saying that the groups provided great value to the organization and others saying the group provides little value. Given the mediocre average scores summarized in Table 1, let's explore how your organization could go about applying Agile strategies to improve each of these IT functions.

Improving Data Management

I've worked in dozens of organization over the past two decades helping them to improve their IT processes, and invariably the data management team has proven to be the black sheep of the organization.  This stems from the data management body of knowledge being out of step with the rest of IT industry, the majority of their "thought leaders" still clinging to the heavy-handed serial strategies of the 1970s. The primary effect of this cultural impedance mismatch has given us at least two generations of data practitioners with few concrete skills beyond physical data modeling and database administration. Two cases in point — the data management body of knowledge has virtually nothing to say about database testing, and very little to say about actually fixing problems in databases via a technique such as database refactoring.

The good news is that the Agile community has made great strides in Agile data techniques. There is a growing number of people doing test-driven database development, continuous database integration, Agile data modeling, and database refactoring. I am also seeing a healthy surge in interest Agile data warehousing/business intelligence (DW/BI) such as Data Vault 2.0, which I suspect is motivated by the poor track record of traditional approaches to such efforts and by the dynamic challenges posed by big data. A significant task faced by existing data practitioners, as you can easily surmise from this list of new techniques, is gaining the technical skills required to implement these practices. We've had a series of books come out in the last few years covering these topics, including Test-Driven Database Development by Max Guernsey III, Recipes for Continuous Database Integration by Pramod Sadalage, Agile Analytics by Ken Collier, and Refactoring Databases by Pramod Sadalage and me.

Improving Enterprise Architecture

Over the past decade, I've seen a gradual, albeit not complete, acceptance of Agile strategies within the enterprise architecture community. Successful enterprise architecture teams are adopting lightweight, flexible, and highly collaborative Agile strategies. Agile approaches move enterprise architects from being reviewers of architecture documents to being actively involved with both business stakeholders and software development teams. The hard reality is that enterprise architects who don't roll up their sleeves are most likely to be either ignored by development teams or will be blocked via teams that produce the architecture models they require solely to get through the architecture "quality gate."

As Dr. Dobb's Editor, Andrew Binstock adroitly observed in In Search of Agile Architecture, the Agile architecture space currently suffers from several fundamental challenges. First, one side effect of the de facto standardization around Scrum is the dumbing down of an entire generation of software developers regarding software architecture and design skills. Yes, there are a few bright spots with books such as Lean Architecture by Jim Coplien and sites such as Manifesto for Software Craftsmanship and Agile Modeling. Sadly, these topics are rarely mentioned, let alone covered adequately, in those ubiquitous two-day Scrum certification workshops. Second, the Agile community still suffers from a culture that is often anti-modeling and against up-front thinking. This prevents many Agile practitioners from even considering Agile architecture practices regardless of how lightweight those techniques are. Third, most Agile thought leaders prefer to focus on other topics such as addressing cultural issues, continuous delivery, and DevOps — all important topics. However, this has the side effect of starving Agile architecture of face time at conferences, in blogs, and in developer-oriented publications.

Improving IT Governance

As you saw in Table 1, the organizational function of IT governance fared poorly. As with the challenges faced by the data management community, much of the IT governance literature seems to be based on 1970s waterfall thinking.  This style of governance is focused on a command-and-control mindset that promotes a series of quality gates where project artifacts (typically documents) are reviewed. One of the great ironies of such governance programs is that they themselves are ungoverned and almost always out of control — a sure sign that this is the case is that the governance programs rarely have metrics, or other feedback mechanisms, that focus on the effectiveness of the program itself. Another process smell indicating that your governance strategy is out of whack is when the governors claim that their approach is paradigm specific. The reality is that you govern traditional teams in a much different manner than iterative teams, and in a different manner than Agile or lean teams.

Agile and lean governance strategies are instead based on motivation, enablement, and short feedback cycles. The goal is to motivate those who are begin governed to do the "right thing," whatever that means for your organization, instead of trying to review it in after the fact. To increase the chance that the right thing happens the job of management is to make it as easy as possible to do those things, to enable them, and arguably to make it difficult to do the wrong thing.  Finally, both the governed and the governors should have access to meaningful, focused metrics that provide insight into what is happening and thereby enable them to take action to adjust their behaviors accordingly.

Improving Portfolio Management

Portfolio management encompasses a range of activities, typically including product and project identification, team initiation, and business roadmap creation. Sometimes it includes governance of ongoing projects and even governance of the operations efforts. Putting some aspects of IT governance in the hands of the portfolio management team can prove to be problematic for many organizations, particularly those that prescribe to the teachings of the Project Management Institute (PMI). The challenge with the PMI is two-fold. First, it is still in the process of identifying good portfolio management practices, which the PMI is very clear about. Second, the PMI tends to promote generic project management strategies that often prove to be heavy-handed and better suited to the physical construction industry (where they are dealing with laws of physics) instead of the IT industry (where we are dealing with patterns of human behavior).

Luckily, Agile strategies for portfolio management do exist, focused on collaborative lightweight techniques for high-level planning, project identification, and business roadmap development. Agile portfolio management techniques focus the human aspects of working together across teams; on initiating and guiding value-driven experiments, along the lines of the "Lean Startup" methodology of Eric Ries; aligning the work of various Agile project and product teams; and motivating investments to simplify your production/operations environments. As with IT governance, Agile portfolio management requires a shift away from a command-and-control mentality to more of a collaborative one. This is often much easier said than done.

Improving Operations and Release Management

Operations functions within most organizations tend to work well, although there is always room for improvement. The primary challenge is a disparate production environment, the result of years of years of architecturally uncoordinated software development projects and a poor understanding of operational realities by software developers. My guess is that about 90-95% of all organizations suffer from this problem. Another challenge, although luckily not as common, is the adoption of overly bureaucratic procedures, often stemming from a lack of pragmatism when adopting frameworks such as Information Technology Infrastructure Library (ITIL) or Control Objectives for Information and Related Technologies (COBiT).

The goal release management is to successfully guide deployment of software-based solutions into production. The release process in many organizations is still far too waterfallish, with checks and balances in place that reflect the questionable quality practices of traditional teams. As your development teams move to a more agile, quality focused approach your release management team will find that it can ease back on the restrictions that it places on Agile deployments.

The good news is that in recent years there are two healthy and related trends in the Agile community: continuous delivery (CD) and DevOps. The goal of continuous delivery, as the name implies, is for software development teams to deploy into production regularly. Instead of releasing into production once or twice a year, development teams taking a CD approach follow strategies that enable them to release monthly, then make the improvements to release weekly, then daily, and sometimes even several times a day. Note that they're not just releasing into testing environments, but instead into live production environments. Companies such as Netflix, Etsy, and Salesforce are reported to release into production many times a day. This enables them to react quickly to market demands, thereby increasing their overall competitiveness. DevOps focuses on how to streamline our software development and operations processes. This can include strategies such as continuous deployment, instrumenting applications so that new functions can be toggled on and off while running in production, instrumenting applications so that they provide real-time operational metrics, standardizing production environments (this is supported by Agile enterprise architecture strategies described earlier), and purposefully retiring technical debt to do so. See my Dr. Dobb's article Top 10 Practices for Effective DevOps for more detailed insights.

We Need Agile IT

The primary take away from the survey is that in many organizations there is significant room for improvement as to how they address common IT functions. We need to do so because the business side of our organizations are often moving faster, or at least want to move faster, than our IT departments can currently support.  In this article, I shared a collection of Agile strategies that help to address many of the common challenges faced by IT departments today. These Agile information technology (IT) strategies are quickly being captured and shared freely within the Disciplined Agile Delivery (DAD) framework. Please stay tuned.

Scott Ambler is a long-time contributor to Dr. Dobb's. Follow him on Twitter.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.