Maybe agile adoption is still too small and most companies are running single pass waterfall projects, or have command-and-control project managers? Well, not if you believe Forester Research and Dr Dobb’s. They indicate that agile methods are now used more than any other approach for IT projects. With 35% of companies using agile and a further 21% using some kind of iterative approach and only 13% using a waterfall approach.

 
 
So if 56% of projects, back in 2009, were already getting the adaptive feedback benefits of an iterative approach why are the CHAOS report findings not getting any better? Also the increase in agile continues and last year Gartner predicted “By 2012 agile development methods will be utilized in 80% of all software development projects”. So why the poor results, and tell me again why the failures we see are PMI problems caused by “their predictive approach” when it appears these approaches are already in the minority?

I have been bothered by the apparent lack of improvement in CHAOS report findings for about 5 years; because I thought the increase in agile methods should have made a big difference. I also use them as an example of why there is a need for companies to switch to agile methods, if the current approach is not working so well, maybe we should try something else.

In the early days of the Agile 200x Conferences we had 150 to 300 attendees and they tended to be the innovators and early adopter organizations. However, in the last 5 years, the conferences have swelled to 1,200, 1,500 and more attendees as agile clearly crossed the chasm. Attendees are not just from software companies anymore, but local government, utilities and risk adverse insurance companies are all using agile methods, and yet those CHAOS report findings hardly budge.

Shown below are the CHAOS Report findings from 1994 to 2009.

We can see there has been a slight increase in Successful projects, but no “hockey-stick” shaped uptick to mark the arrival of agile or even anything close to the 35% increase associated with agile adoption by 2009. After doing some research it became apparent that their definition of success is quite different from that of the average sponsors.

The CHAOS reports definition of successful is not “functionality within +/- 20% of budget or schedule” as you might think. Instead they calculate a project’s success measure by counting the number of projects that have an initial forecast larger than the actual for cost and time, and one that’s smaller for functionality. This is divided by the total number of projects to calculate the success rates. Standish Group defines its success measure as a measure of estimation accuracy of cost, time, and functionality.

If that sounds confusing then it is appropriate, while their math might be statistically defendable, it is more a measure of estimate variance within a set of projects than happy or disgruntled customers. Maybe our measuring stick is broken, or has a funny scale?

 

That said many useful findings emerge from the CHAOS reports. I attended a great presentation by Jim Johnson from The Standish Group a few years back where he described their studies into the successful projects they found. Here we find some useful endorsements for agile methods. The following table shows the top ranked factors for project success.


 
User Involvement features prominently in agile methods as do Small Milestones. Also agile’s frequent interaction with business and product demos contribute to surfacing Clear Business Objectives. All of these factors contributing to successful projects feature prominently in agile methods.

Returning to Ken’s post, I agree with his initial comment of welcoming the PMI’s adoption of agile methods. There definitely is a need to embrace processes that better fit today’s knowledge worker environments. Predictive planning does not work in high change environments, and command-and-control management is no way to engage or motivate a team. In short, project management needs to move on.

Yet to attribute today’s IT failures, by CHAOS standards, to PMI processes is likely invalid, when agile is now the most widely used IT approach. Depending on your view this association can be seen as “optimistically naïve” because the causal link is weak, or “cynically canny” since the CHAOS Report findings will not change dramatically until they adjust their definitions of success.

For illustration a similarly afflicted statement would be: “I welcome Ken’s endorsement of the PMI initiative and think the test of this support will be, of course, the success of agile methods in solving global warming, which in the past has been less than successful”. The link is just not there.

This post is a bit of as departure for me; I don’t want to get engaged in flame wars and am indebted to Ken for teaching me about Scrum. Yet because we see the CHAOS report findings widely publicized as justification for change, I wanted to share some caution about their application. Also as agile methods become the norm, criticisms about the status quo are criticisms of agile.

I see the irony and duplicity here; when agile was small the CHAOS statistics were the rallying cry for change. Now they are addressing agile projects I am calling them questionable. As they say there are lies, damn lies, and statistics. I guess we need to choose our statistics well or be willing to flip-flop to sets that suit our arguments. Jim Johnson from the Standish group often gets asked what “CHAOS” stands for, one individual suggested “Calling Hallelujah Always Offends Someone” which seems apt here.
 

    1) The Rise and Fall of the CHAOS Report Figures, IEEE Software, 2010
    2) The CHAOS Chronicles, Jim Johnson, 2003