As it turns out, I do not see a problem with having the two, often contradictory, approaches available. The advice they embody are like toolkits to be drawn on as project and organizational characteristics demand. If faced with a custom software development project I would use an agile approach. If asked tasked with scheduling 500 staff through a training course, a traditional approach might be appropriate. I may use a combination of approaches on a single project, for instance agile software development, followed by a traditional deployment and training plan. I think they key is fully understanding the use and applicability of each approach and then making the intelligent selection.

However, many of the tools are do not successfully mix-and-match; for instance don’t try agile development from a set of formally signed-off requirements, and don’t make task oriented Gantt charts for distant iterations. The intelligent mapping and application of traditional and agile techniques is an area I believe we will see significant growth in the future.

Methodology zealots will promote their own approach and dismiss all others as ineffective and that’s OK, we need zealots to evangelize new methods and create clashes to drive debate. Sober analysis and selective blending of ideas does not sell many books in a world obsessed with the Next-New-Thing, but does allow organizations to leverage innovation and manage risk.

I could be a zealot; I was involved in the creation of DSDM back in 1994 and having been using agile methods (Scrum, FDD, XP, DSDM) full time for the last 12 years. However I’m also a PRINCE2 and PMP certified project manager who teaches for the PMI SeminarsWorld Program and was a contributor to the PMBOK v3 Guide. The trouble for me with being a zealot is I can see the other side of the argument, sometimes a structured tool work best, other times a self organizing empowered team gets the job done. I used to feel uncomfortable at conferences, not buying into a totally one-sided story, but I’m not alone, there are a growing number of people successfully blending a combination of agile and traditional approaches.

The problem is that the recipes are not simple; you really need to be master chef at each before successfully creating a good mix. I say this out of experiences of failure, not some lofty view point. I tried and failed many times at combining the approaches before getting reliable results. In the future I will share my ideas for selecting the right mix based on your project and organizational factors, and would like to hear other peoples views on these ideas too.

For now here are some thoughts taken from my 2004 paper for the PMI Global Congress:

“<snip>
Today’s Project Management Guidelines

Agile approaches are in contrast to traditional views on project management. The current PMBOK view of projects is dominated by the Transformation concepts of Taylor. It recommends early in the lifecycle, to break down the project into smaller and smaller pieces using techniques such as creating Work Breakdown Structures.

Current project management guidance is focussed heavily on planning. The overriding theme is that if you plan enough, track against the plan and take corrective action when work deviates from the plan, then you will be successful. If things go wrong then you probably did not spend enough time upfront planning and understanding all the requirements and risks. These recommendations are particularly problematic for software development projects where poor initial requirements and unprecedented technology combinations represent major production risks.

Project management activities, as defined by the PMBOK, are divided into the process categories shown In Figure 1.

Traditional project management guidelines are heavy on project Planning information, but light on project Execution and Control information. The assumption is that execution of the plan should be a simple matter of carrying out the tasks listed in the project plan. However, this is not the case for software development projects which have high risks associated with production.

The PMBOK promotes planning as the central role of the project manager and implies that communicating the planned tasks should be via a top-down dispatching model of work products. Control of the project is done via checking progress against the project plan and when necessary taking corrective action. This model of control is called the thermostat model. The assumption is that the underlying plan is correct and something has happened on the project that needs correction in order to bring it back in line with the plan.

The limitations of a management-as-planning approach, the dispatching model for task assignment, and the thermostat model of control are well documented by Koskela and Howell. The remainder of this explanation describes how agile approaches can be utilized alongside or instead of the PMBOK based approaches, for better project execution and control in software development projects.

Mapping of Agile and Traditional Project Management

The processes shown in Figure 1 will be used to group discussion around the project management best practices for software development. The following sections summarize author recommendations for managing software projects.

Initiating Processes
Recommendation: Use As Is: The early project activities focused on chartering and identifying preliminary scope work equally well for software development projects as they do for more defined repeatable projects. While writing the Project Charter, a description of the approach that will be used to deliver the project should be made clear. When using agile methods, the concepts of iterative development and the production of regular increments of software should be clearly explained to all stakeholders as this may represent a different way of working for some people.

Planning Processes
Recommendation: Use with Modifications: Planning is a critical step for both traditional and agile project management. However, its use should not be concentrated upon at the beginning of the project as this is when least is known about the project. Instead, planning needs to be iterative and ongoing throughout the project; guided by feedback from actual project performance, business changes and technical knowledge growth.

This important point is made in the PMBOK through the concepts of Progressive Elaboration and Rolling Wave planning. Progressive Elaboration speaks to the need to refine scope and evolve plans as more information becomes available. The related concept of Rolling Wave planning which states planning should be iterative and ongoing based on short time horizons.

Unfortunately these concepts are often overlooked; a contributing factor may be the artifacts produced by early planning. Gantt charts and Work Breakdown Structure are difficult and time consuming to update for non trivial projects. Also the tools used to create them do not readily support wholesale refactoring. As a result, the majority of software projects managed with traditional techniques do not undertake the rigorous plan refactoring required to keep pace with the true project work. Either detailed task oriented plans become outdated, or high-level, phase oriented plans are produced that offer little value in way of project tracking or forecasting.

For software projects that contain requirements and technical uncertainty, plans should be feature based instead of developer task based. Attempts to predict the detailed activities required for problem solving will likely fail. Plans should instead be maintained at a high level for the entire project; outlining feature themes (general areas of development) and at a detailed level for the current and subsequent iteration, defining the features selected for development. This switch from tasks to features also affords better visibility into the value of functionality developed to date. Non-functional requirements and risk mitigation activities can be listed as features and assigned a business value by calculating the expected monetary value of the risk they have been created to mitigate.

Executing Process
Recommendation: Use Agile Techniques: The PMBOK is an industry independent guide to project management best practices. As such it contains very little about how to actually undertake the work on any project. The intention is that the appropriate “doing” techniques will be used from the industry in question. For example, there is a wealth of information available externally about how to run civil engineering projects. Therefore, for software projects the most appropriate execution processes should be employed. These include:

Develop iteratively – build software iteratively and incrementally to address the “evaluation difficulties” of software.  Iterative development also supports the rapid mitigation of technical risks by allowing elements of new technology to be trialed in early iterations rather than waiting until an integration phase towards the end of the project and then discovering issues when there is little time left of their resolution.

Iterative development also provides many additional advantages over single-pass approaches, including: better progress visibility, learning opportunities, and options for early benefits realization via interim releases.

Use meaningful metrics – The act of measuring a characteristic often influences its value, this is a trait called the Hawthorne effect and it should be noted and used to good effect on software projects. Also, the metrics employed to track software projects should be “Simple and relevant to the goal”.

Attempts to track metrics that are not relevant to the goal such as Lines-Of-Code developed or developer-hours-worked are likely to lead to an increase in the lines of code written and increased hours worked (due to the Hawthorne effect). These are not relevant to the true goal of delivering valuable software and an increase in the lines of code written is undesirable as it leads to increased complexity and maintenance effort. Likewise increased hours worked can to lead to developer burn-out and an increase in defect rates.

Instead, metrics that are aligned to the true goal should be used as the primary measures. This is why metrics such as features-delivered and project-time-remaining are frequently used as agile project management measures. Features-delivered can be illustrated with a Cumulative Flow diagram as shown in Figure 2.

In Figure 2, time is shown on the X axis and the total number of features to develop is shown on the Y axis. Increases in scope can be seen as steps in the overall features to develop and progress shown as the areas and gradients representing development work.

Empower the team – A traditional top-down, dispatching model for allocating work is problematic for software projects that are hard to estimate and often contain unpredictable dependencies. Attempts to micro-manage developers by assigning individual tasks are likely to fail. Instead it is better to leverage people’s ability to manage complexity by assisting in the definition of high-level goals and objectives and then empowering the team with the responsibility of achieving the goals.

In practical terms this means engaging the team and business representatives in the planning of features, then letting the team select and develop the features for that iteration as they see fit. Progress may be made sequentially through features, in parallel, or via a combination of both as issues and dependencies are uncovered and solved. Overall progress can be tracked via the delivery of completed features but the complexity of feature dependency resolution is best managed by the team.

Controlling Processes
Recommendation: Use Agile Techniques: The controlling processes described in the PMBOK include “Controlling the project” and “Controlling changes”.  The core theme is looking for variances from the project plan and then taking the appropriate action. This is a thermostat model of control and implies the project plan is correct and a deviation from it requires intervention.

In software projects where there is a high likelihood that the initial plans are flawed due to evaluation difficulties and execution risk driven changes, a thermostat model is not appropriate. Instead scientific experimentation and root cause analysis is required before correcting the plan as opposed to correcting the project. In unprecedented software projects, plans represent today’s best forecast of how we think the project will unfold, as opposed to a defined path that must be followed.

Iteration Reviews – At the end of each iteration progress, requirements, risks and process effectiveness are reviewed. Having these regular checkpoints assists with project steering and provides updated trend and goal setting information.

Change Requests, Defects and Risks – Change requests and defect reports can be prioritized in with the list of remaining features to be developed. This way the impact of scope creep is immediately apparent as new features need to be traded off against existing features planned for an iteration in order to stay within the iteration timebox. Using a similar approach, risks mitigation actions may be inserted into the prioritized feature list based on the expected monetary value of the risk the will avoid or mitigate.

Controlling Flow – By taking a holistic view of the development process, bottlenecks or constraints in the feature delivery may be identified and removed. In this way the delivery of features to the business can be maximized and the principles of Lean production and the theory of constraints utilized. Cumulative flow diagrams can be used to identify constraints in the development process. Widening portions in the graph highlight a slower activity and can alert managers to the need to exploit the constraint.

Closing Processes
Recommendation: Use As Is: The Closing Processes defined in the PMBOK provide good general guidelines for activities to consider when closing software development projects.

Summary
Many software projects possess characteristics that challenge traditional approaches to project management including volatile requirements, evaluation difficulties and significant execution risks. Agile approaches address these issues through iterative development, goal seeking, and leverage the extreme modifiability of software. When undertaking projects with high execution risks, agile methods should be used alongside the appropriate traditional project management techniques to provide additional project execution and control mechanisms. Agile approaches to project management also offer alternative tracking and reporting metrics which do not create large change workloads as projects progress. "

You can download the full paper here.

Download using_agile_alongside_the_pmbok.pdf