Why Agile Projects Fail
The purposely provocative title of this article is meant to capture the attention of people with two opposing mindsets. The first are the proponents of agile methods who will want to pick apart any arguments against their preferred approach. The second are those who oppose agile and are looking for justification for their concerns. The goal of this article is to reduce the opposition between these two schools of thought, and to provide some food for thought to those who are undecided.
The truth is that development projects following agile methodologies have failed. Projects following waterfall and RUP have also failed. Just as the cliché that one should not judge a book by its cover is often true (I missed out on the intellectual pleasure of Robert A. Heinlein's Friday for many years due to that primitive prejudice), one should also avoid drawing a conclusion about a topic based on the title alone (The Art of War is more about how not to fight).
I will not go into deep detail of agile methodologies today. There are too many to cover adequately in a single reading, and the specifics of these methodologies do not have a direct impact on why agile projects succeed or fail. Agile is as much a way of thinking about projects as it is a set of tools that can be applied to projects, and the aspects that make up successful thinking can apply to agile and non-agile methodologies.
Both definitions of agile in one online dictionary include the word "quick." The word agile evokes thoughts of athletic prowess and graceful, fast feline predators. It is not surprising the people who chose to adopt agile methodologies for the first time do so when facing a tight deadline. Perhaps choosing a methodology by its name will be the new cliché.
People who have followed agile methodologies for multiple projects will tell you that they provide quick results, and can show you artifacts that prove it. What they may have forgotten (or simply neglected to mention) is that successful agile projects have a few advantages to be successful with agile methodologies.
For an IT shops' first agile project to be successful, the project must be the right project for that shop to adopt agile techniques. The project should be small, in both time frame and team size. The managers of the project must be willing to either adopt agile deliverables as they are, or adapt their method of measurement. Most importantly, the project team either needs to have enough members already experienced in agile or be given the latitude to get through the learning curve.
Agile methodologies rely on continuous and open communication among all levels of stake holders. For traditional waterfall groups, it is unheard of for a developer to contact a business sponsor or client to find out what is meant by a particular requirement description. In an agile project, the ability to do so can mean the difference between success and failure.
Agile methodologies can be used successfully to deliver a large project, but not by a team that is learning to use these techniques for the first time. Large projects often include many team members, at multiple locations. An IT shop can grow a small agile team to a large, distributed team over a few projects, but it is likely to be disappointing to try to assemble a large team from scratch, even if all of the members are experienced in agile. Like many adjectives, agile has different meanings to different people in different contexts, and two highly experienced agile project members could have entirely opposing approaches. Large agile teams must be grown organically rather than assembled randomly. These caveats to applying agile projects are most likely the root of the general consensus that agile is not appropriate for large projects. Perhaps this caution is a good idea when introducing agile methodologies to a team for the first time.
In describing the ingredients of projects that can succeed with agile above, it is important to note that the different modal operators of probability used are chosen specifically, rather than simply to provide variety in wording. Project managers new to agile must be willing to adapt. Teams new to the agile approach as a team need to either have enough members with agile experienced or be given the opportunity to stumble and learn. Communication across stake holders can impact results. There are exceptions, such as when those with roles that include being the translator of business requirements to technical requirements have good relations and communications with business and development and are highly competent in those communications.
This section began with a reference to a dictionary definition. In addition to the current English meaning of the word, most dictionaries include the history of a word, its etymology. At that same reference link, the etymology of agile is given as "from Latin agilis, from agere to drive, act." So, apparently the meaning of agile in the English language evolved from its origins to include the notion of "quick." Agile methodologies can evolve within an organization to be what we want them to be, though it is unlikely that they can be exactly what is desired the first time out.
The introduction mentioned that there are opposing mindsets about agile. It also included a reference to The Art of War, and these are not coincidences. An agile project team with members of both camps has two strikes against it from the start. It only takes one more strike to be out.
Many proponents of agile tend to be over zealous in their belief of the superiority of their preferred agile methodologies. Although their enthusiasm may win support from the undecided, it also can cause those opposed to agile methodologies to push back even harder. It also can frighten those who are participating in an agile project for the first time, because agile methodologies are very different from waterfall approaches and change causes anxiety for many. Where patient mentoring will frequently overcome FUD (Fear, Uncertainty, and Doubt), ignoring the FUD of others generally results in greater FUD.
Most opponents of agile are against it because of FUD. Sometimes rightly so, because agile is not a panacea and is not the perfect methodology for every combination of projects and teams. Those most opposed to agile methodologies are non-developer team members. Experienced waterfall project managers and analysts are highly trained in creating complete definitions of deliverables and measuring progress towards those deliverables with specific tools. Even though agile is still a measurable methodology, the measurements are defined as the project progresses instead of up front. This can be too far outside of the comfort zone of some people. And, people who are uncomfortable will often do whatever it takes to become comfortable again. For some, that is achieved by adapting and learning—by becoming agile themselves. For others, the approach is to make every effort to bring things back to the way they are used to. These latter individuals most likely do not do it consciously, but they will sabotage an agile project. How? By doing exactly as they are trained. By trying to capture every minute detail for a story definition rather than outlining it and providing feedback during the daily testing. By trying to fit too many features into a single iteration. By thinking a milestone was missed because an iteration did not deliver all of the features planned. By not delivering requirements that are completely defined because the due date for the requirement has not yet arrived on the calendar. By running an agile project in a waterfall mentality. In short, by not being part of the agile process by being agile themselves.
To repeat, those who oppose agile processes and sabotage agile projects most likely do not do it consciously. If everyone isn't on board with the agile process at the beginning of the project, it still can succeed. If everyone isn't on board with agile by the end of the project, that project will most likely have been a failure. If everyone pulls their own weight, but pull in a different direction, something is bound to come apart.
Those who have delivered successful agile projects have good reason to be confident in their approach. Successful agile projects result in solid software, often delivered either under-budget or with more features than originally projected. The iterative approach of providing rapid, demonstrable deliverables quickly builds enthusiasm for both the clients and the developers. Those who began the agile project with FUD and then learned to adapt and adopt new processes are those who become the biggest supporters of agile. It is important that they remember that they had their doubts at first if they are to convince others who suffer FUD that they, too, can become successful agile project members.
Hopefully, you realize that this section has many redundancies to the first section. Project deliverables may be software and documentation, but it is people who deliver them.