Waterfall Development - So bad its not funny any more

When is Waterfall development of any use to a project? First you need to look at what this wonderful process is, taking a quote from a Website for Project Managers and IT Directors "Waterfall development makes it easy to keep your project under control. It limits the amount of cross-team interaction that occurs during development, it's relatively easy to estimate, and it allows for greater ease in project management since plans aren't constantly being revised."

If you don't see atleast 3 major red warnings in that statement, stop now, as the rest of this article will mean nothing to you. Waterfall defines itself as a sequence of isolated independent functions, each of which feed into each other, they being

• Evaluate the problem
• Propose a solution
• Design the architecture
• Develop the code
• Test the solution
• Deploy and use the system
• Maintain the solution

It is difficult, if not impossible to evaluate the complete solution at the start of the project, without first carrying out detailed analysis of what is being asked for, and also looking at specific options through prototypes or exploratory development. By leaving testing to so late in the process, a change or defect resulting from testing could potentially affect the whole design of the solution, and therefore the whole basis of the development, resulting in costly changes to design and code that have been signed off much earlier in the project.

Also testing so late in the process means that many of the initial resources that created the analysis and design models have moved on to other projects and therefore unavailable to justify their original assumptions, or even validate changes that need to be made as problems are found in the development and testing phases. Finally what happens if half way through the development process, the business needs to alter to counter changes in the market or business sector in which they exist? Do you start again, re-evaluate the whole problem, re-design the solution, then re-code and re-test the deliverables.

From the above information we begin to see that Waterfall is fundamentally flawed, however this is just one view, the technical one, and does not take into account the business side of Waterfall development. For that we need to look at which organisations favour a Waterfall development process. Any organisation that bases its business model on the amount of change that must be managed with such a rigid process will always propose waterfall.

One of the apparent strengths of waterfall is that it is relatively easy to estimate; unfortunately this is not true. It is almost impossible to estimate the size of any IT project. However certain organisations do, and they then propose a fixed price contract, where any changes to the schedule or deliverables are managed through a change request process. The contract around these bids are heavily weighted in favour of the bidder such that any changes must be paid for by the customer because the customer was unable to define everything they needed up front before offering the bid to tender, or for many different reasons, their original needs change.

I don't think I have ever seen an IT project, where all the requirements are fully understood by the business at the start, or will not change due to internal or external influences to the business, internal or external politics, or they fact that the closer you get to delivering something, the more you begin to engage stakeholders and they begin to understand just what they can do with what they will receive.. These changes are often unavoidable due to the rapid advances that occur into today's market.

What this means is that an organisation taking the lowest fixed price bid, will then have to deal with a huge number of change requests, all of which have a price attached, and can often double or triple the final cost of the project.

This is how major consultancies operate, they know full well change is the only certainty in an IT project, and therefore their business model is structured around winning low value fixed price bids, that are ramped up over time through change requests.

So what is the solution? Any software development project has distinct phases, as defined above, but what is required is a more flexible use of these in a framework which

• Provides rapid feedback of progress.
• Can pro-actively re-act to changes by the customer.
• Deals with a risk first approach, so that big problems that could affect the whole project are dealt with in the early stages of the project.
• Provide gated milestones on which the progress can be measured.
• Requires continuous and constant customer input and feedback to ensure the project is delivering what is required.

This approach can be summed up in the beliefs and approaches put forward by the Agile Manifesto ( http://www.agilemanifesto.org/ ) who all believe there is a better approach to software development and have spent many man years proving it. The approach can be summed up in the following points

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

This breaks down into an approach centred on Iterative Development. A project is broken down into lots of small steps. Each step may look like a Watefall project in its own right, but only lasts 1 - 2 weeks. At the end of the iteration, a fully working system is available, in the early stages, only as a prototype to view progress, but as the project continues the system becomes more and more like the end product.

However by breaking down the project into lots of mini sub projects a number of benefits arise. Every few weeks a entire project is completed, management begin to receive continuous and repeated feedback on the entire project. This approach allows for a risk first development. The highest risks to the project are those which are addressed first in the initial iterations. The areas of greatest unknown are fleshed out and expanded on at the beginning of the project. In the extreme it quickly becomes apparent if the approach taken can actually deliver, or whether a significant change is required in the project. Compare this with a typical large scale Waterfall project, in which development may not be even addressed until months or years after the project started, what then happens if the initial assumptions for the development of the software, which will have influenced analysis and design, suddenly turn out to be unachievable.

By building a new version of the product, each week or month, customer interaction becomes a stable part of the project, rather than a customer stating the requirements, and then waiting for development to be completed, with Agile they become part of the team, they become involved in all of the decisions within a project, providing feedback and correction as the project continues.

Finally change can occur, its a fact of life so get used to it. In Agile change is accepted, in fact to certain degrees it is encouraged and controlled. As discussed in this article already, change can come from many directions, and an iterative approach means provides the ability for the project to accept it, and then change the focus and/or direction of the project as required.

Agile development, rather than present an idealistic approach presents a pragmatic approach in which ever changing needs and drivers are recognised. It also recognises the fact that software development is still in its infancy although maturing slowly. It is time the software industry faced up the realities of software delivery. Projects which take months or years to deliver, in an industry which measures itself in weeks between technological change and revolution, need to adapt or die.