Agile From The Trenches - A real world example

My last project was a 12 month green field development of a new broadband provisioning platform for a major ISP/Telco in the UK. I was the 2nd person to be recruited as Development Manager, the one other person recruited was a developer who was carrying out initial prototype work for the Enterprise Architecture team to prove that what was wanted by the business was actually achievable. The interview, in June 05, was pretty short, basically it went along the lines of "The business want X delivered by 1st Nov and they are not sure what X is, can you do it ?"

Always one for a challenge, how could I turn this down? All I could repeat during the interview was that if I joined, I would commit to delivering something by the 1st Nov, if and only if

1. The requirements were not fixed
2. Cost and resource were not excessively constrained
3. The project adopted Agile development processes to enable delivery

The company was already dabbling in Agile and had used an Agile consultancy in the past on other projects although results had been mixed. However we had no strong connections with the consultancy other than using 1 or 2 of their staff as back fill. What therefore follows is a synopsis of my project notes.

Please note, before we go any further, that this is not a detailed discussion of Agile development. For that there are plenty of other websites and papers published. I have no intention of discussing Unit tests, Ant, Cruise Control, Continuous Build and release and the multitude of other concepts associated with modern day software development. If anyone wants this kind of paper, feel free to contact me. However instead, given the nature of the forum, I am trying to focus on how we discovered, used and delivered requirements. Suffice to say we used all of the above and more during delivery.

Background

The nature of broadband internet in the UK has been going through change in the past 3 years. British Telecom ( BT ), the incumbent supplier of copper to the vast majority of UK households have been forced, by the government, to open up their exchanges in a process known as Local Loop Unbundling ( LLU ), whereby ISPs are allowed to install their own broadband kit into a BT exchange and thus provide broadband to UK house holds directly rather than through wholesaling from BT themselves.

As you can imagine this has effectively meant the removal of monopolistic control over household broadband in the UK and has seen the UK become one of the leading users of broadband in Europe. LLU has paved the way for a new wave of investment with the opening of the market and introduction of next generation offerings. However it is not without its problems and for those who read The Register, you will have seen the numerous problems that many ISPs have gone through during the move to LLU provisioning, never mind the growing number of user based complaint websites such as:

• Orange Problems

ISPs have invested heavily in new hardware and software systems in order to offer their own broadband services. Projects of this size are never without problems. Also within the UK at the moment there is major land grab by the ISPs, as can be seen by the number of free internet services being offered along with paid for phone services :-

• Carphone Warehouse
• Orange
• Sky

This land grab is at the forefront of what is likely to be a major consolidation of ISPs over the next few years. Most analysts believe that the 20 or 30 large scale broadband service providers are likely to be reduced down to no more than 4 or 5 through partnerships, takeovers and business failures.

The business which recruited me had spent a number of months examining the sort of approach required to enter into this turbulent market, and there had been an architect element involved for at least 3 months, the result of which was the production of an architectural blueprint for the overall system. This was very high level and was a complete end to end view where many of the components we needed to build were nothing more than boxes on a Visio diagram.

To give an idea of the scale of the project, the business was already wholesaleing over a million broadband lines, and was planning to migrate at least half. This required the delivery of a complete LLU solution made up of a number of distinct systems, including :-

• A migration system to migrate the above mentioned existing ISP's customers from BT systems to their own. This required several interfaces to BT systems to coordinate exchange by exchange migration.
• A service management system to manage the on going service offered by the ISP.
• An inventory management system to handle the huge amount of hardware and networking equipment required to provide broadband services
• Activation System
• Fault management systems to allow front line support staff to diagnose faults, and then raise and manage the faults with backend support teams and 3rd party providers.
• Integration into existing OSS (BBMS)

In essence we were required to deliver a system in 6 months that was enough to operationally run a broadband provider. On top of this the business had produced what they considered a Business Requirements Specification, but this was huge, and was the entire set of requirements for their entire systems as would be needed over the next 2-3 years.

Where as the Business Requirements were a view of the Strategic system, one of the first decisions made on the project was that anything delivered by Nov 1st ( i.e 6 months away ) would be Tactical, would deliver the basic needs of the business and provide a framework on which to grow to meet the strategic needs over time. This was quite a big win at the time, and probably proved to be the key success factor of the project.

The Tactical system was considered disposable and only required to fulfil the limited set of requirements to allow entry into the LLU Market. The Strategic system would mainly consist of COTS components, but as this would require a complete RFP process and many months of vendor negotiations it would have been impossible to deliver anything Strategic in 6 months.

One final note is that there was always a view that the business never believed we'd ever deliver this, this was evident in the fact that the organisation created to deliver this piece of work was entirely 100% contract ( most on 5 days notice ), and thus gave the business the option of firing us all if they decided to pull the plug. This attitude also meant that it was often hard to get access to key stakeholders, but read on, it was damn good fun to work on......

Recruitment

The first step was to recruit the team, and we decided that in order to deliver the tactical components in the time frame required we needed to keep the team lean. We initially decided to only recruit staff with explicit experience of Agile development, and after one or 2 quick wins recruiting some excellent agilists we were starting to come across the fact that everyone has Agile on the CV and everyone has XP experience, this often amounted to the fact that they had either sat next to another programmer or had written some unit tests once. ( Sorry if you don't know eXtreme Programming, you'll have missed that joke !)

After some very frustrating interviews we changed direction slightly and started looking for people who wanted to get involved in agile development, may have never worked in a true agile delivery team, but were keen to. This eagerness meant we were recruiting some of the best people, who had been held backby companies not wanting to move to agile, and now we were giving them the chance. It was sort of a moral contract with them, the senior members of the team would teach/train/mentor them in agile development if they brought along an open mind and some good coding skills.

I think this is one of the key successes of Agile, the really good people want to work in Agile. They see that pairing, cooperation and communication is so much more fun than silo based development.

In addition a core component was the implementation of a Remedy based fault management system. Agile processes tend to be the domain of the Java and .NET world but we took the approach to use Agile across the whole programme of work. This led to some interesting times with the Remedy team who had come from a very traditional waterfall background.

Requirements

So we were starting to build a team, recruiting really good people. One of the first hires was a superb Technical Analyst, who although he had not worked in the agile space before, knew the problem domain inside out.

The first thing we started to do was rewrite the requirements, basically define a subset of requirements that would deliver the bare essentials of a tactical system. With these roughly understood we started holding some basic planning sessions to see if our view of what the business wanted would be achievable by 1st Nov, and also to validate that what we thought the business wanted and would go live with was correct. These were taken directly from XP Planning games and ran the same way, with as many people involved providing quotes for size.

You can imagine the results of some of these meetings, "how could we deliver so little ?", "why don't you just hire more people ?", "why don't you just deliver more ?", but with some explanations about limiting risk by delivering the absolute minimum and building on that, and giving the business a roadmap of all the remaining requirements post 1st Nov we begun to get an agreement and an understanding of the task ahead.

Note that these estimates were pretty much finger in the air, in some cases we had no idea how to achieve the requirement, but in these instances we gave them the highest priority to be addressed at the start of the project. Even if it just meant that the analysts looked at first and expanded them, we could immediately get a better view of the work required and either address immediately or downgrade in priority to be addressed later in the project.

At all times every requirement we wrote was traced back to one or more of the business requirements so that the business always knew what we were intending to deliver, could see how much we had developed, had in test or was complete and when future requirements would be delivered. This pretty much reflected a Scrum catalogue. In the later stages it also provided a valuable tool in managing change. When a new requirement was brought to the table, or an existing one changed ( or even dropped ) we could see exactly the impact ( and cost ) on the current system.

This matrix was a simple Excel spreadsheet with the business requirements on one axis and the tactical requirements on the other, ordered by release ( 1st if Nov being Release 1.0 ). An 'X' in a cell meant the requirement was due to be delivered and showed which business requirements it traced to.

Early Prototypes

While the initial planning was taking place, those staff that had already been recruited started working on prototypes or exploratory development of key requirements. This was often fed back into the planning sessions and in one instance caused a major concern when a fairly innocuous requirement turned out to be a major piece of work. Some of the prototypes were total throw always, just ideas to prove if something could be done, while some of them became the basis for actual deliverables. In fact the first deliverable of the project was an Expediant system. This has just enough functionality to prove what we needed to prove. It even placed live orders. However this was quickly thrown away as we started the real development.

Many of the prototypes were demonstrated in front of many different stakeholders during these early stages. Customer Service Representatives ( CSRs ) were given examples of the system they would manage users through or which their actual users would be using to sign up. These could be sometimes PowerPoint, sometimes actual rough mock ups of the screens themselves. Architects were given presentations of underlying concepts to ensure they fitted in with their enterprise vision. Business stakeholders attended weekly show cases of progress to date; initially this was often just discussing the results of operations on the business data as our key workflows began running, but eventually started to include use of screens and real world use of the system.

Iterations

The project was split into weekly iterations, giving us approx 21 iterations before we had to go live. This was mapped onto a single wall in the office with every requirement as a separate index card. Very quickly, at a glance, anyone interested in the project could see what had been delivered, what was being worked on in a particular week, and what was planned in the coming weeks.

We also introduced a simple colour coding system using small circular stickers. A card ( and therefore requirement and associated use cases ) either not started or just in development contained no stickers. Once it has completed development it was given a blue sticker. This effectively meant ready or in test. Once it had passed test, the blue sticker was covered with a green sticker. If the card failed test, the blue sticker was covered with a red sticker. The card progressed through the blue - red - green cycle until complete. Very quickly any one examining the board could see progress in great detail, a multitude of red dots showing problems very very quickly !!.

As stated before all requirements were prioritised, and this directed the placement on our wall. We knew roughly how many man days of development we could achieve each week, and could fill up each week with that number of requirement man days minus time for defects, testing and admin work.

This ordering also gave the analysts the list of requirements they needed to work on and in which order. We took the decision to document each and every requirement as a Use case. We now had 2 choices, wait for every use case to be delivered so that we had a view of the entire system, or work on the key requirements now, get these to the development teams as fast as possible and ignore some lower priority requirements until later in the project. This turned out in some cases that lower priority requirements where often not addressed for up to 4-8 weeks, and some requirements were not turned into Use Cases until the last 2-3 weeks of the project, however these were always P3, and muted agreement with the business that these could slip if really necessary.

An iteration ran from Wed - Tues of the following week. The reason for this is that we had a fixed amount of work to complete by the end of each iteration, and we really tried to achieve this. However if we overran, or had problems, no one wanted to be around on a Friday evening trying to get an iteration complete, it was much easier to work late on a Tuesday, not that we did it that often.

Every iteration started with an adhoc planning session to review what we thought the current estimates were and deal with any changes in these assumptions. These changes could then be fed into the plan via the wall and also to the business at our regular update meetings.

At the end of an iteration we looked at what had been achieved ( in terms of man days functionality delivered ) and altered the amount of man days possible in future iterations. This obviously meant that the next iteration planning session had to take into account these changes and either add or remove work to/from an iteration. This feedback loop was useful as it validated the approach in terms of meeting 1st of Nov delivery, and also showed us when we needed to turn up the heat for a few days when back log of outstanding work was starting to build

Standups

Every day of the project started with a standup. This was a gathering around the project wall and basically consisted of the same 3 questions to every member of the team.

• What did you do yesterday?
• What are you doing today, and is anything stopping you?
• What are you planning to do tomorrow, and is anything stopping you?

Two types of people were allowed to attend, team members and visitors. Only team members were allowed to talk during the meeting and only to answer the basic questions. Visitors could only respond to direct questions asked of them. Any issues that came up during the stand-up were taken aside and addressed individually. This kept the meeting to around 15 minutes and ensured everyone knew exactly what was going on.

Showcases

Once a week the key business stakeholders came together at the same project wall to discuss the current deliverables. Here we provided demonstrations, walk throughs and white boarded discussions as needed. The stakeholders could bring any new requirements and/or changes to the project at this meeting. At which point a quick estimate of the work was done by the team leads, and then the project wall was used to see if the requirement could be added to the project, when it was possible to add it, and what, if anything had to be removed from the current scope. In essence we were full. We had limited contingency and so the business quickly got used to the fact that for every man day of new work or change they wanted, they had to lose the same number of man days from undelivered requirements currently in scope.

Team Working

While we had distinct functions in the team, project manager, architect, analyst developer, tester etc, we also ensured there was never a silo attitude. It was made clear from the start that everyone was responsible for the deliverable and any problems were the team's problems, not a team member's problem

Everyone producing a specific component worked together, developers, testers and analysts all sat together, worked together and delivered together. This allowed excellent communication between the various aspects of the team and meant that no one responsible for that component was ever more than 1 or 2 desks away from the rest of the team.

Once the scope of an iteration was agreed the analysts would begin producing the specific use cases based on the agreed requirements. The analysts would then walk the developers and testers through the use cases to make sure they were clear what was meant. Developers would then begin working on implementing the use case, while the testers would begin writing scripts and prepping to test the use case. Once the developer had completed his work, all functional and unit tests were complete and the code worked as part of the current master build, they would then sit down with the respective tester and take them through what they had delivered. The tester would then run through the scripts they had written, reporting any issues back to the developer and this cycle would repeat until the tester signed off the use case as having been delivered. At all times the analyst was available to answer questions and help both the developer and the tester to understand what they had created in the use case.

One of the key successes we had was with QA/Development resources designing automated tests together up front to build reusable test components that QA could stitch together and maintain. The analyst would often play a part in this design too.

We effectively worked a staggered 3 week cycle, the Analysts always working one week ahead, the developers working on the current week and the testers working one week behind.

We did however have one or 2 developers that either did not or could not work within an Agile team environment. While it would have made sense, in a perfect world, to try and work more closely with them and encourage them, with the tight timescales and size of the project, it was easier to replace them. I suppose this was one of the benefits of having an entirely contract based staff.

Handling Change

The old adage that change is the only constant held true on this project and came in 3 main areas

Business changes to requirements. Like any project in a fast moving industry, business requirements change, or initial requirements, on reflection, are found to be wrong and need to be changed.

Functional changes to requirements. The same as business requirements, often functional changes are required due to issues found during development. Perhaps a piece of work, once defined, is deemed to require too much effort and therefore a simpler version is required. This change may even have to be fed back into the business requirements.

Changes to 3rd party systems as the project progressed. One of the biggest areas of change we encountered was unexpected changes to downstream third party systems. Documentation for these system was often poor, missing or plain wrong and associated test systems provided by the the 3rd party barely, if rarely, worked. As such the only option was to build against the specification, test it as soon as possible and when it didn't work, find out why, change the requirement and fix the code

An example of this final sort of change occurred 2 weeks after we went live. The 3rd party system just started responding with a different code. Due to the agile processes we had in place, and everyone were well versed in, we had an analysed, developed and tested version out within 12 hours.

Go Live

The project went live at 6:00am on the 1st of November, over the next 6 months we delivered a further 13 releases of the system, some were defect fixes, but most were planned scope releases, up to and including version 5.0. It was one of the most successful LLU provisioning systems to go live and the Telco quickly took over all other competitors in the number of customers it could process each week to the point that the ISP was picking up extra provisioning capacity due to other ISPs not able to fulfil their own quotas.

It is worth pointing out that the system, as delivered, provided a genuine competitive advantage - all the more amazing was the fact that we had started considerably later than the competition. Also due to the initial scope being delivered on time and successfully, the systems were taken forward with further development not originally on the programme roadmap,

Finally, when the system went live, we were able to deal with operational defects quickly and effectively whilst doing ongoing development due to the high quality of the initial delivery.

Conclusion

I personally believe that it would not have been possible without the use of Agile practises. We embraced change at all stages, we never said "no", but we often said "yes, but...." the stakeholders always felt in charge and always knew what was available, what was possible, and what was not possible.

The requirements changed throughout the project, as we delivered more, the business got to see more, and this either got them thinking or, confirmed views of requirements they initially were not entirely clear on. They had a clear and direct feedback mechanism with which to bring change to the project, and also a clear and direct feedback of the impact of any change.

Here are a few statistics for those interested.

• The project took exactly 6 months from inception to delivery.
• The team consisted of 1 project manager, an architect, an analyst, 8 developers, 4 testers and 2 system administrators.
• Approximately 800 requirements where were proposed by the business, were reduced to approximately 200 as the bare minimum, of which 95% where delivered for the first release.
• We only worked one weekend overtime in the whole duration of the first stage of the project and late nights were an exception not a rule for the duration of the first phase.
• We went live on time and in the first 3 months had no more than 2 man days outage in total.

This was something of a unique project; it had a fixed date and a vision, but unclear requirements and the business at the start did not even know if they wanted to be involved in the LLU market. To keep momentum we often had to tell the business what they wanted. We took a high risk decision to work on requirements as we went along and the only planning we did was what would fit on a 20ft wall.

Personally I have never believed there is a single methodology that suits all situations, whether it's Prince 2, RUP, SSADM, or the myriad of new wave process coming from the Agile community. What is required is a project toolbox, from which various techniques and processes can be pulled and used to deal with a variety of problems. In essence you need an agile Agile Process.

This project re-affirmed my belief in Agile processes, in that they should be requirements driven, but not all requirements are needed up front. You do, however, need a good view of the end game from which to start from and you embrace change, and embrace it like you never have done before !!

So Were We Agile ?

We certainly were not XP or Scrum or DSDM, but we borrowed from the all where needed, but we were certainly Agile in our beliefs and in our practices and only constrained by the environment and the customer's willingness to embrace what we wanted to do against their need for us to deliver something. The Agile Manifesto states :-

• Individuals and interactions over processes and tools

  Well our processes were few and adhoc, developed on the fly to deal with real problems. We used just enough tools to get by, but not enough to bog us down, and had a common set that everyone installed and used to a make pairing easier. Key to the success was the use of iterations, standups, show cases and planning sessions to which everyone was involved and contributed.

• Working software over comprehensive documentation

  We delivered on time, we had working software available throughout the project ( OK, most of the time ! ) and the only documentation we produced was use cases and a user guide.

• Customer collaboration over contract negotiation

  They told us what they wanted, we told them what they could get and what they actually needed, but at all times we moved ahead, we never stopped to negotiate. Right from the start, one thing we never did was delivered something they didn't need.

• Responding to change over following a plan

  Our plan was a 20ft white wall with index cards and lots of sticky Bluetack, that's all we needed. Any cards not currently completed could be moved around by any of the key stakeholders. Existing ones could be removed, and new ones added. We abandoned Microsoft Project on day 3 of the project.