I've worked on a couple projects that were remarkably similar. Both were successes when compared to other new products, but neither lived up to original expectations. Each took much longer to ship than planned. Both passed through a period of classic waterfall development and evolved into iterative development.
In the first couple of years, a lone solitary programmer (never me) worked alone on a design for a new infrastructure. This programmer worked in secret, off the books, because he felt the company was neglecting an important strategic area. He had grown tired of short-term hacks that solved similar problems, and he wanted, for once, to solve some underlying general problems. Many outsiders agreed that the infrastructure was important and offered advice, but they did not help.
Eventually, this programmer found a strong management sponsor. The sponsor was usually responding to an urgent market pressure and needed a miracle. Conveniently, the lone programmer had already addressed many of the time-consuming underlying issues. The code solved a more general problem than necessary, but it made a specific solution at least feasible.
At this point the code was valuable because it had already eliminated many bad ideas. Alternative designs had been tested and found wanting. Conflicting priorities had been cleverly reconciled. The design showed conceptual integrity and elegance. The code was far above average in quality. Future development could save months or years. But certain weaknesses were inevitable.
The code was written in solitude and fit one person's ideas of what was easy or hard, of what was important or minor. Everyone has a high tolerance for certain types of complexity. The code was eccentric and over-engineered in certain ways. The programmer implemented some exhaustive solutions just because he knew how. Many elements of the framework would never be necessary and would hinder unanticipated developments. Some code would never be understood by anyone else. A few design decisions would satisfy no one and could have been avoided with earlier feedback.
Anyway, the code was now ready for a massive waterfall of development. A new product needed to be feature complete in six months and ready to ship before the end of the year. These projects received half a dozen new programmers, hand-picked, and some excellent domain experts, who would stand by helplessly as code accumulated.
Next came the Feature List. For the product to be successful, it must have 150 specific features, carefully itemized during meetings with all who would otherwise condemn and derail the project. The absence of a single feature would mean failure.
Features were divided up evenly between developers. They ran off to their individual caves and began working on their lists in no particular order. All features were essential, so it did not matter what got done first. Some developers started with the scariest features, and others with the most fun features. It was a matter of taste, and it did not matter in the end.
The number of lines of code doubled or tripled. The final item on the checklist was checked off on the day of the deadline. Every feature indeed had been implemented to satisfy the specification in some way. Unfortunately, few of these features interacted with each other or even acknowledged each other's existence.
The domain specialists were appalled at how little seemed to work. The listed features actually implied and required many missing features. Were the programmers being obtuse? Didn't they realize that feature A, to be useful, needed to save its result for feature B? And why did feature B not share its options with feature C? A demo was possible, but any realistic work-flow discovered countless obstacles and hazards. The product was completely useless. The code was a haphazard collection of fragments. The fragments were not necessarily ugly, but they were eccentric in new inconsistent ways. No single developer had even seen more than a quarter of the code.
Nevertheless, management sponsors were happy. The code could be demoed, so the project was no longer in danger of immediate destruction. Marketing was ready to make public presentations and to set dates for shipments to customers. The product was buggy, but superficially, it just seemed to need some cleaning up.
Before the product could ship, more than half of the existing code would need to be rewritten. Large expensive chunks of functionality would be dropped permanently. Redesigns would become necessary to support the unexpected work-flows. The character of the product would change drastically. More than another year would pass before the first fragile version was entrusted to a beta client.
At this point, the project switched to a lightweight process like Extreme Programming. The domain experts and internal users finally had a voice. They pointed out that the product did not actually do anything useful yet. They started at the beginning of their work-flows and itemized a few steps that must function first. Yes, there were many more exciting features in the program, but they were useless until these basic problems were addressed.
Developers worked for several weeks on those first few steps, and removed many hazards that made those steps dangerous. The domain experts used a new build of the code every day and pointed out what the programmers did right and what they botched. It took longer than anyone expected to get it right. Then they repeated the process on the next few essential steps. The user experience slowly morphed into something simpler and yet more powerful. Everyone regretted not having started the project this way instead of cranking out so much buggy code.
The developers slowly discovered what the others had been writing during the six months of waterfall development. They found many inconsistent versions of the same functionality. They found stretches of unstructured cut-and-paste code. They found dead code, unfinished code, ignored errors, and comments full of warnings. What were they thinking? Who was supposed to clean up this mess? They began to rebel against the original architecture when it ignored and hindered their specific problems. Slowly the design evolved. Unused generality was removed so that new generality could be added. Conceptual integrity began to reappear.
Everyone was counting on rapid visible progress because of special circumstances. Progress was indeed visible, and heroic goals appeared to be met, but the cost was high. Before customers finally accepted the product, another couple years of debugging and refactoring would pass, with reduced staff.
Going directly to iterative programming and skipping Niagara Falls might have saved, in the long run, a full calendar year at full staffing. Without the isolation, the lone architect could have arrived at a simpler and more powerful architecture.
Of course, sponsors would never have supported the original project under such scenarios. Iterative development looks out-of-control unless it resembles "bug fixing." Programmers and domain experts are not trusted to design a product all by themselves. In practice, iterative development seems to be the only way successful products ever get designed. Most projects do not survive the waterfall.
Bill Harlan, 2002
Return to parent directory.