30 August 03 - 15:17The Waterfall paradox

There's a lot of confusion regarding that grandaddy of software development process models, the so-called Waterfall. So many things are meant by the name that it has long ago ceased to be useful as a name, but people still argue as if the name "Waterfall" pointed to something definite. I think that's the source of much apparent disagreement among people who discuss software development processes.

One common point of agreement is that "Waterfall" applies best to projects where little discovery needs to be done and the problem domain is well known, in contrast to Agile processes which supposedly work best in innovative or high-uncertainty projects where a lot of learning is to be expected as the project unfolds.

If I know the problem domain, then indeed I can usually divide solving a problem in step A, then step B, then step C. Cooking feels like that; although I've taken it up recently enough that there's a lot of discovery going on, there's always a certain order things are supposed to be done. If I follow that, and get the timing right, and the proportions, I end up with an edible meal. Well, sometimes.

This is one of the main characteristics that people seem to have in mind when they talk about Waterfall - a (mostly) fixed sequence of steps, that have to be done in the right order.

A different, distinct idea that people seem to have in mind when discussing Waterfall is the precise list of steps that are to be performed in any software development project. The original Waterfall article had six - they went "Requirements, Analysis, Design, Coding, Testing, Deployment". Sometimes one or the other is omitted in a description of the process, and there are many variants, such as ones in which "Testing" is distributed throughout the project (smart move). The main point is that in any project the steps to perform are those, in that order.

There's no similarly generalized model for cooking that holds for all recipes I've done so far (and I dont know that many). Sometimes it's "gather ingredients, mix, bake, serve". Other times you do these in a different order, omit one, do one twice or more. There are many other types of steps, too. (There's even some rigorous research into what works and what doesn't in cooking, a lot of it counter-intuitive. Cooking, like programming, is far from trivial.)

What bugs me about that "standard" Waterfall, the one that goes requirements - design - implementation, is where did these phases come from ? Why are they the "standard" way of doing things ?

It strikes me - that's the paradox - that the Waterfall phases are a wonderful way to learn about a problem. As a generalized problem solving framework they work beautifully.

First we define what the problem is, expose our assumptions about it, formulate concrete and achievable goals, etc. Right there in the "requirements" phase we learn a lot about the problem - if we do it right, sometimes we no longer have a problem by the time that phase is over. Then we move into "design", which is where we have the opportunity to think about what The System is that we are proposing to modify, explore possible unintended consequences of our proposed actions to solve the problem, and so on. Again an opportunity to re(de)fine our understanding of the situation. And then, in "implementation" we get to test our models against reality; that's where we learn whether the system responds to our interventions in the expected ways.

If we truly know the problem domain, chances are that there is a specific sequence of steps that apply. If you're designing a compiler, you can famously farm out the work to as many subteams as your compiler is expected to have passes. Only if we're tackling an unknown problem does it make sense to adopt the generic Waterfall phases, and then as a strategy for learning.

The generic techniques defined by the "classic" Waterfall phases work best for discovery, but the Waterfall process is said to work best in projects where there's little discovery to be done. Ironic, or what ?

- Software Development - No comments / No trackbacks - § ¶

08 August 03 - 13:37Metaphor in Extreme Programming

It appears that I once wrote a definition of XP's Metaphor practice that someone liked enough to reproduce in a document of the Extreme Programming "Process Model". Interestingly, I wrote it up as an activity - not as a state or as an output.

Identify, or build, or evolve, a way of describing and talking about the system which a) serves the criterion of utility - talking about the system in this way rather than in another way makes it easier to write the next story, write the next test, or name the next class - and b) can be shared by all the people involved in the conversation that building the system constitutes.

Thanks, whoever you are at Fraunhofer IESE who saved this from oblivion. I still like this quite a bit.

- Extreme Programming - No comments / No trackbacks - § ¶

06 August 03 - 00:39Project Management, the movie

...and it's called "Lost in La Mancha"; the project featured in the case study is a cinematographic one, Terry Gilliam's disastrous attempt at the story of Don Quixote. A must-see if you're involved with projects of any type, in any capacity. The parallels to what many of us have experienced in software projects are striking.

Gilliam's attempt was, in a sense, version 2.0. The adaptation had been attempted once before, by none other than Orson Welles. Beset by financial and other difficulties, Welles never completed his Quixote, though he kept at it undaunted by even the death of his two lead actors.

Of course, it is routine for software projects to be attempted even when the example of a disastrous predecessor should be cause for concern. Perhaps filmmakers are as a rule more superstitious, or more cautious, than software professionals. Gilliam, at any rate, seemed to be keenly aware of the fate Welles' version had met.

At first glance it might seem as if Gilliam was indeed a victim of "The curse of Don Quixote". Every disaster you can think of apparently happened on that project. Lead actors unavailable for health reasons. Contractual difficulties. Sets wrecked by storms. Animals refusing to cooperate at inopportune moments. The lot.

Gilliam, then, was a victim of Murphy's Law. Or was he ? There's a story Jerry Weinberg tells, of a survey he did, long ago, of projects that had failed at IBM. He had noticed an interesting pattern: all the failed projects had experienced some form or other of bad luck; from engineers going on sick leave at unscheduled and inconvenient times, to fire ravaging offices (and the backup copies of the software). He was about to conclude that the leading cause of project failure was plain and simple Bad Luck. But then he had the idea of going back and doing another survey, this time of successful projects. He found that the successful projects had also had back luck, but by careful planning or astute management had been steered to completion. Backups had been made offsite in case of fire, people spread knowledge around so that sick leave wasn't so disruptive, etc.

In the case of Gilliam, there's an almost exact parallel. We learn here, for instance, that a previous production, The Adventures of Baron Munchausen, experienced many similar difficulties - ill actors, crew problems, animal cast problems, funding problems, conflicts with the studios... Yet Munchausen eventually hit theaters. It also did badly in the box office, but that's another story; what Gilliam remembers - and the painful memories come back to haunt him on the making of Quixote - is his "rotten luck" in production.

The Mancha Case Study also has more to it than just a case of rotten luck. One of the movie-about-the-movie's authors, Louis Pepe, says it best when he says:

One of the interesting discoveries in the editing process was to see how many events that seemed insignificant at the time, turned out to be signs of the production's vulnerabilities.

That is also one of the interesting discoveries from watching the movie. The sense of impending doom is there from the start. Indeed, if you have an occasion to see this movie with coworkers, and I hope you will, a fun game for the after-movie dinner party might to debate the question, "Just when should it have become obvious that the project was going to tank ?"

As early as pre-production, the movie was prey to "budget pressure", in software a slightly less well known cousin to the more familiar "schedule pressure". Gilliam knew he needed about twice the money he actually had on hand to make the movie he wanted to make, but he went ahead all the same. How many software projects can be predicted as failures from launch day, when a schedule is announced that is half the "realistic" figure that anyone with any amount of experience would estimate ?

"Lost in La Mancha" is full of Dilbert moments. Gilliam mentions at one point that the detailed production schedule that has been drawn up has absolutely no room in it to cover for unexpected events. But he stresses - repeatedly - that he works best under "impossible" conditions. (Risk management ? Who needs risk management ? Wimps need risk management.)

At another point, one of Gilliam's aides tells a worried crew: "We haven't been able to get firm input on this from his doctors, but we are going to move forward on the assumption that lead actor Jean Rochefort will be back among us, and in good health, by October 16th." One of the crew raises a hand and asks the obvious question - "But what if Rochefort isn't back on the 16th ?" To which the answer, inevitably, runs: "Look, this movie is getting done. No matter what, we are moving forward with this movie and getting it done. I can't answer you. I don't know what we'll do if he isn't back by the 16th. We'll come up with something."

We cannot help but be sympathetic to Gilliam. If you think about it, success with a project, any project, great or small, is something of a miracle. A project consists of visualizing a possible future and making it happen, which inevitably displacing some other reality that would have come to pass if we'd stood by and done nothing. The authors of the documentary make much, perhaps too much, of the parallel between Don Quixote and Gilliam himself.

Isn't any director, or any project manager, someone who tilts at windmills ? The problem, as Gilliam puts it, is that sometimes the "windmills of reality" fight back. And they do nothing as gentle as just "tilt".

- Software Development - one comment / No trackbacks - § ¶

05 August 03 - 09:48Project head games

From an article (free registration required) on adapting Extreme Programming to game development, by Thomas Demachy of ExtremeGameDev.org and french games vendor Titus:

Recently, during a meeting with our programming team, we discussed a level design tool we needed to create. The level designers hadn't yet completely laid out their requirements, except to say that they needed the tool fast. The word "fast" triggered some strong reactions from the programmers. The programmers said that they would need at least three months to create this tool, that they needed detailed specifications, and that most of the time would be used to "shield" the code against manipulation errors. Finally, the programmers said that since they couldn't really say when this tool would be ready, the whole production schedule would be delayed.

If you're a producer or a project manager, I bet you've heard this kind of argument before. When discussing schedules, somebody always says something like: "Needless to say, we'll be late." And indeed, we are frequently late.

I can almost visualize the meeting Thomas describes above, I've been in similar situations so often.

I suspect that many schedule "problems" have the nature of self-fulfilling prophecies. We believe that we will be late, and this creates a lot of psychological pressure. Under pressure, we invent half-baked "solutions" to the schedule problem. Because we are under pressure, these solutions backfire. When our schedule-saving "solutions" backfire, we don't perform as well as we might otherwise have... and we end up late.

The brief description Thomas gives of a planning meeting is quite rich in observations, if you're familiar with this kind of situation. For one: to me, the description suggests one prevalent mood - resentment, on both parts. "We had a good schedule but you messed it up." Arguments for future blame are being readied - incomplete requirements and manipulation errors on the part of the designers, unresponsiveness on the part of the programmers.

Notice, too, how the project's reality has not changed one iota between the start of the meeting and the end. Rather, both teams have discovered something about their shared reality. They have found that the designers depend upon the programmers for one part of their work - work that is crucial to the project's overall success. Or perhaps it is more accurate to say that an assumption has been exposed - the assumption that the designers depend on the programmers for this part of their job. It is possible that the work could be done externally, for instance. In any instance, this is new information - it should be a cause for rejoicing, and not interpreted as "delaying (i.e. damaging) the schedule."

Part of the problem is the lack of clarity of the designer's request, and the reciprocal lack of clarity on the part of the programmers as to their ability to meet the designer's conditions of satisfaction. But here too, the reaction is counterproductive: it is to "delay the whole production schedule" - that is, adding a project-wide buffer. But the buffer does not do away with the unclarity; if anything, it is going to conceal it. Knowing that the schedule has been extended and the problem "dealt with", both teams are less likely to spend more time on clarifying the requirements. The programmers are saying as much when they state that they intend to write code to "shield" against manipulation errors, i.e. misunderstandings on the part of the designers.

A schedule is not reality. A schedule is a tool for learning about reality.

- Software Development - one comment / No trackbacks - § ¶

04 August 03 - 16:19Eliminate half the autos

We've performed [...] a number of experiments with developing information systems in stages. We start with a skeletal system that does some minimal job. After the client has had a chance to get the feel of the new system, we sit down and specify the next round of incremental improvements. [...]

I'll never forget the reaction of the first client who learned that he could terminate his project when only about eighty percent of the specification had been implemented.

- "Do you mean it's finished ?" he asked, in disbelief.
- "Yes, finished."
- "You mean I can just keep on using the system as it is, and don't have to suffer through any more updates ?"
- "That's right - unless you decide you want something later on."
- "I suppose I'll have to pay the full estimate anyway ?"
- "No, just the amount that's been spent so far - about sixty percent of the original figure."
- "Do you mean that the last twenty percent was going to cost me forty percent of the money ?"
- "Only if we made the budget. More likely it would have cost you half or more, if we'd overrun."
- "But I'm done three months ahead of schedule. I would have been willing to pay extra for that."
- "That's all right. It's a bonus because of our superior methods."

I know it's hard to believe if you haven't seen it happen, but I've seen it more than a dozen times and heard about others. It does sound a bit like the proposal to eliminate all the auto collisions by eliminating half the autos - since two cars are involved in each collision.

In our case, we could start with the observation that ninety percent of the aggravation in programming projects comes at the end. Therefore, the way to get rid of most of the aggravation is to chop off the end - the part where you're getting every last bit and twiddle in place. That would be ridiculous if the most important matters were left to the end, so make your job easier by doing the important things first and never doing the unimportant ones. Or hardly ever.

Jerry Weinberg, Rethinking Systems Analysis and Design... 1988 !

- Software Development - one comment / No trackbacks - § ¶

02 August 03 - 19:09Running out of integers

I recently had the interesting experience of watching a developer run out of integers.

- "Is task X done and integrated ?"
- "Just about; I'm done with the testing and coding, but I'm waiting for project management to give me a task number. The project Intranet hasn't been updated in a few days and the task I'm working on isn't in there. Once I have that I can tag my own changes in the story's branch, then integrate in the mainline."

If you're like me, you didn't bother to follow the techie bits at the end. It's bad enough, in itself, that a developer should waste time making the version control happy when she could be spending the same time designing, testing or coding. But that's another topic.

If you're like me, you experienced a twinge of anxiety as soon as the word "waiting" came up. Spending time stuck in the middle of a task, waiting for something or someone to unblock you, is one of the most effective paths to losing your concentration, making mistakes on that task, forgetting some detail critical to completion, or some such disaster.

- "If it's just a number you need, make one up. Last I looked, the Intranet was in the 300's. Pick something in the 500's and you'll be safe from clashing with some other task's number."
- "What, me choose a number ? Without asking the project manager first ?"

Of course "waiting on a number" is, in this case, a symptom of a deeper issue; task numbers are purely a matter of policy.

There's an Intranet application that was developed under Lotus Notes by the project manager to keep track of technical tasks. (Not quite the appropriate terms, since this is a self-described Extreme Programming team, and they would use "story" for "task" and "customer" for "project manager". I've used the "traditional" terms to avoid confusion, since this isn't really an XP specific issue.) Technically, "task number" is just an arbitrary field, so there's not even a problem with having two tasks having the same number; the database will cope with that just fine.

Not only are the sequential task numbers only a policy, they're an arbitrary one without any observable benefit. In planning discussions, people will often have an exchange along the following lines:

- "Now, I've been wondering about the status of #243..."
- "Um, what's #243 again, remind me ?"
- "It's the one about running on Sybase."
- "Oh sure, I got done with that two days ago..."

As a token for identifying a task, "Sybase support" would do just as well. In summary, task numbers are, in this particular team, both a source of unnecessary confusion and a cause of unnecessary delays.

Still the team continues to use them, because they give the appearance of following a policy, a "process". And, I strongly suspect, because they are a common feature of software tools which purport to "assist" software development planning - they're "just the way such things are done".

This hunch was confirmed when the same team's PM unveiled the newer Intranet application he'd had developed by one of the company's interns, with unified management of planned tasks, defect reports and feature suggestions. Among the new features of the new application, users were now required to identify themselves by login and password... and were grouped by roles, with only specific roles having, for instance, the right to create a task.

I experienced a moment of pure despair came when it was explained that "due to way access restrictions are implemented, at the moment you cannot modify a feature request or defect report that you have just created; but we'll fix that in the next release".

Compared to that, perhaps running out of integers wasn't such a bad problem.

- Software Development - three comments / No trackbacks - § ¶