30 May 03 - 09:46The Zeroth Law

The end user, an external customer, had a problem with server crashes. We couldn't reproduce the crashes internally, but the customer was insistent that we should provide a solution quickly.

After examining the server logs, technical support came up with an analysis of the crashes, and suggested how to alter the server's behaviour to work around the cause of the crashes.

The internal customer wrote a user story for the suggested behaviour. He described what needed to happen, and was available to answer questions.

The development team outlined a test plan. It specified what steps would be taken to verify whether the new feature was working properly. The QA team reviewed the test plan and gave it a green light.

The development team implemented the acceptance test script. They ran it against a current version of the software. The test failed. QA reviewed the test implementation and gave it a green light.

The development team implemented the necessary code changes. They ran the acceptance test for the new feature, and they also ran all the acceptance tests for other features, to prevent any regressions. The tests all passed.

The development team packaged the changed binaries as a service pack and handed them over to support, who handed the service pack over to the end user.

By this time, and having observed the whole process, I was asking the following questions: What is the customer expecting of us ? What promises have we made, implicitly and explicitly ? How sure are we that what we just did will in fact solve the customer's problem ?

"We have followed our process to the letter. We've provided a solution that should work - if it doesn't, the new server logs will help us pinpoint the problem much better. And the patch will work anyway. Why shouldn't the customer be happy ?"

Three days later, an angry email from the customer drops into Support's in-box. The customer's problem isn't whether we are following the process. The customer's problem isn't how confident we are that we've properly implemented the behaviour that support thought would work around crashes. The customer's problem isn't the quantity and quality of information we're getting from the server logs. The customer's problem is that the server has been crashing randomly for several iterations. We haven't solved that problem - why should the customer be happy ?

Moreover, the customer's problem has now become that we are not solving the customer's problem. It's no longer a technical problem, it's a psychological one; but we are still trying to solve it on the technical level. And more and more that is becoming the problem.

My version of the Zeroth Law of Quality: if you're not solving the customer's problems, it doesn't matter what the "quality" of your product is; or that of your development process.

- Software Development - two comments / No trackbacks - § ¶

28 May 03 - 16:11Night owls

It's been more than a week since my last entry, and although I have already figured out what I want to blog about next and what set of priorities would suit me better, I'm confronted with a problem of execution.

It seems that I am only able to write between the hours of 11PM and 2AM. One of the choices I made since my last entry was to get much more sleep. I've been going to bed early for a whole week, on occasion getting up one hour earlier to deal with routine tasks such as updating my work diary or reading other people's blogs. With respect to my own writing, though, the result seems to be "blogger's block". Or even writer's block : on my to-do list was an article I'd promised to write for an online newsletter, and I was only able to get that done by burning the midnight oil yesterday.

This reminded me that I know (or know of) a great many programmers who are also night owls. I seem to remember that I had my most productive moments when I coded through the night. At my current client, a small software vendor who grew from a "two guys in a garage" operation to a reasonable first round of financing, former developers turned managers still revel in telling tales of their all-nighters six years ago.

I'm wondering if there is something particular about programmers, that they can so easily stay up late; or is it that people who can easily stay up late make better programmers, or otherwise find a programming career suited to their temperament ?

- The Universe And Everything - No comments / No trackbacks - § ¶

21 May 03 - 07:21Choices

According to Jerry Weinberg, "a consultant never has the right amount of business, but is always either too idle or too busy." I started this blog back in January, when I was still in the I state - desperately idle. Then I landed a contract in April; ever since, I have been in state B - waaay too busy.

In this state of busyness, it is essential to make choices. There are only twenty-four hours in a day; eight to nine of these are now devoted to helping a team improve their software's quality and master the Extreme Programming way of doing that. That's eight to nine hours that I previously spent on other activities; I've had to eliminate most of these activities from my schedule. Some of these choices were hard to make; perhaps not all were wise.

Keeping up with blogging the (almost daily) thought was one of these choices. The blog made the cut, but only barely; it's still taking priority over some things I know I ought to do, or had promised to do. That is nagging at me. I still think that blogging is excellent practice for writing and for putting my thoughts somewhat in order; that it makes a good complement to the other, private diary I've also been keeping about my work experiences.

One solution I want to explore is to write more about topics related to my work, and make the blog part of my work regimen.

- The Universe And Everything - one comment / No trackbacks - § ¶

15 May 03 - 16:32Feedback

I owe Jim Bullock thanks for the idea that a "methodology" (dirty word, but see what "process" landed me into the last time) should be self-correcting - even if you don't follow all the instructions properly, the instructions you do follow should make the errors obvious.

This suggests that feedback is fundamental - people will get some of the instructions wrong. There is a consequence with respect to the often-asked question "in which order should the practices be adopted", irrespective of the methodology in which the practices are embedded. Adopt first (and with most care) whichever ones provide the most feedback.

In Extreme Programming, what provides the most overall feedback is velocity. A few other models have some equivalent; as far as I can tell velocity is more or less the same as the BCWP of earned value, or the burn rate in Scrum's burn charts.

I consider that a team is doing well at the "Planning Game" (an Extreme Programming core practice) if they estimate the cost of all the individual features in a project, then compute, at each iteration end, a number called "velocity". Velocity (V) is the total of individual estimates for all newly implemented features which passed acceptance tests since last iteration. Costs are often estimated in man-days; an iteration is two calendar weeks. (No messing around with iteration start or end dates "so we can deliver what was supposed to be in the iteration".)

This gives an overall project estimate D = S / V (duration, in iterations, is total scope, expressed in days, divided by current velocity). You can vary D to vary S, or vary S to vary D; but V is a measured quantity.

With a decent process to start with, and only this one out of all twelve XP practices, I suspect you could do OK if you were determined to monitor V and improve the process by changing one thing at a time, and keeping whatever increased V. It might take a while but you'd get there. Or if you were content with the current V, just monitor it for signs of trouble, and not do any more "project management" than that. (This, I suspect, is the essence of Scrum.)

Strangely enough, I've observed several self-described "Extreme Programming" teams that were unable or unwilling to do this, and instead gave estimates for the next iteration's worth of stories only. They never updated a whole-project plan based on new velocity figures. This made their "velocity" entirely useless for feedback purposes, and thus, I'm tempted to say, for any practical purpose. I suspect that the collecting of estimates and actuals was done by reflex, as a "managementy" thing to do. (Another problem - if you estimate only an iteration's worth of stories at a time, it's likely that you'll adjust your estimates, not necessarily consciously, so that even if you're slowing down your velocity remains about the same iteration after iteration.)

Just because you call something feedback doesn't make it feedback; it's only so if you take the information collected and feed it back into whatever you're using to control the system on which you collect information. You have to be prepared to take action: velocity went down and we weren't expecting that, so we have to do something about how we work.

(Johanna is also writing about feedback, and Frank about the importance of an overall plan, frequently monitored.)

What are you using for feedback ?

- Software Development - six comments / No trackbacks - § ¶

14 May 03 - 21:44Empirical content of tests

I'm not sure what, in my previous testing entry, Eric from PARC is disagreeing with here. (It's just dawned on me that Out Of The PARC is a collective blog. I don't think I know Eric; I met Ian at XP2002. Hi Ian !)

I do not, actually, "reprove" smoke tests, or even manual testing... in the context of a huge pile of legacy code which is being gradually brought up to XP standards of quality.

Building on the previous entry, I want to suggest that there is a trade-off involved in writing tests - rather than purely a matter of economics as Eric suggests. An automated test written to Test-Driven standards is a good indicator of whether a feature is working; on the other hand, one such test only tests a very narrow slice of functionality. A smoke test, or any kind of manual testing, is a good indicator of whether large chunks of the software are not badly broken. The tradeoff is diagnostic precision (granularity of defects caught) vs. coverage.

Note how the two type of tests have opposite "directions". I'm interested in whether specific features are working when I can count on the software, as a whole, to be mostly functional. I'm not expecting any specific test to fail, in other words - I'm surprised when one does fail. By contrast, a smoke test is valuable to me when I'm not confident that the software, as a whole, is mostly functional. I'm expecting a smoke test to fail often - I'm almost surprised when it passes.

I have a hard time articulating the notion crisply, but it seems to me that we could define the "empirical content" of a test akin to Popper's "empirical content" of a theory. Something along the lines of, the empirical content of the test is the number of possible "states" in which the software behaves in such a way that our expectations about a test passing or failing are not met.

Consider an automated acceptance test. It only tests one feature of the software. Our expectation is that the test will continue passing. Now, there are a vast number of ways that the program could be broken elsewhere, in a different feature; there are also a vastly smaller, but still large number of ways that the feature concerned by this test could be broken. The empirical content of one acceptance test is small, but significant. Also, distinct tests don't overlap much in what they test, so a large number of acceptance tests yield a large empirical content.

Consider a smoke test. We are expecting it to fail rather often, when we break the program badly. But there are a vast number of ways that the program could be (more or less subtly) wrong, and still pass the test. Its empirical content is small. Also, two distinct smoke tests will fail for mostly the same broken states, so that adding further tests of the same kind increases empirical content only slowly.

I don't find my own description of "empirical content" at all convincing. Yet I have a hunch that, somehow, the idea has some value. Feel free to improve on it if possible.

- Testing - No comments / No trackbacks - § ¶

12 May 03 - 23:25Falling in love

At some point during the past week, I became aware that I haven't yet said a single word of Dale Emery's weblog. I've been linking to Dale via my list of favorite "Community" blogs, but I also want to gush. (Note that not all the blogs that I read regularly or that I like appear in that list. I'm not quite sure what the criterion is, actually.)

One entry I appreciated was "Dale Emery, Bureaucrat". I was rereading that today, and one small bit that I had not noticed on first reading jumped out at me as particularly valuable. (I tend to have that experience with good writing - I can revisit it time and again and still extract value each time.)

Dale says at one point: "By that time, I'd fallen in love with [Peter] Block's ideas on Stewardship." The metaphor of falling in love with ideas strikes me as particularly apt; particularly generative. (A generative metaphor is one so compelling that it gives rise to an entire "class" of related or derived metaphors, often useful ones.)

For instance, if by the previous paragraph you were thinking that I'm making too much of one metaphor, you might say I've become infatuated with it. A moment's passion, soon consummated and forgotten. People fall in love with ideas all the time, without necessarily spending the rest of their lives with these ideas. And it's better to have loved and lost than never to have loved at all...

I'm wondering what ideas I've fallen out of love with. And what happened.

- Blogversations - No comments / No trackbacks - § ¶

11 May 03 - 21:43Working, as opposed to not broken

One of the interesting debates sparked by the discipline of Test-Driven Development, a subset of Extreme Programming, centers on whether the style of unit testing it recommends really does have anything to do with "testing", as traditionally understood. For instance, Ian Smith at PARC wonders about the distinction between testing that is performed to find (or repair, or prevent) defects, as opposed to tests written to (implicitly) document the software's design (unit tests) and specifications (acceptance tests).

I'm more inclined to use a different distinction; one which will distinguish between "smoke tests", or indeed much of the post-development testing that goes under names such as "beta testing", and the more focussed kind of testing favored by XP.

"Smoke test" is a term used in the manufacture of electronic equipment : if you turn the thing on, and smoke escapes from it, you know something is wrong. In software, a smoke test is a set of operations exercising major functions of the system; if it crashes or malfunctions badly at some point, further testing is usually not warranted - it goes back into development for debugging.

In my experience, much beta testing, and in fact much of all post-development manual testing, is barely one step above a smoke test. The traditional advice to users when "beta" quality software is released : "Bang on it, and report back if it breaks". Manual testing most often reveals defects accidentally, as it were; it serves to verify, not that a specific feature of the system is working as designed, but to check that the system as a whole isn't badly broken. (I understand that skilled testers, and professional testers, can write test plans that really validate functionality. I've just never worked anywhere that was the case. Some teams must have them, though.)

This, by the way, is why a post-development test "phase" always takes longer than expected. The first thing it finds is the really huge, really obvious defects - when the system is doing something entirely different from what it should (such as crash). Then the smaller defects take longer to spot, because you have to look harder to see if the system is doing what it's supposed to, or something slightly different. And so on - "there's always one more bug".

An XP-style automated test is supposed to test just one thing, just one feature. It should fail only if that feature is broken, or (of course) if a feature it depends on is broken. In the latter case, there should be a "smaller" test which also fails. Potentially, there could be several such levels of failing tests, corresponding to levels of features built atop each other. Ultimately, this should end with a unit test "close" to the defect, and in fact pinpointing its location, so that correcting the defect is almost instant. These tests only pass if the feature they test is really working - not just "not badly broken".

This distinction has interesting consequences, which I'll explore in more detail in the next entry on testing. In particular, it plays well with another important distinction, this time between two contexts of testing (maintenance of code with too few tests, vs. greenfield development in the TDD style).

- Testing - one comment / No trackbacks - § ¶

09 May 03 - 13:47Names for tests

Mike Clark writes about learning tests. These are unit tests written to discover something about an API, or third-party tool, which we don't yet know well enough to use confidently.

Extreme Programming is so keen on automated tests, we practitioners keep coming up with new (and apparently useful) distinctions; if this goes on we'll soon have the eskimos beat in the names-for-snow category.

There are developer tests (the ones coders write to inform our design) and customer tests (the ones written by the customer, or QA people, or coders on behalf of the customer, and which validate features). There are functional tests, which are "bigger" than unit tests, and bigger still are integration tests, which test entire systems end-to-end. There are learning tests and there are "name that design" tests.

As yet unnamed are the tests written to expose a defect, and the ones you write to prove that your design doesn't yet do what it's supposed to.

I've also been thinking about "what tests say", how we might characterize the content of a test. I'll be coming back to that.

What names have you been using for test categories not listed above ?

- Software Development - one comment / No trackbacks - § ¶

07 May 03 - 17:54Not enough

Hal Macomber writes:

I do too much writing about uncertainty, lean practices and methods, and the linguistic action perspective and not enough writing about what we find is the most difficult aspect of lean. It is Womack's and Jones' fifth lean principle: pursuing perfection.

I wish I knew what I am not writing about enough. I'm not that clear yet - I'm lucky when I can find something I'm sure I know something about, and can therefore write about. (If you've got ideas about that, please comment.)

Then again, this (almost) daily message in a bottle, this letter to mostly unknown correspondents, this regular "writing kata", in other words this very blog - is my modest way of pursuing perfection in an area I care about: putting my ill-formed thoughts into a shape where they can be useful to others. (Not just my thoughts, of course, but also the ones I happened to have gleaned from others.)

The phrase "pursuing perfection", however, does land me into trouble occasionally. I find that people tend to hear it as "trying to reach perfection right now", and since that is impossible they give up on reaching perfection at all. I've found it hard to make it clear that I want to reach perfection gradually, by a process of continuous improvement - but in order to even have a chance to do that, I need to have a clear idea of what perfection would require of me now.

In the same vein, Kent Beck says:

How good the design is doesn't matter near as much as whether the design is getting better or worse. If it is getting better, day by day, I can live with it forever. If it is getting worse, I will die.

- Blogversations - four comments / No trackbacks - § ¶

06 May 03 - 15:17The Salad Standard

I have a keen interest in software development "processes".

I really shouldn't use the term "process" before stating a few key distinctions to prevent common misunderstandings. It's one of these words, like "bug" or "requirements". Words that we use all the time, but often to convey a variety of subtly different meanings, which can result in harmful confusion. What you're reading is a blog and not a book, though, so I'll just make a mental note to revisit the topic later.

There are (at least) two aspects to a process. One of them consists of answering the activity question, "What am I supposed to do next". The other is the outcome question: "What results am I supposed to get from doing that particular thing next".

In Extreme Programming, for instance, "write unit tests before code" belongs in the former category; in the latter we find, among others: "intentional style of coding, emergent design, fewer defects and regressions, better maintainability".

The Salad Standard gets its name because these two things are like oil and vinegar - they don't really mix, you only get the impression they do when you've done a lot of stirring. So in the head of a person who just "got" Extreme Programming you have a nice emulsion. In due time, it can happen that the emulsion goes "cold", and activity and outcome separate again.

As a result, a team can end up with some people (often managers) who believe that they will get certain results, and other people (often developers) who perform activities that do not always contribute to these expected results. The Salad Standard says this is a recipe, not for Caesar's Salad, but for disaster.

Separation can occur when we attempt to transfer the process to other people's heads. I claim that it often occurs when the transfer is resisted even slightly by the people involved, and we do not pay attention to their objections.

There are parts of Extreme Programming that are supposed to keep you stirring. That Feedback value thingie, for one. Still, I see too many teams who are using "Extreme Programming" without doing much stirring. (I have to use scare quotes again, because my friends in the XP community will argue that if you are not paying attention to feedback, you're not really doing XP. Nolo contendere, as they said in Caesar's time.)

I also find that XP, in and of itself, isn't enough to keep you supplied with vinegar - the "What am I after" stuff. You have to know what you want, and keep your eyes on that, and both are very hard to do - whatever your "process" is.

Update: the Salad Standard suggests, for instance, that the related Van Vleck's Comb isn't the whole story.

- Software Development - two comments / No trackbacks - § ¶

05 May 03 - 19:24Information hiding

I have three sons aged 1, 4 and 7. Among the many practical problems this entails, there is the category of non-toy items (digital camera, computer, wristwatch, glasses) being taken for toys (or in the case of the older, more worldwise child, "things I have a right to play with"). I used to experience some anxiety about this, worrying about some of these things getting broken.

There are some people who prefer to keep such things hidden from their kids, up out of reach, or under lock and key. My experience is that that is almost always a hassle, and not worth the trouble. As a kid (this is my week for childhood recollections) I could outsmart my mother most of the time.

Mostly I trust the kids to learn about what's precious and what's not, what they can play with as if it was theirs and what they need to take extra special care with, etc. There was limited breakage in the initial stages, but this mostly works.

**

Long ago, I felt that I needed to be careful about what I told people. Not all information was right for every pair of ears, and some had to be altered. This was particularly true of two categories of information - how I felt about people, and things I'd done or said which affected them. Put bluntly, I felt OK about lying in order to avoid hurting people's feelings, or to avoid getting them mad at me, or to get them to approve of me.

On balance, I've found that that too isn't worth the trouble. It's hard enough knowing with any precision how I feel about the world and how I interpret my experiences, when I have that knowledge I share it freely, without distortions. I trust people to make good use of the information, and withhold rather little.

**

When it comes to software design, my sympathies lie with languages like Smalltalk, where for instance it is possible to label some methods as "private", but this "information hiding" is not something enforced - as it is in the Java language, which prohibits by run-time verifications the calling of private methods. Rather, it is presented as extra information - "this happens to be a private method, and I trust that calling code will make responsible use of this notation". You're not forcibly prevented from calling a private method, which saves Smalltalk the expense of a convoluted mechanism such as is found in Java for the cases where you really do need to call a private method.

- The Universe And Everything - two comments / No trackbacks - § ¶

04 May 03 - 23:11Change and constancy

When I was a child, I was always anxious to become an adult. Being a kid was a drag; not being allowed to decide for myself, take risks on my own behalf, be really responsible for my own life. I waited for the transformation into an adult to complete; though I hadn't the foggiest how people became adults, it was obvious that at some point they did. I only had to look at my parents, who decided everything and were totally in charge.

I didn't just wait. I monitored the progress of the transformation. I kept looking back over the previous few years of my life, noting with grim satisfaction what a complete idiot of a kid I'd been two or three years ago. Today, I decided on each occasion, I was for all intents and purposes a different person from that kid who knew so much less about life, was allowed none of the cool things I could now get away with (such as smoking, once I'd discovered that). There were some memories I shared with that person, and that was that.

I caught myself doing that again, quite recently. In the past three years I've come to see my life as a software engineer with such new clarity; not just that, but gaining new insight into my goals as a person, into what my family means to me; unexpectedly gaining some "people skills" I would never have thought I'd acquire; making some radical changes in myself - I quit smoking, for instance; and so on - a new "me", to be sure, a radically different person.

Different enough that I'm now able to appreciate the irony - my perspective on life, and on my own growth, has remained remarkably stable over the years. A lot of me has, actually. Probably the most substantial change is all the weight I've put on. As for becoming an adult: I've found I'm not very good at taking decisions, the risks I take often backfire, and being responsible for my own life is a drag; I decide too few things and I'm rarely in charge. And I've become aware that my parents have, and always had, mostly the same problems.

The good news is that the changes I perceived were real, even if I tended to exaggerate their importance. They happened against a background of constancy without which I actually wouldn't have been aware of any change, and which, precisely because it was a background, I didn't notice. But they did happen, and my memory of these regular "upheavals" is evidence of both change and constancy, two sides of one coin - growth.

What is your experience with change and constancy ?

- Private Life Of The Brain - No comments / No trackbacks - § ¶

02 May 03 - 14:54Certifications, degrees, teamwork (II)

Esther contributed some thoughtful remarks in response to my entry concerning Cem Kaner's Software Testing 2 exam.

Cem also sent a reply, which he gave me permission to reproduce here:

From: Cem Kaner

Laurent, here is some additional information about my course, which you
criticized in your blog.

The majority of the students' work in this course is done
collaboratively:

• The course included 5 assignments, which could be done by any size
  group of students and submitted jointly
  
• The structure and many of the details of the first test were
  announced in advanced, and students were encouraged to think together and
  to jointly plan how to set up their development systems. The exam itself
  was a solo effort, but Assignment 3 was a tougher version of the exam
  that students (who had just finished the exam) were encouraged to
  collaborate on
  
• The course also included bonus assignments, typically research
  projects in which one or a group of students figured out how to do
  something tough and then presented their findings to the class. These
  could be done in collaboration.
  
• The final exam involved a thorny research issue--how to drive
  OpenOffice's Calc through its API. Five students picked up research
  projects (and collaborated in a variety of combinations) to figure this
  out in preparation for the exam (and present their research to the
  class).
  

Throughout the year, in this class as in all the classes I teach,
students worked together in teams of several sizes.

But at some point in every class, I require students to show me what
they, individually, can do. Without this, several students will coast
through on the performance of their peers and miss the educational
benefit of the course.

Teamwork is important. But so is individual competence. I see nothing
wrong in encouraging the development of both, and designing evaluations
that involve each.

I replied as follows to Cem:

Cem, thank you for the additional information on the testing course, as well as for the feedback on the blog.

It's possible that my wording in that blog entry did not reflect my intent. I am in no position to criticize your testing course; all I know about it is what you just told me. I did wish to express uneasiness with the text of the final exam for the course, as posted on your own blog.

I suppose that part of the uneasiness stemmed from the strong stress on the words "ALONE" and "MAY NOT" in the exam text. I had interpreted these visual cues without much context. You have provided further context, which leaves me less puzzled and less uneasy.

I should add that I remain somewhat puzzled and uneasy.

Are we doing the same sort of thing when we test a course (by testing, on an exam, the people who have completed it), as when we test software ? Perhaps the analogy is faulty, but it seems to me that an exam is sort of like an "acceptance test" for a course, whereas the content of the course is akin to an implementation.

We've established that teamwork and collaboration are important, as is technical competence. It seems to me that at some time, we'll need tests which "test for" teamwork and collaboration skills just as much as we "test for" technical skills.

In her entry, Esther lists a number of skills which contribute to effective collaboration. (Are they skills, or personality characteristics ?) Is a testing course supposed to teach these, if people didn't possess them previously ? How is a person's mastery of these "skills" to be assessed ?

- Software Development - one comment / No trackbacks - § ¶