Agile + Reuse = Efficient Projects

A couple of years ago I wrote a paper with some reflections on how the dynamics of the capital markets could help illuminate the dynamics of agile software development projects. Briefly: the Efficient Market Hypothesis explains how the market ideally will reflect all information available to investors. One point that is often overlooked about this is that an efficient market also reflects all implications for the future based upon the current information available. (In other words, if you think a stock price will rise in two weeks you buy it now, you don’t wait two weeks to do it.)

I then elaborated the idea of an Efficient Project: agile developers try to construct software that way. Just look at the so-called XP Customer Bill of Rights: “You can cancel at any time and be left with a useful working system reflecting investment to date.” The system reflects all the information available to date.

Furthermore, the YAGNI (You Aren’t Going to Need It) principle of XP says that the system should reflect only the information available to date. Don’t implement anything that isn’t implied by the requirements you have now. Don’t try to second-guess the future.

There’s even a parallel in there to the markets: momentum investing is one of the causes of market inefficiency; it causes bubbles, with people investing beyond what the current information (e.g. company revenues) implies. “Momentum implementation” occurs when projects implement features beyond what is called for by the requirements.

This is great advice. It’s a way of keeping systems from adding useless functionality. However: I think there has been a traditional tendency in the agile community to go too far with this. Probably because the agile movement began with projects with truly unstable requirements, a tendency has grown up around this to consider the future to be entirely unpredictable.

But the future is in fact rarely entirely unpredictable. The only project with entirely unpredictable requirements would be one in which you sit down in front of a white sheet of paper and say, “I’m going to do … something – anything.”

The fact is that there is a continuum of predictability of the future, of requirements. But the agilists have made it a bit too easy on themselves in that respect, and tend not to look hard enough for those aspects of the system which are predictable.

And that’s where reuse comes in. Reuse is all about being able to predict the future. In some ways it is the mirror image of agile. It says, “I can predict the future in these important ways, and I can implement a system that reflects the implications of this future.” It's the "You Are Going to Need It" of software engineering.

And this isn’t just dreaming. One of the most successful examples of this is product line development. The people in Nokia lay out an entire vision for future software in their phone families, and implement the corresponding system now. The product line developers also study where the future is unpredictable, say, in terms of features, and even then try to constrain the unpredictability to squeeze out any amount of partial predictability they can. For example, they introduce variation points in feature models that say, “okay, we’re not sure exactly what a system’s features will be, but we can get it down to variations on this theme and implement based on that.”

It’s tough analysis to do, there’s no silver bullet, but it’s rewarding when it works. And agilists tend to exclude this type of analysis due to a mindset that focuses on unpredictability.

This is not to say that agile and reuse are opposed. On the contrary, they are probably the two most important software engineering techniques we have, and can work together to balance the unpredictability and the predictability of systems and their features.

Surprised to hear this? You shouldn’t be: speaking of silver bullets, probably the most important software engineering paper ever written was No Silver Bullet by Fred Brooks (Martin Fowler once told me that no other paper had had so much impact with so few words.) Brooks said that the software development problem was essentially intractable, and that there were only a few truly powerful tools to combat the problem. Two in particular he mentioned: software reuse was one of them. Another was the idea of “growing a system” – essentially the idea of incremental, agile software development.

Why did he single out reuse and agile? Because he said that essentially the only solution to the software problem was to write less software. Reuse is the way to write less software when the future is predictable. Agile is the way to write less software when the future is not predictable. Leave either one out and you end up writing more software than you should. Use them together and you have an efficient project - a project in which the amount of software written is not too much and not too little.

Reuse and agile – if they were good enough for Brooks, they should be good enough for us.