The trigger of this post came from an interesting discussion I had with Bas Vodde during a Scrum master training session that he held in Helsinki. I had a very weak voice after a bad cold, so with some effort, I told him the story of the most agile project that I had ever heard of (*). It was about building apartment buildings in Cairo, Egypt. The development companies make the plans, present them to potential customers and when the first floor is bought they start building the apartments and leave rooftop ready for building the 2nd floor. When the 2nd floor is sold they start building it and so on and so forth. I understood that the apartment needs in Cairo are really big, so people are willing to move into a place under construction. The weather also allows for the buildings to stay unfinished for a long time.
Now, if you build like this you obviously need a strong foundation to sustain 5, 10 or 20 floors and you must have that in place from the beginning. You cannot change your mind after the first 3 floors and say "I'm gonna build a 20 story building even though my foundation was only for 5".
My question to Bas was: how can we actually build a strong enough foundation to last for several releases of a software product. "You don't have to", was Bas's reply. The software is not like a concrete and steel building. The software is soft, you can change it. That answer has gotten me wondering: how soft is the software?
The Linux kernel and Eclipse are notable examples of products with an evolution path that did not require breaking everything down. However, I also have good reasons to be skeptical. I have had the privilege to contribute to successful software products and I have friends who have worked with such products. In many cases, entire releases were dominated by refactoring, rewriting, re-architecting. Microsoft did that with several releases of their operating system and of Internet Explorer, although they apparently managed to avoid it in the Office suite. Netscape and then Mozilla foundation did it with the Communicator/Firefox browsers. Nokia, the company where I worked until recently, is giving up on not one, but two software platforms (Symbian and Meego) just because all the attempts to re-engineer them were not bringing fast enough results. Being smart does not make you immune, but it might help in selling such a release by adding a few fruity treats on the side.
Despite all the evidence, the myth of the easiness of software changes has become pervasive in the software development community. This is my attempt at explaining how this legend developed.
- The most obvious and irrational reason for the belief may be the term itself: software. Coined by John Tukey in the '50s, the term is defined in opposition to the hardware. Language and beliefs are linked together. The saturated repetition of statements as a way to create beliefs has been used already in the political arena or as a brain-washing technique. I do not suspect any malicious intent behind the use of the word "software", but its unchallenged, widespread usage could create the belief that the software is, you know... soft.
- The success of the software upgrade practice. Companies big and small, producing software for personal computers, phones, services, all can upgrade their software and deliver corrections. Sometimes it can be done remotely, without a human ever touching the hardware device. Bringing corrections and updates to a software system already in use is easier and cheaper and many times less messy than upgrading a house or recalling cars for correcting defects.
Does it actually prove that changing software is easy? It only proves that deploying changes in software is easier than in other industries, but it says nothing about how easy it is to do the changes.
- The software is a product of the mind. The software lives in a computer memory, but it gets created in the mind of the software developers. It is a mind model, expressed in certain programming language, of a particular job that needs to be solved. Without any physical limitations you can bend and stretch the model any way you like, can't you?
A small software project has at least few thousand lines of code. The typical products that I helped building contained interdependent components that amounted to millions of lines of code. At that scale your mind can play funny tricks. The job of the mind is to make the world around comprehensible, which leads to a lot of simplifications. You will miss many details, which will result in mistakes. Most developers understand it at a rational level, but when someone, say a tester, tells them that their mind has failed them they will instinctively become defensive. Maybe because admitting that your mind can produce the wrong result is close to admitting insanity. It is personal.
It might also be one of the reasons why history is full of stories of great people like Mahatma Gandhi, Martin Luther King or Nelson Mandela who led a long fight against corrupted mind models. Those are examples on a whole different level, but they show that people can cling to even the most unreasonable ideas for their whole life. It was more comfortable to patch for centuries a model of the Universe centered around a flat Earth, rather than use the evidence to create a new model. Mind models do not bend easily, even when faced with overwhelming evidence.
Although not everything I have written was soft, I have also built enough soft software myself to bare witness that such a feat is possible. It is really hard to build soft software (**). It requires a lot of discipline and it initially takes more time than writing the spaghetti version. And there will always be temptations not to do it. Ken Schwaber describes the way how the software ends up stone wall hard in his talk at Agile2006. I have witnessed myself and heard from other people about projects that had been ruined in the exact same manner described by Schwaber.
Is there a recipe for developing soft software? In the world of the magical "5 things" that you need for success, that would be: hire good engineers, provide them with good business guidence, provide them with a nice work environment and then let them do their job. Easy, isn't it?
-------------------------
(*) Mind you, this is just a story, do not take it as a fact.
(**) The reasons why you would want to build soft software are explained nicely by Joel Spolsky in a post called Things You Should Never Do.