The unscripted collective art of software engineering

Are we human or are we dancer?
My sign is vital, my hands are cold
And I'm on my knees looking for the answer
Are we human or are we dancer?
"Human" by "The Killers"

The history of software development places the roots of modern software in the aftermath of WWII, making it a young trade at the scale of human history. Like the aviation or car industries before it, software started as an activity carried out by a few pioneers, individually or in small groups. A typical child of the 20th century, software turned into a mass production activity as soon as its value became clear and the advances in computer hardware and programming languages allowed it.

As an industry, the software development was faced with and interesting question: "How should the teams of people working on software be organized?". I started my career in the software industry at the end of '90s. The prevailing model at the time was the waterfall model, sometimes drawn in the fancy shape of the V-model. The linear flow of activities in the model reminds me of the successful industrial manufacturing, with the assembly line as the centerpiece. The software is passed forward, from one end of the line to the other, in a well organized fashion. The choice of production model should not come as a surprise. The assembly line had been the most successful way of organizing mass production, so it was probably viewed as the best practice of the day.

The limitations of the waterfall model have lead to the emergence of a new style of software development: "one that breaks down hierarchy, that features dynamic social structures and communication paths, and that values immediacy. This [...] style often bears the label “agile,” but that is just one of many characterizations of a broad new way of developing software [...]." The new style is iterative, with organizations today using 1-6 weeks iteration cycles and continuous integration of software. The natural consequence of such a short iteration cycle is that the linearity and sequencing are completely abandoned: testing can start before there is any code (e.g. test driven development), design can happen after the code is already functional (refactoring), requirements may be discovered at any point in time. If I had to pick a single most important contribution of the agile SW development community, the departure of SW creation from the assembly line model would be my choice.

Why did the assembly line organization not work for SW creation? The best hypothesis I have seen is that the SW organizations are closer to self-organizing systems than to assembly lines. In "Agile Software Development with Scrum" Ken Schwaber argues that the software activities fall into the realm of complex systems (as opposed to simple or chaotic systems). Complexity science, the discipline that studies self-organizing systems, would then explain very well why a simplistic linear model fails when many teams are involved in the creation of the SW (1). Although the complexity theory would explain why the waterfall SW organizations cannot succeed, it does not give much guidance on the practical matters of how to deal with SW teams in their daily work.

Understanding what the software development is could be one way to figure out how the software developers should approach their work. As a graduate of an engineering school I am sensitive to the arguments of David L. Parnas and Steve McConell, who favor an engineering approach for the development of commercial software. Parnas compares the separation of the software engineering and computer science with the similar split of electrical engineering and physics. "An Engineer", says Parnas "is a professional who is held responsible for producing products that are fit for use". That type of responsibility requires different training and work methods than what scientists need.

While the engineering approach to software appeals to me, I would point out that not all engineering spun off of science. The civil engineering activities developed out of necessity. The core of the mathematics used in construction design had been developed by the ancient Egyptians and Greeks, but the modern science needed by civil engineers emerged in the 17th century with Isaac Newton's classical mechanics. Later, advances in physics and chemistry have provided engineers with new or improved construction materials, better heat insulation and illumination. However, the lack of modern science has not prevented people from using empirical methods to build impressive constructions during the ancient times or the Middle Age. After the science became advanced enough, civil engineering also morphed into a modern engineering activity, i.e. "the application of scientific and mathematical principals toward practical ends". Re-basing the civil engineering on science has lead to an interesting phenomenon: a distinction between architecture and civil engineering started to develop. I think this is no coincidence. There is a considerable overlap between architecture and construction engineering, but the architects want to make a clear statement that apart from science and mathematics there is also art needed when designing spaces inhabited by people.

I believe that the software today is still in a mixed, undifferentiated state. Although voices like Parnas' and McConell's get louder, there is still a great deal of confusion between software engineering and computer science. In addition, there is no formal recognition yet with respect to the empirical and artistic aspects of software development. I choose to describe the software development as an unscripted collective engineering art.

Art. Let me first clarify that art in this context may refer to either art or craft. The difference between the two does not affect the rest of the arguments, so I will use art and craft interchangeably. 

There have been already pledges to include the software among the artful activities. In The Pragmatic Programmer. From Journeyman to Master, Andrew Hunt and David Thomas make extensive arguments in the same direction: "Programming is a craft. [...] As a programmer you are part listener, part adviser, part interpreter and part dictator. You try to capture elusive requirements and find a way of expressing them so that a mere machine can do them justice."  Fred Brooks attributes much of the joy of the software development to the artistic freedom programmers have: "The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures."

Jonathan Wallace considers that the software development is completely dominated by art and it should free itself from the engineering approach. That goes one step too far for my taste. The software products solve practical problems, so I prefer to keep the engineering sense of "fit for purpose" in the software. Much like the architecture, I see the software as balance between art and engineering. It is a similar approach to what James Dyson advocates for industrial design and uses for building vacuum cleaners, washing machines or hand dryers. It is the "intersection of liberal arts and technology" that Steve Jobs has nurtured at Apple.

Describing the software creation as an art meets few objection that I would like to address right away. The first is that art is not compatible with a commercial, for profit, enterprise. Rob Austin and Lee Devin reject the idea that artists "have it easier". "The idea that arts practice is less rigorous than business practice is a major misconception. We’re not proposing that you give up structure or discipline or fiscal responsibility." Artists creating wealth for themselves and the people and companies around them are not rare at all. Salvador Dali or Luciano Pavarotti are just two famous examples. Some artists are more gifted in financial matters, but that is true for any other profession. The Worldcom and Enron scandals or the recent banking crisis proved that samples of fiscal irresponsibility or lack of financial skills can be found among bankers and accountants as easily as in other domains.

The other objection I heard frequently is that creativity and art are not compatible with mundane tasks like testing or bug fixing. This objection comes from those that argue that SW development is not an art, but also from those that think that some parts of SW development should be considered art. The latter ones argue that “boring” tasks like testing and bug fixing should be done by less creative (usually an euphemism for “lesser”) software professionals, because those tasks kill the creativity. I have already argued (over-passionately) why the SW developers should do testing. Looking at the artistic professions, I would say that the most successful artists always balance between creative freedom and discipline. The theater actors perform their most successful plays almost every night for years, although the creative phase is largely completed shortly after the opening night. Sting performed "Every breath you take" during his concert in Helsinki, although you can read it from his interviews that he's been past that phase of his career for a long time. They carry on with their “boring” tasks and still find the way to continuously create something new and interesting.

Collective art.  Although the mobile applications for iPhone and Android have lowered significantly the entry barrier, developing software still involves at least a handful of people. Projects with few dozen people are quite common and products including software made by few hundred people are not rare. And some great products require thousands of people, even for efficient teams like Apple's iPhone.

A collective art perspective would set SW developers in a similar category as actors, symphonic orchestra or jazz players, team sports athletes. While the mentioned professions have a lot to teach about art and teamwork there is one impediment in the direct transfer of knowledge to software development: size. None of the arts mentioned requires thousands of people. Mass production employs thousands of people, but we have already seen that mass production way of organizing work is not suitable for software development. In our times, military operations are probably the only thing similar in profile and size.

Unscripted collective art. A play or a symphony are based on very detailed scripts or scores. The score of a symphony specifies the notes (i.e. the sounds) that need to be played, in a very strict sequence and with volume indications (piano, forte, etc). It also specifies the relative duration of each note, with tempo indications marking the changes in the speed of execution. Some composers go as far as recording the beats per minute in the score, which makes the time marking absolute. The play scripts record a similar pattern: you have there the words that need to be spoken and their order, stage design suggestions, interpretation suggestions, etc.

Is there any room left for creativity in the scripted arts? Apparently yes, if you consider that the same play or piece of music can have so many different masterpiece interpretations. Not enough, though, if you think that ignoring some of the absolute markings of the script is a common way of creating new interpretations. And far from enough if you think about a jazz ensemble improvising extensively during their shows. 

There are at least two levels on which the software is unscripted. On one level, there is no recipe on how to write software. A software program is never written twice. If you need the same code in two different contexts you just wrap it in a separate library and re-use it. If the copyright (2) will prevent you from re-using the code than you have to write it in a different manner anyway. A software programmer is constantly faced with a new situation, for which there is no pre-defined script. The design and architectural patterns provide a certain level of relief, but they are more high level guidelines than true scripts on how to write software.

At business level, the product that needs to be built is only vaguely known before it is built. Walter Isaacson's biography of Steve Jobs contains a story about the initial design of the iPhone. The team had used the iPod as the basis of their design, so the initial prototypes had a scroll wheel. Only after seeing the prototypes from another team working on a tablet product had they realized that the phone must use a multi-touch screen with a virtual keyboard and rubber band scrolling. The iPhone was a completely new design at the time, so one might think that these type of uncertainties are not applicable to established product categories. For established products the situation is much worse, as exemplified by a short review of the last 10 years in the mobile phone industry. Motorola was the dominant mobile phone manufacturer in the '90s and the script looked great for their business. Then the world moved into the 3G era and Motorola missed that move. Nokia, the new dominant mobile phone maker, changed the script again, introducing camera phones that would eventually become a convenient alternative to point-and-shoot cameras. Samsung followed, but others, like Ericsson or Siemens failed. Then RIM changed the norm again with an email friendly phone that supported perfectly their email services. Apple introduced the iPhone in 2007 and the AppStore in 2008 probably thinking about market dominance for the years to come, only to see their dreams shattered by Google's Android. The mobile phone industry is not the exception, rather a fitting example of the disruptive innovation, a theory aggregated together very well by Clayton M. Christensen.

The unscriptedness seems to be very difficult to manage inside companies. It requires massive changes at the core of the company to modify the script. Despite their huge financial resources, Microsoft and Google find it very difficult to break out of their previously successful patterns as a desktop software company and as a search and advertising based one, respectively. In case of Apple it took the energy, dedication and dictatorship of an exceptionally gifted CEO to change the vision of the company in the late '90s. In the bigger picture the lack of adaptation inside companies is not a problem. Those which do not adapt have their role diminished or disappear altogether and new companies take the lead.

If you reached the end of this long post, then I owe you a word of caution. The model presented above is based on intuition. I do not have a proof of the model and the authors that I quoted did not have one either. The fact that I am not alone in this way of thinking does not validate the model in any way. The fact that many great minds of the past shared the intuition of a flat Earth placed in the center of the Universe did not constitute proof.

If the unscripted collective art model is a valid one, then it might be possible to combine the learning from the collective arts on how artists develop their skills and how ensembles work together, with the lessons from the military on coordination of a large number of people in a rapidly changing environment. It is not an easy task, but the alternative of figuring out everything from scratch is even more daunting.

--------------------------------------

(1) Dave Snowden provides an interesting new framework for measuring and guiding complex organizations, including those creating software.

(2) Isn't it interesting that software uses business practices that have been invented for protecting arts?