The unscripted collective art of software engineering

Are we human or are we dancer?
My sign is vital, my hands are cold
And I'm on my knees looking for the answer
Are we human or are we dancer?
"Human" by "The Killers"

The history of software development places the roots of modern software in the aftermath of WWII, making it a young trade at the scale of human history. Like the aviation or car industries before it, software started as an activity carried out by a few pioneers, individually or in small groups. A typical child of the 20th century, software turned into a mass production activity as soon as its value became clear and the advances in computer hardware and programming languages allowed it.

As an industry, the software development was faced with and interesting question: "How should the teams of people working on software be organized?". I started my career in the software industry at the end of '90s. The prevailing model at the time was the waterfall model, sometimes drawn in the fancy shape of the V-model. The linear flow of activities in the model reminds me of the successful industrial manufacturing, with the assembly line as the centerpiece. The software is passed forward, from one end of the line to the other, in a well organized fashion. The choice of production model should not come as a surprise. The assembly line had been the most successful way of organizing mass production, so it was probably viewed as the best practice of the day.

The limitations of the waterfall model have lead to the emergence of a new style of software development: "one that breaks down hierarchy, that features dynamic social structures and communication paths, and that values immediacy. This [...] style often bears the label “agile,” but that is just one of many characterizations of a broad new way of developing software [...]." The new style is iterative, with organizations today using 1-6 weeks iteration cycles and continuous integration of software. The natural consequence of such a short iteration cycle is that the linearity and sequencing are completely abandoned: testing can start before there is any code (e.g. test driven development), design can happen after the code is already functional (refactoring), requirements may be discovered at any point in time. If I had to pick a single most important contribution of the agile SW development community, the departure of SW creation from the assembly line model would be my choice.

Why did the assembly line organization not work for SW creation? The best hypothesis I have seen is that the SW organizations are closer to self-organizing systems than to assembly lines. In "Agile Software Development with Scrum" Ken Schwaber argues that the software activities fall into the realm of complex systems (as opposed to simple or chaotic systems). Complexity science, the discipline that studies self-organizing systems, would then explain very well why a simplistic linear model fails when many teams are involved in the creation of the SW (1). Although the complexity theory would explain why the waterfall SW organizations cannot succeed, it does not give much guidance on the practical matters of how to deal with SW teams in their daily work.

Understanding what the software development is could be one way to figure out how the software developers should approach their work. As a graduate of an engineering school I am sensitive to the arguments of David L. Parnas and Steve McConell, who favor an engineering approach for the development of commercial software. Parnas compares the separation of the software engineering and computer science with the similar split of electrical engineering and physics. "An Engineer", says Parnas "is a professional who is held responsible for producing products that are fit for use". That type of responsibility requires different training and work methods than what scientists need.

While the engineering approach to software appeals to me, I would point out that not all engineering spun off of science. The civil engineering activities developed out of necessity. The core of the mathematics used in construction design had been developed by the ancient Egyptians and Greeks, but the modern science needed by civil engineers emerged in the 17th century with Isaac Newton's classical mechanics. Later, advances in physics and chemistry have provided engineers with new or improved construction materials, better heat insulation and illumination. However, the lack of modern science has not prevented people from using empirical methods to build impressive constructions during the ancient times or the Middle Age. After the science became advanced enough, civil engineering also morphed into a modern engineering activity, i.e. "the application of scientific and mathematical principals toward practical ends". Re-basing the civil engineering on science has lead to an interesting phenomenon: a distinction between architecture and civil engineering started to develop. I think this is no coincidence. There is a considerable overlap between architecture and construction engineering, but the architects want to make a clear statement that apart from science and mathematics there is also art needed when designing spaces inhabited by people.

I believe that the software today is still in a mixed, undifferentiated state. Although voices like Parnas' and McConell's get louder, there is still a great deal of confusion between software engineering and computer science. In addition, there is no formal recognition yet with respect to the empirical and artistic aspects of software development. I choose to describe the software development as an unscripted collective engineering art.

Art. Let me first clarify that art in this context may refer to either art or craft. The difference between the two does not affect the rest of the arguments, so I will use art and craft interchangeably. 

There have been already pledges to include the software among the artful activities. In The Pragmatic Programmer. From Journeyman to Master, Andrew Hunt and David Thomas make extensive arguments in the same direction: "Programming is a craft. [...] As a programmer you are part listener, part adviser, part interpreter and part dictator. You try to capture elusive requirements and find a way of expressing them so that a mere machine can do them justice."  Fred Brooks attributes much of the joy of the software development to the artistic freedom programmers have: "The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures."

Jonathan Wallace considers that the software development is completely dominated by art and it should free itself from the engineering approach. That goes one step too far for my taste. The software products solve practical problems, so I prefer to keep the engineering sense of "fit for purpose" in the software. Much like the architecture, I see the software as balance between art and engineering. It is a similar approach to what James Dyson advocates for industrial design and uses for building vacuum cleaners, washing machines or hand dryers. It is the "intersection of liberal arts and technology" that Steve Jobs has nurtured at Apple.