In my previous article, “The Technology Executive and the Software Craftsman,” I discussed how software developers like me who were trained in the earlier software engineering disciplines that were managed based on the waterfall model felt that there was something fundamentally flawed with the model. We knew that close collaboration between developers and clients usually yielded better results, but the formal waterfall process created barriers to that collaboration. In this article, I will explore the fundamental flaws of the assumptions underlying software engineering as managed by waterfall models. I will then discuss how Agile software development methods address those flaws by addressing the inherent design-centered process that we know as software development harnessing its people-centric and social aspects.
Engineering Software as a Flawed Assumption
While the waterfall model, when applied to software development, has taken its share of disparagement over the years, the failings of the waterfall model itself are merely symptoms of what I see as a flawed assumption about software development. That assumption being that software is engineered at all. Engineering as defined by Wikipedia is “the application of mathematics, empirical evidence and scientific, economic, social, and practical knowledge in order to invent, innovate, design, build, maintain, research, and improve structures, machines, tools, systems, components, materials, and processes.” Engineering frequently assumes formal methods and disciplined sequencing of development steps. It is no accident that when the development of software is viewed through the lens of engineering that the waterfall model takes center stage as the incarnation of the software development process.
Many a long-time practitioner of software development believed that viewing software development as an engineering discipline was attempting to shoehorn a more iterative process into a formal engineering discipline when in fact the process of developing software was another animal altogether. In current software development thinking, we talk more about software as being designed rather than engineered. Part of the Wikipedia’s definition of Design states, “Designing often necessitates considering the aesthetic, functional, economic and sociopolitical dimensions of both the design object and design process. It may involve considerable research, thought, modeling, interactive adjustment, and re-design. Meanwhile, diverse kinds of objects may be designed, including clothing, graphical user interfaces, skyscrapers, corporate identities, business processes and even methods of designing.” When we view software development through the lens of design which involves interaction, collaboration, and adjustment, the problems with the engineering-centric view become apparent.
Software Design as a Better Paradigm
Engineering infers technical competence, adherence to standards, sequenced collaboration, and top-down management to enforce process and standards since those are assumed to determine timeliness and quality. Design infers technical competence, collaboration, managed discord, and servant leadership to remove obstacles to progress in response to uncertainty. So why does assuming software development is an engineering discipline so often result in late projects and dubious quality? There are two reasons. First, engineering assumes that a problem can be almost completely understood up front, and only minor adjustments will be needed on the back end of the process to account for requirements changes. Second, and in what is bound to be a controversial comment, engineering values process and standards over individual judgment and intuition. In practice, because software systems tend to be complex and involve the sometimes subjective judgments of stakeholders, we seldom have a clear picture of the problem to be solved early in the software development process. Rather, the requirements become clearer over time, sometimes taking 30-50% of the project timeline to become well understood and enumerated. Engineering processes are not tolerant of such uncertainty while design process assumes uncertainty.
Design is a better model for software development simply because the process of design acknowledges the uncertainty of requirements and seeks to adjust and redesign as needed. At this point, some will say, “but design is part of any disciplined software development process.” To those I say, treating design as a discrete component of the software development process denies the reality of rolling uncertainty and evolving clarity of the requirements. Rather, the entire software development process from end-to-end is an ongoing design process where the understanding of the problem evolves and necessitates almost continuous solution readjustment.
Clarity Through a Shifting Paradigm
When we get used to the idea that software development is better thought of as design process than as engineering discipline, key discriminators of success begin to emerge. First, we acknowledge that project timelines cannot be established at the beginning of the project. My own experience has shown that committed timelines can only be established after about 30-50% of what will become the total timeline has elapsed. Knowing this is important because executives and managers may have differing expectations about the timeliness of software delivery that need to be managed. This is not to say the Agile methods cannot establish and meet deadlines but rather that deadlines are established well after the project is underway. This is a fundamental shift in thinking about and setting delivery expectations. Second, the role of the project manager shifts from managing to deadlines to managing collaboration and removing obstacles to success. This is a critical distinction because many formal project management methods lean toward top-down management which is not amenable to iterative collaboration. It’s important that emphasis is placed on finding project managers who act more servant leaders versus managers of milestones since managing uncertainty means that risks will arise that must be mitigated collaboratively. Finally and most importantly, viewing software development as design highlights the fact that software development is an inherently social activity where interactions between people are valued over formal processes which seek to control such interactions. I have written extensively about the necessary tensions that exist in software development that need to be managed in order to produce great software. Engineering approaches seek to eliminate these tensions using process while design approaches acknowledge the existence of tensions and harness them to produce better results.
The Essence of Agile
Agile software development methods continue to evolve and are far from perfect. However, Agile methods more closely match the iterative and collaborative processes that align more closely with the true nature of software development. Still, engineering mindsets persist and manifest themselves in Agile shops. Agile by its very nature replaces tightly controlled engineering steps with a necessarily fuzzy set of guidelines. For example, the Agile Manifesto states the following:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
I run into Agile practitioners who seem to replace the word “over” with the phrase “instead of” in the above guidelines. I don’t mention this to impugn but rather to highlight that dogma that is necessary for engineering discipline sometimes creeps into methods that are pragmatic by their very essence. Because Agile is inherently pragmatic and experimental, I find it valuable to continuously reinforce the “spirit” of Agile over its “letter.” Here are few examples.
Spirit versus Letter
The Agile Scrum method has a number of rituals that are intended to promote collaboration. Among those are sprint planning, stand-ups, showcases, and retrospectives. When we first adopted Agile Scrum at one job, we tried to be rigorous with requiring that these rituals were held in uniform ways. Over the course of time, some in the development staff began commenting on how they were not getting value from some of the rituals, for example, stand-ups and lobbied to do away with them. We went back and forth about whether stand-ups were needed. The more dogmatic supporters of Agile maintained that they were a required part of the process and that their use should be enforced. The more pragmatic Agile supporters argued that stand-ups were not needed because they were accomplished in other ways. After listening to both factions, I asked what was the “spirit” of a stand-up. Obviously, the spirit was to ensure the teams had a daily progress checkpoint. I found that while some teams needed the structure of a formal meeting because the team members worked independently, other teams worked more collaboratively on a daily basis and kept themselves more constantly informed of progress. In the former case, the team needed stand-ups to assess progress while the latter team had ad hoc stand-ups without the need for formality.
So what are the lessons to be taken away? Software development is an inherently social process where technically competent people come together with their customers and iterate through a process that more closely resembles design than engineering. While there are some required steps needed to ensure fitness and quality, much of the Agile process in inherently fuzzy and open to pragmatic experimentation and adjustment. In Agile, one must keep in mind the spirit of the tenets of the Agile process and be careful to adhere to those intentions lest unreasoning dogmatism creep in and attempt to bend the people to the process rather than bend the process to the people. This does not mean that Agile development should devolve into an uncontrolled process. There are firm needs to enforce application lifecycle models and quality gates to ensure that best practices are followed and that traceability and validation can be maintained. But these necessities need to be balanced with the flexibility and pragmatism that Agile represents.
When a proper balance is struck between necessary practices and areas where experimentation and flexibility are promoted, you will find that your Agile teams will work more closely together to find solutions to their own problems. When teams are permitted to be experimental, they also tend to be more cohesive leading to more engaged and satisfied team members. If you focus on attending to the essence of Agile over strict doctrine, I think you will receive better results, and your teams will thank you for it.