Essential Agile

In my previous article, “The Technology Executive and the Software Craftsman,” I discussed how software developers like me who were trained in the earlier software engineering disciplines that were managed based on the waterfall model felt that there was something fundamentally flawed with the model. We knew that close collaboration between developers and clients usually yielded better results, but the formal waterfall process created barriers to that collaboration. In this article, I will explore the fundamental flaws of the assumptions underlying software engineering as managed by waterfall models. I will then discuss how Agile software development methods address those flaws by addressing the inherent design-centered process that we know as software development harnessing its people-centric and social aspects.

Engineering Software as a Flawed Assumption

While the waterfall model, when applied to software development, has taken its share of disparagement over the years, the failings of the waterfall model itself are merely symptoms of what I see as a flawed assumption about software development. That assumption being that software is engineered at all. Engineering as defined by Wikipedia is “the application of mathematics, empirical evidence and scientific, economic, social, and practical knowledge in order to invent, innovate, design, build, maintain, research, and improve structures, machines, tools, systems, components, materials, and processes.” Engineering frequently assumes formal methods and disciplined sequencing of development steps. It is no accident that when the development of software is viewed through the lens of engineering that the waterfall model takes center stage as the incarnation of the software development process.

Many a long-time practitioner of software development believed that viewing software development as an engineering discipline was attempting to shoehorn a more iterative process into a formal engineering discipline when in fact the process of developing software was another animal altogether. In current software development thinking, we talk more about software as being designed rather than engineered. Part of the Wikipedia’s definition of Design states, “Designing often necessitates considering the aesthetic, functional, economic and sociopolitical dimensions of both the design object and design process. It may involve considerable research, thought, modeling, interactive adjustment, and re-design. Meanwhile, diverse kinds of objects may be designed, including clothing, graphical user interfaces, skyscrapers, corporate identities, business processes and even methods of designing.” When we view software development through the lens of design which involves interaction, collaboration, and adjustment, the problems with the engineering-centric view become apparent.

Software Design as a Better Paradigm

Engineering infers technical competence, adherence to standards, sequenced collaboration, and top-down management to enforce process and standards since those are assumed to determine timeliness and quality. Design infers technical competence, collaboration, managed discord, and servant leadership to remove obstacles to progress in response to uncertainty. So why does assuming software development is an engineering discipline so often result in late projects and dubious quality? There are two reasons. First, engineering assumes that a problem can be almost completely understood up front, and only minor adjustments will be needed on the back end of the process to account for requirements changes. Second, and in what is bound to be a controversial comment, engineering values process and standards over individual judgment and intuition. In practice, because software systems tend to be complex and involve the sometimes subjective judgments of stakeholders, we seldom have a clear picture of the problem to be solved early in the software development process. Rather, the requirements become clearer over time, sometimes taking 30-50% of the project timeline to become well understood and enumerated. Engineering processes are not tolerant of such uncertainty while design process assumes uncertainty.

Design is a better model for software development simply because the process of design acknowledges the uncertainty of requirements and seeks to adjust and redesign as needed. At this point, some will say, “but design is part of any disciplined software development process.”  To those I say, treating design as a discrete component of the software development process denies the reality of rolling uncertainty and evolving clarity of the requirements. Rather, the entire software development process from end-to-end is an ongoing design process where the understanding of the problem evolves and necessitates almost continuous solution readjustment.

Clarity Through a Shifting Paradigm

When we get used to the idea that software development is better thought of as design process than as engineering discipline, key discriminators of success begin to emerge. First, we acknowledge that project timelines cannot be established at the beginning of the project. My own experience has shown that committed timelines can only be established after about 30-50% of what will become the total timeline has elapsed. Knowing this is important because executives and managers may have differing expectations about the timeliness of software delivery that need to be managed. This is not to say the Agile methods cannot establish and meet deadlines but rather that deadlines are established well after the project is underway. This is a fundamental shift in thinking about and setting delivery expectations. Second, the role of the project manager shifts from managing to deadlines to managing collaboration and removing obstacles to success. This is a critical distinction because many formal project management methods lean toward top-down management which is not amenable to iterative collaboration. It’s important that emphasis is placed on finding project managers who act more servant leaders versus managers of milestones since managing uncertainty means that risks will arise that must be mitigated collaboratively. Finally and most importantly, viewing software development as design highlights the fact that software development is an inherently social activity where interactions between people are valued over formal processes which seek to control such interactions. I have written extensively about the necessary tensions that exist in software development that need to be managed in order to produce great software. Engineering approaches seek to eliminate these tensions using process while design approaches acknowledge the existence of tensions and harness them to produce better results.

The Essence of Agile

Agile software development methods continue to evolve and are far from perfect. However, Agile methods more closely match the iterative and collaborative processes that align more closely with the true nature of software development. Still, engineering mindsets persist and manifest themselves in Agile shops. Agile by its very nature replaces tightly controlled engineering steps with a necessarily fuzzy set of guidelines. For example, the Agile Manifesto states the following:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

I run into Agile practitioners who seem to replace the word “over” with the phrase “instead of” in the above guidelines. I don’t mention this to impugn but rather to highlight that dogma that is necessary for engineering discipline sometimes creeps into methods that are pragmatic by their very essence. Because Agile is inherently pragmatic and experimental, I find it valuable to continuously reinforce the “spirit” of Agile over its “letter.” Here are few examples.

Spirit versus Letter

The Agile Scrum method has a number of rituals that are intended to promote collaboration. Among those are sprint planning, stand-ups, showcases, and retrospectives. When we first adopted Agile Scrum at one job, we tried to be rigorous with requiring that these rituals were held in uniform ways. Over the course of time, some in the development staff began commenting on how they were not getting value from some of the rituals, for example, stand-ups and lobbied to do away with them. We went back and forth about whether stand-ups were needed. The more dogmatic supporters of Agile maintained that they were a required part of the process and that their use should be enforced. The more pragmatic Agile supporters argued that stand-ups were not needed because they were accomplished in other ways. After listening to both factions, I asked what was the “spirit” of a stand-up. Obviously, the spirit was to ensure the teams had a daily progress checkpoint. I found that while some teams needed the structure of a formal meeting because the team members worked independently, other teams worked more collaboratively on a daily basis and kept themselves more constantly informed of progress. In the former case, the team needed stand-ups to assess progress while the latter team had ad hoc stand-ups without the need for formality.

Lessons

So what are the lessons to be taken away? Software development is an inherently social process where technically competent people come together with their customers and iterate through a process that more closely resembles design than engineering. While there are some required steps needed to ensure fitness and quality, much of the Agile process in inherently fuzzy and open to pragmatic experimentation and adjustment. In Agile, one must keep in mind the spirit of the tenets of the Agile process and be careful to adhere to those intentions lest unreasoning dogmatism creep in and attempt to bend the people to the process rather than bend the process to the people. This does not mean that Agile development should devolve into an uncontrolled process. There are firm needs to enforce application lifecycle models and quality gates to ensure that best practices are followed and that traceability and validation can be maintained. But these necessities need to be balanced with the flexibility and pragmatism that Agile represents.

When a proper balance is struck between necessary practices and areas where experimentation and flexibility are promoted, you will find that your Agile teams will work more closely together to find solutions to their own problems. When teams are permitted to be experimental, they also tend to be more cohesive leading to more engaged and satisfied team members. If you focus on attending to the essence of Agile over strict doctrine, I think you will receive better results, and your teams will thank you for it.

Best,

Charlie

Harmonizing the Voices: Getting Software Usability Right

Getting the chance to redesign a commercial software product is a rare opportunity in the life of a software development leader. I’ve been fortunate enough to have had that opportunity twice in the past decade. The first was a technology and user interface overhaul of an investment management system. The second was a complete redesign of an interactive response technology (IRT) system used to manage the operations, drug supply, and patient engagement for clinical trials. In both cases, these systems are intended to have a lifespan of a decade or more, almost an eternity for software. In the case of the IRT system, it is possible to have longitudinal clinical trials that last for 15-20 years that need continuous support on the same technology platform.

Because the lifespan of these systems is so long, you seldom get to redesign them. It was essential to the business that we got these redesigns right, especially the design of the user experience (UX). If we did not, we’d spend much valuable development capacity tweaking the UX designs rather than spending that capacity on new features that add to the value of the product and generate revenue for the business. Further, just like when you meet someone new, you only have one chance to make a good impression. If your UX is clunky or off-putting, at best you’ll frustrate users, and at worst you’ll diminish your competitive edge.  Moreover, by UX, I do not just mean the look and feel of the user interface (UI), I mean the entire experience of using your software from UI to workflow to help capabilities and the like.

Why it is hard to get UX right

Often when we think of UX design, we think of graphic artists and developers prototyping screens and making things look pretty. While that is part of design, the design process when done right is a social endeavor that brings together product stakeholders with often competing and sometimes contrary views and finding order in the chaos. The goal of the design engagement is to harmonize the many discordant voices by synthesizing their needs and opinions into a coherent design that not only meets their needs but provides a means to develop sustainably on the vision of the design.

Prospective customers often have firm notions of what they need and want from a system, but their view can be clouded by prior bad experiences. Users often have well-reasoned operational expectations of the system, but their view can be biased by how the system works now and sometimes opposed to trying new ways to accomplish the same task. Product managers are experts in understanding market, prospective customer, and user needs and translating those into product features. However, they frequently lack the capability to systematically and constructively solicit feedback on UX concerns and synthesize those into a holistic plan for the UX design. In short, there is an abundance of really great ideas but no good way to manage the social engagement that leads to great designs.

It is important to take a pause here and note two principals that guide my decision-making about UX design. The people who designed the UX for the previous system will inevitably have strong opinions about how the next system should behave. While their opinions need to be aired and considered, my first principle says, “If the same people who designed the last system design the next, the next system will end up looking suspiciously like the last.” This is not an indictment of the previous system designers but points to the need for a new synthesis of prevailing views focused through the lens of the business strategy. Further, there is a temptation to believe that UX design is little more that “twiddling some pixels” on a screen and that developers are suited to handle it. My second principle says, “Never have developers determine your UX design.” This is not an indictment of developers but points to the need for more rigor in UX design.

How to get UX design right

Getting to a great UX design is neither easy nor inexpensive. However, doing so pays dividends in the end. In the two system redesign cases I cite above, I engaged a local UX design firm to help us sort it out. Electronic Ink in Philadelphia, PA knew how to engage stakeholders and bring harmony to discordance. I cannot overemphasize the value companies such as this add. They have the people, process, and technology to engage the cacophony of stakeholders and harmonize their input into a solid path toward thoughtful and sustainable UX design.

Here’s how the engagement worked:

The first part of the engagement was intended to provide stakeholder reconnaissance for the current state of the existing system and begin to discover the required components of the future state. In the case of the IRT engagement, the various system users were defined as personae and were job shadowed and interviewed to assess the good and bad aspects of the current system state. The engagement revealed nine key problem areas. “Frustration scores” comprised of physical effort, cognitive load, task repeats, and error potential were produced for each of the nine areas. The frustration scores largely dictated the priority of each of the nine areas. Here is one such example:

current.png

The current state assessment revealed key issues and non-issues with the current design. This helped us understand what was working and what was not so that we could avoid past mistakes.

The engagement then turned to the future state. During this assessment, Electronic Ink designed possible solutions for addressing the problem areas identified in the current state as well as bring their design expertise on best practices to bear. Stakeholders were then engaged in an iterative process of discussing and refining the proposed solutions intended to address the nine problem areas identified in the current state exercise.  For reference, the “Excessive Navigation/Clicking“ issue was labeled as priority “#3”. Here are some examples of the suggested mitigation approaches:

future1future2

The by the end of the future state engagement, we had over 20 distinct UX solutions specifically designed to address the current state issues while bringing industry best practices to bear on the solution. The engagement also delivered a conceptual framework on which to hang key system features based on stakeholders’ understanding of user workflow.

The second part of the engagement involved the active collaboration of UX designers and developers. The Electronic Ink team used the artifact developed as part of the future state engagement to develop interactive wireframe models. Stakeholders then assessed the wireframe models iteratively. After each iteration, the wireframe models were refined and taken back to the stakeholders for reassessment. At the end of the iterative assessment process, we ended up with wireframe models of key usability patterns assembled into a “pattern library”. These usability patterns were used again and again within the product to produce repeatable and familiar workflows, visual controls, other cues making the product consistent and easy to use. Further, our UX staff was able to take the design patterns and extend them after the engagement ended.

Selling the spend to stakeholders

In the end, we invested heavily in the engagement, and I must admit that selling it to the executive team was no easy matter. The executives were skeptical about investing in a six-figure consulting engagement for something that seemed a rather simple and intuitive task. However, the longevity of the platform dictated the need to get the UX design right, and the competitive advantages that we could gain were compelling. Even then, I built in engagement checkpoints where we could end the engagement and money spent if value was not apparent. The two-part engagement was also valuable since we would take the findings from the first part and implement them ourselves if implementation seemed fairly obvious. In the end, we completed both parts of the engagement.

Internal stakeholders were probably the toughest to sell since they believed that resources within our company could produce a UX design. My prior experience with these engagements was able to convince them that while we had much of the capabilities needed, the ability to engage stakeholders and elicit insights and translate those to UX designs was not among those internal capabilities.

Finally, external customers were probably the toughest to engage. In the case of the investment management system, customers were eager to engage, but much coordination was needed since customers were widely distributed geographically. The IRT system case had a different problem. Pharmaceutical companies are correctly concerned about the security surrounding their intellectual property and seldom allow third parties to shadow their work. We were able to find proxies for end users since our company uses our product on behalf of customers in many cases.

Results

The results obtained were similar in both UX design engagements. Once we completed development and deployed the finished product, feedback from end users was phenomenal. Users not only praised us for “listening to them” and including features that mattered to them but commented how the system almost anticipated their needs before they realized they needed it. The consistent design patterns mitigated user training burdens. In the case of the investment management system, the improved design reduced the total number of distinct user screens from about 1500 in the previous system to about 800 in the redesigned system.

The product development teams not only delivered the new product but now had the tools to sustain the UX design in a systematic and disciplined way since the design patterns removed much of the guesswork for how the product would behave. However, most importantly, the business received a compelling new product to drive business growth for another decade.

Clearly your UX design engagement will be unique to your business. What is not unique is the social aspect of the engagement needed to get to a great design. Few companies are equipped to harmonize and make sense of the cacophony of great ideas found within their organization. If you have the opportunity to revisit the UX for an existing product or innovate on a new product, I strongly recommend seeking out a partner to help guide you. You will be amazed at the results and won’t have a moment’s regret. Oh, and your customers will be forever grateful for the thoughtfully designed and easy to use software products!

Best,

Charlie