Reconciling modern design practices with agile software development methods

Most software fails because it is poorly designed. Buggy apps, unfriendly enterprise systems, insecure websites, predatory games, hate-amplifying social networks, racist AI, and carbon-spewing cryptocurrencies make us miserable, cost us billions and exacerbate the climate crisis. Yet, software design is rarely studied (compared to, e.g., software testing) and is widely misunderstood. Software professionals tend to equate design with graphical interfaces (user experience design), system organization (architectural design), procedures (algorithm design) or feature selection (feature design). However, design is really just determining the properties of some object. Rather than a phase in an idealized lifecycle, design permeates the development process and is inextricable from both analysis and programming. "Design thinking" includes, but cannot be reduced to, problem-solving, planning, decision-making, learning or creativity. My research program is about improving software professional's design thinking and design outcomes. I propose three interconnected projects that come at design thinking from different angles: practices, refactoring and theory development. Practices. Numerous software design practices have been proposed but never empirically tested. We will conduct a series of controlled experiments to test three practices: pair designing, affinity mapping and generating multiple high-level design candidates. Each experiment is similar: software professionals are randomly assigned to the treatment or control group. They design a system on paper and these "paper prototypes" and graded to see if the practice leads to significantly better designs. Refactoring. Developers continuously reorganize software code to improve its structure ("refactoring"). However, there is no empirically-validated method of measuring structure quality. Instead, refactoring is a completely manual process based on experience, intuition, subjective criteria, and unvalidated professional guidelines. We will combine diverse software metrics into a sophisticated mathematical model of structure quality, and implement it as an easy-to-use tool that tells developers how a code change affects the system's structure. Theory Development. One of the most effective software design practices is called user-centered design (UCD). However, UCD is at odds with agile methods-the way most software teams work. We will conduct a longitudinal grounded theory study to understand how UCD and agile methods can be reconciled. This research will be disseminated to our industry partners via presentations and to the wider software engineering community through meetups, blogs, podcasts and professional articles. The tools and recommendations we generate will lead to more usable, useful, sustainable and maintainable software, helping Canada grow its $75 billion/year software industry.