# How to build perfect software There are many aspects to creating an engineering team that builds wisely. You need diversity, freedom, challenge, resources, and so on, but even if you have all the right ingredients, the default processes that skilled engineers and designers naturally develop will result in complex, hard-to-use implementations with no market. The classic errors are to focus on ideas and objectives, not problems; to focus on the wrong problems; to misjudge the value of solving problems; not using ones' own products; and in many other ways to misjudge the real market. In this article I explain how most organizations end up on the wrong track, and then I provide a broad overview of the process I use to create and guide competent teams into producing valuable solutions. It begins with a simple realization: we don't know what we need to build until we start building it. --- The way most organizations build software today is inefficient, misguided, and doesn't scale. They are slow-motion tragedies but can be fun to watch from a safe distance. The most popular design process in large businesses today seems to be form of "Idea Driven Development". This feeds off the belief that the only thing you need to make huge profits are great ideas. The theory goes that ideas are rare and valuable, so the trick is to capture them. It's like non-musicians being awed by a guitar player, not realizing that great talent is so cheap it literally plays on the streets for coins. The main outputs are expensive "ideations", concepts, design documents, and finally products that go straight into the trash can. It works as follows: The Creative People come up with long lists of "we could do X and Y". I've seen endlessly detailed lists of everything amazing a product could do. Once the creative work of idea generation has happened, it's just a matter of implementation, of course. So the managers and their consultants pass their brilliant, world shattering ideas to "user experience" designers. These talented designers take the tens of ideas the managers came up with, and turn them into hundreds of amazing, world-changing designs. These bountiful and detailed designs get passed to engineers, who scratch their heads and wonder who the heck came up with such silly nonsense. They start to argue back but the designs come from up high, and really, it's not up to engineers to argue with creative people and expensive consultants. So the engineers creep back to their cubicles and hotdesks, humiliated and threatened into building this gigantic but oh so elegant pile of crap. It's bone-breakingly hard work since the designs take no account of practical costs. Minor whims might take weeks of work to build. As the project gets delayed, the managers bully the engineers into giving up their evenings and weekends. Eventually, something resembling a working product makes it out of the door. It's creaky and fragile, complex and ugly. The designers curse the engineers for their incompetence and pay more consultants to put lipstick on the pig, and slowly the product starts to look a little nicer. By this time, the sales managers have started to try and sell the product and they find, shockingly, that no-one wants it. After twelve months of intense marketing, the product still isn't making profits. Worse, it suffers dramatic failures. The company quietly shelves it, buys a competing product from a small start-up and rebrands that as its own Version 2. #### The Lesson The main lesson is quite straight-forward but hard to swallow: ideas are cheap. No exceptions. There are no brilliant ideas. The starting point for a good design process is to collect problems that confront people. The second step is to evaluate these problems with the basic question, "how much is it worth to solve this problem?" Having done that, we can collect a set of problems that are worth solving. Good solutions to real problems will succeed as products. Their success will depend on how good and cheap the solution is, and how important the problem is. But their success will also depend on how much they demand in effort to use, in other words how simple they are. --- The second most common design process is led by engineers, and fails in a very different way. Really good engineering teams and small firms can usually build good products. But the vast majority of products still end up being too complex and less successful than they could be. Management correctly identifies some interesting and difficult problem with economic value. In doing so they already leapfrog over any team following the Idea driven process I described above. The team starts to build prototypes and core layers. These work as intended, which empowers everyone involved. They go off into intense design and architecture discussions, coming up with elegant schemas that look beautiful and solid. Management, who are usually competent engineers themselves, come back and challenge team with yet more difficult problems. We tend to equate value with cost, so the harder the problem, and more expensive to solve, the more the solution should be worth, in their minds. The team is full of highly competent engineers who love building stuff, and so that's exactly what they do. They build and build and build and end up with massive, perfectly designed complexity. The products go to market, and the market scratches its head and asks, "seriously, is this the best you can do?" People do use the products, especially if they aren't spending their own money in climbing the learning curve. Management gets positive feedback from its larger customers, who share the same idea that high cost (in training and use) means high value. Meanwhile somewhere across the world, a small team is solving the same problem, and a year later they smash the market into little pieces. #### The Lesson Organizations that fall into this trap are characterized by a team obsessively solving the wrong problems to the point of ridiculousness. The products tend to be large, ambitious, complex, and unpopular. Much open source software is the output of this processes. It is insanely hard for engineers to stop extending a design to cover more potential problems. They argue, "what if someone wants to do X?" but never ask themselves, "what is the real value of solving X?" The main lessons are: * Making stuff that you don't immediately have a need for is pointless. Doesn't matter how talented or brilliant you are, if you just sit down and make stuff, you are most likely wasting your time. * Not all problems are equal. Some are simple, and some are complex. Ironically, solving the simpler problems often has more value to more people than solving the really hard ones. So if you allow engineers to just work on random things, they'll focus on the most interesting but least worthwhile things. * Engineers and designers love to make stuff, and this inevitably leads to complexity. It's crucial to have a ***stop*** mechanism: a way to set short and hard deadlines that force engineers to make smaller, simpler solutions to just the most crucial problems that are in the *critical path*. --- ### How to make perfect software every time We start with a realization: we don't know what we have to make until after we start making it. Coming up with ideas, or large-scale designs isn't just wasteful, it directly inhibits truly accurate solutions. The really juicy problems are hidden away in valleys and crevices of the market landscape, and any activity except active scouting creates a fog that hides those crevices and valleys. You need to keep mobile, pack light, and move fast. The process works as follows: We collect a set of interesting problems (by looking at how people use technology or other products) and we sort these from simple to complex, and observing patterns of use. We take the simplest and most dramatic problem and we solve this with a minimal plausible solution, something akin to an MVP but far cheaper and nastier. This is our first "patch". Our process is based entirely on patches. A patch solves exactly **one** genuine problem in a **brutally** minimal fashion. We apply only **one** measure of quality to patches, namely "can this be done any simpler while still solving the stated problem?" We can measure complexity in terms of concepts and models that the user has to learn or guess in order to use the patch. The fewer, the better. A perfect patch solves a problem with zero learning required by the user. Our product development consists of a patch that solves the problem "we need a proof of concept" and then evolves in an unbroken line to a mature series of products, through hundreds or thousands of patches piled on top of each other. We do not do anything that is not a patch, and we do not create patches for any reason other than solving a problem. We enforce this rule with a formal process. We build our projects into a supply chain where each project can provide problems to its "suppliers" and receive patches in return. This supply chain creates the *stop* mechanism - when people are impatiently waiting for a solution from us, we necessarily cut our work short and forgo the "nice to have" extensions (aka technical debt). Individuals within the organization are free to work on any of the organization's projects, and provide patches at any place they feel it's worthwhile. No individuals "own" any project, except to enforce the formal processes. A single project can have many variations, each a collection of different, competing patches. Projects export formal and documented interfaces that evolve from *draft* through *stable* to *deprecated*, such that upstream (client) projects do not need to be aware of changes happening within supplier projects. Thus, multiple supplier projects can compete for client projects, in effect creating a free and competitive market. We tie our supply chain to real users and external clients, and we drive the whole process by rapid cycles so that a problem received from outside users can be analyzed, evaluated, and solved with a patch in a few hours. At every moment from the very first patch, our product is shippable. This is essential, because a large proportion of patches **will** ultimately be wrong (10-30%), regardless of what process you use. Only by giving the product to users can we know which patches have become problems and themselves need solving. This process is a form of "hill climbing algorithm", a reliable way of finding optimal solutions to the most significant problems in an unknown landscape. You don't need to be a genius to use this approach successfully, you just need to be able to see the difference between the fog of activity and real progress towards solving real problems for real users. A really good manager with a good team can use this process to build world-class products, rapidly and accurately. But to get the most out of it, the manager has to use the product continuously, from day 1, and develop his or her ability to smell out problems such as inconsistency, surprising behavior, and other forms of friction. We naturally overlook many annoyances but a good designer picks these up, and thinks about how to patch them. Design is about removing friction in the use of a product. --- This aproach is a reliable, repeatable way of developing world-class products that delight users with their simplicity and elegance. This process organizes people into flexible supply chains that are able to navigate a problem landscape rapidly and cheaply. They do this by building, testing, and keeping or discarding minimal plausible solutions, called "patches". Living products consist of long series of patches, applied one atop the other. The process works best for software design, and some open source projects already work in this way. However many or most still fall into classic traps such as over-engineering and "what if" design. Wikipedia is a good example of how this approach can be applied to the non-software domain, and the "shanzai" model used in China applies something similar to hardware. I use this approach daily to build, and help others build world-class commercial products.