Member-only story
Listen to users, but only 85% of the time
How Black Swans Can Save Innovation in a Data-Driven World
TL;DR: Data-driven design is a proven success factor that more and more digital businesses embrace. At the same time, academics and practitioners alike warn that when virtually everything must be tested and proven with numbers, that can stifle creativity and innovation. This article argues that Taleb’s Black Swan theory can solve this dilemma. It shows that online experimentation, and therefore digital design, are fat-tailed phenomena and, hence, prone to Black Swans. It introduces the notion of Black Swan designs — “crazy” designs that make sense only in hindsight — along with four specific criteria. To ensure incremental improvements and their potential for innovation, businesses should apply Taleb’s barbell strategy: Invest 85‒90% of resources into data-driven approaches and 10‒15% into potential Black Swans.
Over the past two decades, we have witnessed a shift from classical design to what John Maeda calls computational design (Maeda*, 2019: pp. xi-xii). With this shift — and amplified by works such as The Lean Startup (Ries*, 2008) and Sprint (Knapp, Zeratsky, & Kowitz*, 2016) — came a strong focus on minimum viable products (MVPs), continuous iteration, and user testing. One specifically popular form of testing in this context is online experimentation (also called A/B testing). That is, an (un-)finished digital design can be put live first and then quantitatively evaluated with huge amounts of real users. The possibility to do this is one of the core features of computational design (Maeda, 2019: pp. 155–157). Businesses can benefit significantly from leveraging this approach. For instance, Sheppard, Sarrazin, Kouyoumjian, & Dore (2018) have found that companies that employ data-driven design, continuous iteration, and user centricity “[outperform] industry-benchmark growth by as much as two to one.” All of these have become staples in many digital businesses that live by the rule of “if it’s not tested it’s just another opinion.” In this respect, especially online experimentation is sometimes considered a jack-of-all-trades solution (Speicher, 2021a). There are companies today that have such extensive experimentation infrastructures that one can assume virtually everything is an online experiment (cf. Bakshy, Eckles, &…
