Confirmation or falsification in your user research?
Whenever you run a study, are you trying to confirm or falsify your beliefs?
Many times, when I talk to designers about their user research practice, they (quite subconsciously) tend to say that they believe it is important because it confirms their design choices. This statement always makes me uncomfortable.
Daniel Kahneman says that confirmation bias is probably one of the most potent biases we, as humans, experience. It causes tunnel vision, the fact that we only see about 5% of what’s around us. It is due to the fact that we only look for what we hope to find. This also means that we miss out on a lot of things that we either don’t expect or we hope are not there.
I’ve been recently reading a series of lectures offered by Richard Feynman titled “The meaning of it all”. Feynman, when he describes what science is, says:
“…science is a way of finding things out. This is method based on the principle that observation is the judge of whether something is so or not. All other aspects and characteristics of science can be understood directly when we understand that the observation is the ultimate and final judge of the truth of an idea. But ‘prove’ used in this way really means ‘test’ in the same way that a hundred-proof alcohol is a test of the alcohol, and for people of today the idea really should be translated as, ‘The exception tests the rule’. Or, put another way, ‘The exception proves that the rule is wrong.’ That is the principle of science. If there is an exception to any rule, and if it can be proved by observation, that rule is wrong.”
He goes further with this thought and says:
“The exceptions to any rule are most interesting in themselves, for they show us that the old rule is wrong. And it is most exciting, then, to find out what the right rule, if any, is. The exception is studied, along with other conditions to produce similar effects. The scientist tries to find more exceptions and to determine the characteristics of these exceptions, a process that is continually exciting as it develops. He does not try to avoid showing that the rules are wrong: there is progress and excitement in the exact opposite. He tries to prove himself wrong as quickly as possible.”
As I read it, it struck me — so many of us, designers, are not looking for “proving ourselves wrong as quickly as possible”. Quite the opposite: we try to prove ourselves right and we see exceptions as a confirmation of the rule. In other words, we skillfully avoid falsifying our preconceptions and design assumptions, and seek proof that whatever we put in front of our users, is correct. We are aiming to prove ourselves right rather than wrong.
It is, of course, quite gratifying. When our test subjects see what we see, we feel professional and skillful. But at the same time, we most likely miss the most insightful lessons we could get out of that test. Lessons that might expand our design worldview, that might make us think harder and perhaps come up with propositions that are revolutionary rather than repetitive.
Think about that: what would happen if you chose to approach the next user research study as an exercise in falsification rather than confirmation? How would you change the way you prepare it? The way you ask questions? The way you analyze the results and apply the findings to the next iteration of your solution? What if you chose to “find excitement” in determining the characteristics of the exceptions you find and see how they are changing your design rules? Would it take you on a different design path? Would it be more rewarding? I hope it will.
Aga Szóstek, PhD is an experience designer with over 19 years of practice in both academic and business world. She is an author of “The Umami Strategy: stand out by mixing business with experience design”, a creator of tools supporting designers in the ideation process: Seed Cards and the co-host in the Catching The Next Wave podcast.
