Do we actually do proper research?
No, but for good reason.
Some time ago I had a conversation about research and experimentation with one of my best friends who has a doctorate in biochemistry and does scientific research for a living.
Something that he was adamant about was that what we product people do could hardly be called ‘research’ — at least not by scientific standards.
When I described what we call “hypotheses”, “experiments”, “research”, etc he couldn’t have been more offended by our loose definitions of his beloved scientific terms.
In his eyes we abuse the scientific process, take the concepts in name only and apply them with no consideration of any scientific rigor.
Two things which became apparent through my conversations with him were:
- How rarely we calculate (or even know about) appropriate sample sizes
- The fact we almost never have a ‘control group’
By scientific standards without appropriate sample sizes our results would be inconclusive and without a control group our experiment hardly stands up to the standards of scientific rigor and therefore any findings would be quickly dismissed.
But we don’t work in the field of scientific research so we live without them, and I also believe we do so for a good reason.
First, let’s look at sample size.
The hardcore scientific way would be to calculate the appropriate sample size for testing your hypothesis. But more often than not that number is going to be in the hundreds, perhaps thousands of more.
The reality is it’s hard to scale customer interviews into the hundreds and thousands. It’s just not viable in the product world to be interviewing that many people.
Unlike in the scientific world where it’s not unheard of to spend years, even decades researching a space, in product we don’t have that luxury.
Rather we’ve got to move at speed so we choose smaller sample sizes — and this is part of the tradeoff.
I’m sure some of you are thinking — “what about that commonly quoted 5–8 people sample size for usability testing?”
First off, I’d encourage you to read this thread by Jared Spool about why that 5–8 users statistic is a myth. He also has a longer version here.
But a further reason why this is a myth is that a definitive number would be difficult to pin down because it’s contextual. 5–8 might be correct for some context but certainly not all.
Samples sizes would change depending on how much margin of error you’d be willing to accept, how confident you wish to be in the results, how many users you have, total market size, and how you’ve segmented all that. All of this again is contextual and dependent on what the hypothesis you are trying to test is.
But as I said earlier it’s not so much a bad thing.
The downside of not reaching the appropriate sample size is that your results will contain baises, a high error margin and low confidence.
But we’re often ok with this as product people. In fact, I’d argue that in order to perform well in product you must be comfortable with this fact.
As Product Managers, we need to learn to make decisions in a fog, with limited information. This is where intuition and experience becomes a key part of decision making.
A core tenant as to why we seek to be data-informed, not 100% data-driven.
We trade off high confidence, error margin and unbiased results by sampling smaller sizes to keep moving at speed. We then fill the gaps with our interpretation of the data, intuition, experience and existing knowledge.
In saying all this I have found value in applying the rigor of sample sizes to your more quantitative research methods (surveys, multi-variant testing, etc) as they are often scalable enough to do in a robust way without incurring too much additional time/cost.
And the ‘control group’?
As for the control group, I’d argue that we don’t need one.
Product work is complex and ambiguous. We would struggle to design a test well enough where we were able to control all the parameters to have a control group.
Thus, if I was to compare what we do to the scientific world I would say we are comparable to research in behavioral science. Both deal with the vast complexities of human behavior.
It’s no surprise then that the behavioral science world often receives criticism over its methods and findings as their tests seldom pass the same scientific rigor found in other fields.
Thus similar to in product we rely on more anecdotal feedback. We acknowledge and accept that our experiments are contextual and may be impacted by other variables.
But not enough to degrade the value of the insights and research to render them unuseful. We are still able to find definitive patterns, common behavior sets, etc more than enough that’s required when it comes to product development.
Rather aim for continual discovery
Rather Jared Spool and many others, suggest focusing on continual discovery over fussing about what the ideal sample size is.
As an old colleague (and a mentor of mine in UX) once said, “you’re far better off doing 10 iterations of 8x people interviews than doing one iteration of 80, even if 80 people was the perfect sample size.”
Such is the world of product work — we need to move at speed, we can’t afford to spend years, not even months researching something.
Take too long and by the time you either craft and execute your experiment or action your research, the landscape may have changed again rendering all that work obsolete.
In some ways this shifts the goal of research in the product world to becomes less about “validating” and more about gaining (or losing) confidence.
Really the only time something becomes validated in product is once it’s managed to shift customer behavior (aka achieve an outcome). Until then we’ve simply gained confidence that we’re heading in the right direction or not.
We have an assumption, we test it and depending on the data we receive back we’ll either have one less assumption or a whole new set of ones. This leads to another hypothesis, another experiment, resulting in more data, giving us either more or less confidence. And repeat.
Really in many ways, product work is like chasing a moving target, you’re constantly iterating, shipping new things, learning new things, not to mention new competitors, market changes, and so forth. We, therefore, settle for imperfection, as perfection would be too late.
