Member-only story
What a quantitative user study that nearly failed taught me about metrics
The benefits to collecting metrics while user testing, even with a few users

Our team almost had to consider throwing out a portion of our quantitative user test because some of our users canceled on us.
We needed to look at Time-on-Task to answer one of our research questions. This meant we needed to have enough users to provide statistically significant comparisons.
We were able to scrape together 20 participants, the minimum recommended number for quantitative testing, but two of the participants fell through last minute. That meant we had to scramble to get more.
But during that process, I began to wonder: what if we had only managed to get 18 participants instead? Would we have had to throw out that metric entirely? Why wouldn’t that be useful?
I dug deeper into the subject as a result and learned three crucial things:
- Small samples (<10) are acceptable for time-on-task
- You need to change the way you do calculations with a smaller sample size
- You can make use of small sample size data by changing the way you talk about it to your team