The ultimate user testing guide

Learnings from 10 years of product interviews for research.

Published in

UX Collective

9 min readDec 14, 2019

Good product design requires frequent user tests and interviews to build the right products quickly. But if we want to do them right, it’s just not enough to do a questionnaire anymore — we want to take a deep look inside their brain and understand their honest opinion and behaviour.

Below are a few of the techniques I’ve learned and apply to improve the quality and results of my interviews. They help me getting honest feedback, shaping the product and navigating strategically.

Feedback based vs. Outcome based research

Many teams still rely on pure user feedback when iterating on their product. They refer to comments from the community to make decisions about new features and end up losing focus. It seems easy to just drop a message “What is the #1 feature you would like to see in the new app?” or read through angry messages that someone would love to have more blades on their razor (in case of a physical product).

Listening to users is important, yet there’s a problem with asking them for what they want — they don’t see the entire picture of the product cycle and don’t express what they really need. Instead, they only say what they think would be a good idea. Adding a razor blade might make a cleaner shave, but it might also add to the weight and size, thus making it less handy or portable.

In the book “What Customers Want” Anthony Ulwick suggests an outcome-driven approach — first, you want to understand the job the customer needs to do. And second, how the outcomes of your product will improve that job. If a customer says they want more blades on the razor (their direct feedback), they might actually mean that they want to cut themselves less (outcome of a better product). Approaching the design from starting with the outcome will help you find out what people actually need. Then you can come up with specific features with much more space for creative solutions.

Ulwick then suggests asking customers for metrics, which define the success of the above mentioned job. These are statements you can write down directly as clear value propositions. So if someone asks for more razor blades — instead of producing a razor with 17 razor blades and saying “now with more razor blades”, you want to find out whether “a better razor could help me cut myself less”. This can be written down in a simple format:

minimise the likelihood of cuts
increase the speed of a clean shave
maximise the shaving area

You can then prioritise them by importance vs. market saturation, quickly come up with creative solutions and also market them better “cut your face less with our 17-blade system”. The book obviously goes into more detail.

Find the right users

Before we get to the interviews we actually need to figure out who to test with and how to find the people we want to test our product with. This is sometimes a challenge: The product may be directed at different target user segments, someone from our team might disagree with a segment or maybe we don’t even know it yet and feel unsure who the product will resonate with the most.

Here’s my approach to make sure it’s clear:

1. Understand the goal/outcome of your product and agree on it internally

The first step actually starts way before even thinking of interviews or user tests. While working on the product direction and sketching how it should look, I make sure the team agrees on a version to test. Once we have a prototype, it will be much easier to assume a target audience. If you’er running Design Sprints, you can usually start thinking about testers before drawing the story board. I would reduce the effort creating personas or doing user research before you actually have a concept or product. It has, at least in my experience, proven mostly unnecessary in hindsight.

2A. Define the target audience and agree on it internally

Once I know what the product could look like, I agree with everyone involved on one user segment. Not more, because otherwise we’ll mix different opinions and waste too much time interpreting and sorting the feedback. If you have never tested your product before, I recommend choosing a niche audience, i.e. the smallest subset of your ideal overall user base. It will be the people who you think will appreciate most of the product features. After doing 5 qualitative user tests (interviews) with one audience, you’ll learn a lot. You might also learn about who else you could test with.

2B. Alternative: Take a guess and test your hypothesis

Sometimes we don’t know or can’t decide who to test with, because we might not know who will be the right audience to be interested in the product. In this case I suggest just picking a homogenous group of people you think could give you valuable feedback and trying it out. If you happen to pick the wrong audience, the interviews will immediately tell. You can then either repeat them with a different target audience or re-design the product to try again. Here it’s important to keep the iteration loops as short as possible, otherwise you start overthinking and waste a lot of time.

3. Go online and find testers in online communities

Once I have a prototype and decided on who to look for, I usually go on Reddit and post in one of the following subreddits:
r/usertesting
r/WorkOnline
r/Jobs4Bitcoins
r/PaidStudies/
They are always open to people asking for testers as that drives their community, especially if you incentivise them with money or Amazon vouchers. But you can also post on online forums, Discord groups, Instagram etc. or even ask within your existing community.

To screen them beforehand I write a “secret knowledge question”, to which the answer only truthful responders would know. When we were looking for testers of a day trading app, we needed people with day trading experience. So instead of asking “Do you day trade?” I asked how much they have traded and how they spread their portfolio. The more believable their answer, the more likely I was to schedule an interview with them.

Open and truthful interviews

About 10 years ago, when I interviewed people on trains and busses for the improvement of the public transportation system, I learned to be honest and personal with my intent. I don’t think that it’s a good approach to ask people question after question for the sake of filling out a questionnaire. This limits you to only your set of questions and discounts all the other, potentially interesting stories they might be able to tell you.

I got the most useful feedback after introducing myself with “Hey I’m Andrej, I work for … and I’m supposed to ask you a few questions because they want to make transportation easier for you. Would you mind helping me with that?”. With interviews for digital products I obviously don’t ask if they mind helping me, but instead thank them for the help.

At the start of the interview I immediately try to establish rapport, so they understand me and I understand them — and we speak the same language. I align my speed of speech, pitch and vocabulary to theirs. I always make sure to come in with positive energy, look nice and care about what they say. I tell them something personal, ask them something personal and do what I can to get them into a relaxed and talkative place. If you enjoy the interview, they will very likely enjoy it, too.

Some easy hacks to make interviews better

Most of the suggestions from above are skills that need practice. Here are a few things you can do with no practice:

Have a self-explanatory prototype

Start by having a good prototype and show them what you have without explaining. The more you need to explain the less you will understand what they pick up on on their own and what’s relevant.

Prepare the right questions

If you don’t know what to talk to your testers about, here is a simple way to come up with a few good questions:
Figure out what your goal is with the product (in the ideal world) and come up with as many reasons as possible why the product could fail to achieve that goal.
Be honest and ask yourself things like “Will people actually understand it?” or “Will they see value in it?” etc., depending on the goal. During the interview, don’t ask these questions directly, rather try to understand their mental model so you can answer the questions.

“Don’t worry about saying something negative”

Saying this in the beginning of the interview will help interviewees feel comfortable at saying what they think without trying to phrase it nicely. You want the truth, you will get it.

“This is not a UX test”

I’ve stumbled on some interviewees who have already done user tests and for some reason they think they are supposed to give feedback on UX design (i.e. “this CTA should be higher”). I try to defuse that mindset by letting them know that this is not about UX, but rather about their personal opinion.

Awkward Pauses

Once you have established rapport, deliberately not saying something will make them feel slightly awkward and they will try to fill the silence by talking more. That gives you extra stories.

“Where could you imagine this product in your life”

This gets them thinking about how your product could enrich their life. They might not have a definitive answer but might give you some ideas for improvement.

“What are you currently doing to achieve [solution of your product]?”

Understanding the current tools and habits of your users can give you a lot of insight into the current world they live in. I normally start an interview asking them about their current solutions — i.e. “how would you currently find a suitable insurance?” in case of an insurance-related product.

“Name three things you would like to add”

Please don’t use this as a suggestion for new features! The aim of this question is making them dream up what could be better about the experience in general and might show you which features you don’t need.

“How would you describe it to someone else?”

This is a classic, but lets them recapitulate what they have seen and express it in their own words. They will mainly speak about the parts that were relevant to them and not mention what they didn’t find interesting.

If they say something like “my friend would be interested / like this”, don’t pay attention. They are not their friend and you are not interviewing their friend, you are talking to them so only their experience is relevant. If you are wondering whether they would buy a product and how much they would pay for it, don’t ask. Sell right away, only putting money behind it can assure you that people actually value your product.

Final Thoughts

Running good interviews is very important for improving your product. Google designers recommend to user test as early and often as possible. And even if it’s not super easy it’s also not terribly hard.

In a classic screenwriting book Jean C. Carrière recommends telling stories in your everyday life to improve your storytelling skills for films. I think the same goes for interviews. I like asking people for things they know and make them feel good for telling interesting things from their life, especially those who don’t talk much in general.