User testing

100 hours later: It’s okay to make mistakes with user testing

As a researcher running unmoderated user tests, you will eventually make mistakes. Here’s what to look out for.

In a short period of time, I conducted around 50 unmoderated user tests and watched well over 100 hours of video. Most videos were around 10 minutes long, and most tests had 10 videos. Some much longer, some much shorter. I watched the videos at work, and sometimes I was intrigued enough to even watch them at home. User testing is endlessly fascinating to me.

That said, I didn’t do everything right. Some tests went terribly and I had to repeat them at least once. Others went very well. I learned so much about how to test, what to test, what questions work in an unmoderated setting, and what really doesn’t.

Here are some takeaways that I hope can help you if you’re thinking about using a service like UserTesting.com or UsabilityHub.com.

1. User panels and recruits

Aside from asking the right things, asking the right people the right things is the most important part of unmoderated user testing. Testing works when the people who will use the product are the ones testing it. People who are unlikely to use the product can provide insight, but the insight is less relevant.

You can use panel screeners or qualifying questions to screen people in.

Set the right context

With companies like UserTesting.com, you will usually need some sort of context-setting statement when people begin your test. Provide helpful context to get people into the right mindset, but avoid statements that don’t apply to the user.

For example, don’t ask people to pretend they are someone they are not (e.g., “Pretend you are a doctor looking at patient’s medical history”). If you need doctors, specifically, then find real doctors to test your product.

Similarly, don’t ask people something like, “Imagine that you are shopping for your daughter,” if you haven’t adequately screened in people with daughters.