Biases in concept and usability tests

The last time I facilitated a Design Sprint I was the user tests’ moderator for the prototype. While I had moderated many times before, I was surprised by a newbie impulse. I *wanted* the prototype to receive positive feedback, even if I was aware that some of the most critical findings come in the form of an awkward moment, with a frustrated person, not getting how the product works or the value it adds.

The Design Sprint is not meant to be perfect, but rather fast and productive. However, if in later phases we wish to proceed scientifically after getting a good sense of our assumptions, we will need to ask… Did we unconsciously encourage a positive reaction as moderators? More often than not, the same team that worked on a product works afterwards on its validation. How does this influence our user tests as UX professionals? Below are some common testing biases we should be aware of if we want our business to face reality and succeed. These are not limited to a Design Sprint context; you can be mindful of them anytime you’re doing user testing.

Recall bias

People are bad at remembering, so make it easy.

Interviews are probably the most basic research instrument for UXers and a common user testing session introduction— yet they rely entirely on participant’s recall. But cognitive psychologists have known for a long time that memory is, if anything, inaccurate. These are good practices that will help you lessen the effects of this bias:

During the script creation, add questions the participant will be able to answer easily. Four ingredients that make memories retrievable are:

• Recency — Things that happened recently
• Frequency — Things that happened many times
• Emotion — Things that are emotionally charged for the participant
• Association — Things that are not isolated, but rather part of other topics, people and experiences of the participant

During the moderation, talk slowly and don’t be afraid of silence. As suggested by StatisticsHowTo, remembering takes time, so participants will provide more accurate answers if they feel they can stop for a second and recall comfortably.

Expectation bias

Your mind will show you more of what you want to see, so try to bring unpolluted minds into the room.

This is what was happening to me. Luckily I became conscious about it. It happens when observers make partial measures or notes so that they confirm their prior expectations or worse, desires. As noted in Williams’ et al. psychiatry paper, some techniques that can be employed to reduce this bias include:

• Have observers and moderators who are independent of your project team (But preferably who are also UXers)
• If that is not possible, have some observers blinded to specifics about your platform and the things you’re testing so they are not influenced by your hypotheses
• If that is not possible, at least consider having different moderators/observers from your team, blinded in some phases of your project so they won’t be influenced by preconceptions

Membership bias

A popular person is not necessarily a representative test participant, so make a prior effort to recruit properly.

It happens when the group you’re testing differs systematically from the population you’re studying. A common example in our industry goes as follows: Your clients consider as part of the user profile, let’s say, customer support managers. Then they ask their area directors to “recommend” people for “a study”. Directors think of their direct reports and inevitably the most popular ones come to their mind. Probably the most outspoken or the most rational ones. In any case, you not only end up with a group of people who are customer support managers, but also who share other characteristics that might influence your results. These are my suggestions to lessen this distortion:

• Be as involved as possible in the screening process: if time allows, provide a screening questionnaire so you’re able to pick from different candidates and have a group with many points of view. Common things to ask in a screening questionnaire include how often they use certain devices or apps, their age, and — if it’s internal — how long they have been in the company.
• State clearly to your clients the importance of the participants being somewhat random instead of those who are their friends, or those who will be happy to participate.
• Recommend that your clients provide compensation to their participants. This way, their time is fairly spent on giving feedback to you and they won’t be just making a favor to their company. As a result, the sample inside your profile will be diverse: Not only highly volunteering people will be there.

The most prevalent one: Hawthorne Effect

People won’t behave equally if seen, so make it as comfortable as possible for them to be natural.

This is closely related to social desirability bias; this distortion prevails when your participants behave differently because they know they are being observed. They’ll act more analytical, read your platform in more detail, or be a bit more positive about their opinions than using your product alone.

An obvious answer to avoiding this distortion would be to observe your participants without them knowing so. However, this is not an ethical practice; if you’re recording or streaming them, they should know it.

• Reduce as many possible judgmental questions from your script. People don’t want to look silly, so avoid phrasing your questions in terms of capability or easiness. If you want to track ratings about these attributes, consider asking about them in a separate questionnaire, possibly after the thinking-aloud exercise
• As always, state before starting the test that you’re testing the platform and not the participant

Being aware of these biases makes us better researchers but also workshop facilitators, because we talk about our findings conservatively: It allows us to discuss the risks of jumping into conclusions too quickly, especially if you have participants who are skeptical of your methods. Demonstrate you’re aware of the pros and the cons of what you’re doing. That is an ultimate sign of professionalism.