How to choose the best method of usability testing
So, you’re going to start a usability test. There are more than 10 methods, and here’s the series of questions you have to answer to choose the best method for each specific case.
1. Are there any users for testing at all?
The question may sound strange, but there are situations when you have no participants present, so then you can choose Nielsen Norman’s Heuristic evaluation method. It involves independent experts instead of users, who review the interface looking for potential problems. Method isn’t based on user scenarios, but on the set of 10 criteria (heuristics):
- visibility of system status (give feedback and keep users informed)
- match between system and the real world (follow real-world conventions, make information appear in a natural and logical order)
- user control and freedom (support “undo” and “redo”)
- consistency and standarts (the same elements and controls do the same)
- error prevention (prevent problems, confirm the choice before user acts)
- recognition rather than recall (user shouldn’t have to remember information from one part of the dialogue to another)
- flexibility and efficiency of use (tailor frequent actions)
- aesthetic and minimalist design (no irrelevant or rarely needed information)
- help users recognize, diagnose and recover from errors (indicate the problem, suggest a solution)
- help and documentation (should be easy to search and use)
If you do have respondents, the next question is:
2. Can you meet them face to face?
Are they available for a meeting? If not, you should run Remote Testing, which entails tester and user working separately (in different places and/or at different times). Here you need tools to log user’s actions: capture user’s screen, record his voice, broadcast face.
There’s a trend among the new usability labs to prefer remote testing not because it’s difficult to meet participants in person, but the only reason is the tester doesn’t spend his time on it. It’s only needed to make the study plan, put it in the system, bring in some traffic there, and you get the result. In other words, it costs less money and time: the tester doesn’t meet with participants, doesn’t interview them, doesn’t guide the session himself. But if you only measure users’ behavior without talking to them, you don’t know the reasons they act so — that’s the price you pay to make this cheaper.
I’m not saying remote user testing is a “bad” method and shouldn’t be used. Moreover, remote testing is the only way when you need to test people who, for example, live in a different time zone, different countries or speak different languages. But it’s better to use it when you’ve already known your users’ psychology, and have already had face to face testing.
3. Do you explore what users do or what they say?
If you explore users’ behavior, use the task completion evaluation method. Before the session starts and user gets the task, you have to decide what you’ll measure. That is, what indicators you need and if they are meaningful in the context of your product key metrics. So you have to make sure that you can somehow interpret them and project into the plane of business to decide whether it’s good or bad for your product.
However, during the early low-fidelity prototypes user testing you may want to know what do users say and think, and then the following question arises:
4. Can users talk while doing tasks?
If yes, the most common method is Thinking-aloud Protocol. It was invented by psychologists who studied creative thinking and, to understand what’s going on in a person’s mind, they asked him to talk about what he’s thinking. Later this method was integrated into user research, and brought the following problems along with it:
- it’s hard to talk and interact with the interface at the same time. The user’s activity is also affected, it becomes unusual.
When you ask participants to talk during the session, he turns into a tour guide, like “here in this corner we can see a search field, and here is a banner”. In other words, by giving the instruction to think aloud, you push the user to start looking around more. If you’re interested in his natural behavior, or actual task completion time, it’s better if he don’t speak while doing task.
- it’s hard to talk out loud when you’re working stuff out in your head
Tester and participant are locked in their own fight: user’s trying to focus on task, the tester’s forcing him to talk: “Don’t be silent, say something”, and at some point the user may become aggressive.
- unusually high concentration
When we speak out our actions, we select more carefully every next step and there’s less chance to make a mistake. So the user has time to realize the mistake and there’s a high probability he’ll prevent it before you register.
To reduce these problems you could ask to speak only at certain points in time (critical response technique) or, if the task is difficult enough in its own right, the user speaks after every part of task is finished (periodic report technique).
Another method is the Question-asking Protocol, when you ask the user at certain moments what just happened, what he’s thinking about and so on. These moments are not random, they’ve to be made to the user testing protocol.
If the user can’t speak while testing, but you need to understand reasons he acts as he does, then the question is:
5. Can you attract another participant to comment on user’s actions?
For example, you test a contact center system interface and the user’s on the phone during the session, so he can’t talk to you. Then you could invite a second user to the observation room, who will look at the first user’s monitor, hear what he’s saying and explain what is happening to you. This method called Shadowing.
If there’s no second participant, then user can discuss test record with the tester (Retrospective Testing). This method also can be used along with others, such as the task completion evaluation method.
6. Is the facilitator also an expert in the subject area?
If he is, he could run a coaching session: tell the user what’s going on and assess if the user understands or not. The idea here is not to have a user testing session only, but also to explore if there are any difficulties in onboarding at all, and to identify problems with documentation if any.
There are three more “social” methods — Co-discovery Learning (CL), Teaching and Coaching.
CL method: two participants complete the task together, helping each other. It’s important to be sure these users already know each other and have co-working experience. You could discover thus their vocabulary, how they make decisions, argue, etc. It has special relevance in the context of social products where the decision is made by a few persons in conjunction, e.g. products for families.
Teaching method: one user explores the interface and gets familiar with it, then asked to explain how the system works to novice user. BTW through this method you may learn a lot about the subtleties of how users call controls and certain parts of your interfaces.
Coaching method: participants are allowed to ask any system-related questions of an expert (coach) who will answer to the best of his ability. The tester or a separate expert user can serve as the coach, depends on if it’s necessary to observe how the participant interacts with the interface only, or the interaction between the participant and the coach too.
So as you see there’s no one-size-fits all solution, it depends on what time, money and human resources you have, what stage of development it is, what goals you try to reach. This simple questionnaire helps you to choose the right type of user testing (or multiple types) so you make the results far more accurate.
