Remote usability testing: a playbook
The ongoing pandemic has changed a lot of things. The way we live, communicate, and work; all of it has changed.
As designers, we are creatures who crave interactions, stimulations, and a connection with others in our personal and professional lives. Our capability to empathize with our users is greatly enhanced by in-person observation and conversations.
Though the official guideline is to stay six feet apart, the coronavirus has put far more distance between us designers and our beloved users.
Though I have performed my fair share of remote user testing, I have a heavy bias for in-person user testing. But, the circumstances forced my hand and made me look at remote user testing in a completely new light. After several remote user testing sessions, I decided to share my experience with you via this playbook.
The “what” of remote usability testing
Remote usability tests are like traditional usability tests with one key difference: the participant and facilitator are in two different physical locations. Rather than the usability expert going to a participant’s location or vice versa, the participant interacts with the design in his own home, office, or other location, and the expert watches remotely.
Remote usability testing is a method of remote research that uses an insight platform to record the screen (and voice, depending on the software you choose) of test participants as they interact with your product or experience in their natural environment.
The “when” of remote usability testing
As such, remote testing is a great solution for teams with a limited budget, or for testing products whose users are geographically dispersed. Scheduling a series of online studies can be preferable over conducting no user testing at all.
Following are some scenarios where you should consider remote usability testing:
- Conflicting schedules that prevent in-person testing.
- Participants are geographically dispersed, making travel difficult.
- The team has a limited budget and hence little to spare for usability testing.
- Accessibility constraints limit participants to their own environment, such as software or equipment.
- Tight timeframes, making travel coordination, securing facilities, and recruiting users difficult.
These are the most common reasons for remote testing. I guess we can now add a world-wide pandemic to this list 😷
The “types” of remote usability testing
Now that you can recognize what remote usability testing is and when it is needed, let us try to understand what are the various types of remote testing techniques.
There are two main types, each with its advantages and disadvantages.
Unmoderated
Unmoderated remote testing does not require a researcher to attend each test session (no video or phone call); instead, a software application provides instructions to users, records their actions, and may ask them predetermined follow-up questions.
Moderated
Moderated remote testing involves a researcher meeting with a participant via remote screen-sharing software, which allows the researcher to provide instructions, observe the user’s interaction with the design in real-time, and ask follow-up questions specific to that participant’s session.
A quick comparison of remote user testing types
The “how” of remote usability testing
I hope by now you have a fair idea of what you are dealing with when discussing remote usability testing. Armed with this understanding, next you need to execute the same flawlessly.
Based on my experience I have come up with some best practices to conduct remote testing which has helped me get favorable outcomes. Let us dig into those.
Preparing for RT
As user experience designers we aim to please our users with the best of experiences. Hence, you must ensure a smooth experience for RT and that requires you to be prepared.
First and foremost you must define your goals and your target group. Define your target users and decide which specific areas of the product you plan on testing. Make sure to define a clear user goal and create task-based scenarios.
Next, you need to recruit participants. Here, you need to ensure that you recruit users who fit your target group and persona that you defined in the first step. Since tests are remote, you can recruit people from just about anywhere!
There are three primary participant types:
- Participants contacted via social media platforms who may be interested in testing out your product.
- Participants picked from the alpha users group in your organization.
- Participants sponsored by your organization to participate in RT.
It is also important to know how many users are ‘just right’ lest we yield insufficient results with too few or waste time and resource with too many users. Five users should be ‘just right’ to catch the majority of major usability issues in your prototype. Since attendance is always an issue, I strongly suggest scheduling at least three extra volunteers to account for people who do not show up.
It is important to pleasantly invite participants and provide them with the necessary information regarding the RT. For this, I recommend to create a template invite email with coordinated timings (in case of multiple time zones), instructions for devices and builds, and warm welcoming words. This template can be reused for later RT.
I am providing the template that I have been using to successfully invite participants to RT.
This one may seem like overkill to some, but a well done welcome presentation to participants for RT has saved several mishaps for me.
To err is human and err humans will.
Participants may misinterpret, miss prerequisites, or need a refresher for the RT. A welcome presentation helps address all of these. Note that there is no strict format but make sure you have everything mentioned in the mind map diagram in the next section.
Last but not the least, do not forget to make sure that the design PoC or development PoC that you want to test with is ready and available. I have seen some instances where after setting everything up, the test build itself did not become available on time delaying schedules and leading to the duplicated effort.
Conducting RT
Alright! Now, you are all set to conduct remote testing! Let us dive in!
Firstly, you need to follow the golden rule of RT.
Record everything!
You never know when you will have to refer back to a user comment, a user problem, or even a bug.
But, before you do that, follow another golden rule. For unmoderated RT, ask the participants to record themselves while testing and for moderated RT, inform the participants that you will be recording the session.
Next, you should bring out the welcome presentation. Go through the agenda, remind everyone of prerequisites and allow them some time to get set up.
Clearly explain the structure of RT by setting the right context. You should weave a story and make them the protagonist. Give them a role and a task to perform.
After this, provide them access to relevant links or guide them to open the app/website/product.
It helps to test on multiple devices in the case of software products. Remote usability testing is flexible in this regard and allows us to pick participants with different available devices.
Throughout the RT you should take meticulous notes on user behavior, comments, questions, bugs, etc. I recommend you create a framework on how to take notes. Possibly split the page into 4 blocks and note different types of points in different sections.
An RT session should be no longer than ~40 minutes. And even these 40 minutes should be split into sections with breaks in-between. This makes the participant comfortable and ensures little interference from external factors.
Following time break-up has worked best for me:
Wrapping up RT
After RT is concluded you should cordially thank the participants for their time. This may seem like a small thing but it helps keep the participants coming back for future RT :)
Another important thing after RT has concluded is to go through the recording and meaningfully transcribe them. Then, use your notes and transcripts to derive insights.
In case, the design has major feedback, then you should make appropriate changes and go for a re-test.
If you are awesome and everything goes well, you move on to the next step of preparing an RT report. This report should clearly document user comments, frustrations, bugs, and proposed changes. Also, long hanging items should be highlighted. Plus, far-fetched items should be prioritized as needed.
You may present this report to your peers for their benefit and the leadership to get buy-in, hence make sure the report is crisp and to the point. It especially helps to share these findings with fellow engineers as they can connect what they have engineered to what the consumers are expecting.
The “why” of remote usability testing
Acting under the (correct) assumption that any user data is better than no data, many teams are turning to remote usability testing. And for good reason. Apart from the benefits you’ve seen before, the following are some neat benefits of remote usability testing:
- It is generally less expensive than in-person testing
- You can conduct tests on a lot more number of participants when compared to in-person testing.
- With remote usability testing, you can get insights into how your product performs in various geographies.
- Remote usability testing is scalable. Finding new participants online is less time-consuming and cheaper.
- The insights you gain from testing remotely are closer to the real-world as the participants take tests in their natural environments thereby nullifying the Hawthrone effect.
- The large number of participants recruited for unmoderated remote usability testing help in confidently generalizing the findings.
The “drawbacks” of remote usability testing
From the very start, I have been trying to make a case for remote usability testing. Though it is a great fit in our current pandemic situation, it does come with its drawbacks
- Without the presence of a moderator in an unmoderated remote usability test, the participants may simply leave or perform tasks half-heartedly.
- The observer effect is still present during moderated remote tests even though the participants take the tests in their natural environment. In the presence of a moderator, the participants may not speak up even if they find it hard to use the product.
- Moderated usability tests are generally time-consuming. The moderator has to be present at all the time to not miss out on interacting with the participants and the chance to ask to follow up questions.
- The accuracy of the results of an unmoderated remote test largely depends upon the clarity of the scripts as there is no moderator to guide the participants through the testing process.
- After the completion of an unmoderated remote usability test, usually little or no follow up questions can be asked from the participants.
- Some participants take moderated tests just to make quick money and may not provide valuable inputs.
Tools
This is a difficult situation for everyone that conducts usability tests. Fortunately, there are great remote usability testing tools on the market. However, what you want to avoid is getting stuck with an awful remote usability testing tool.
There are several advantages to using such tools
- Several tools natively support recording others with addons.
- The participants can share their video feed plus mobile/desktop screen at the same time.
- Several tools allow note-taking too.
- Most tools encrypt the entire conversation to keep it private.
- Users can add a custom background to their video feed.
- Some tools like Lookback don’t even require the participants to download an application.
Closing notes
Remote usability testing allows you to test more participants quicker for a lower cost. When done correctly, it can be a secret weapon for teams who iteratively test, implement customer feedback, and focus on MVP.
Every user research method has distinct advantages and disadvantages. Understanding the unique qualities and limitations of each approach will ensure that you’re getting the results you need to build better products and experiences for your customers.
I hope this playbook will act as a starting point for you to jumpstart your remote usability testing journey. Go ahead, make this playbook your own, add your experiences, and style to it. Happy RT!
