Doing remote user research the right way
Framework to help you plan and prepare for user research during a pandemic.
User research is a daunting task. It requires rigorous planning, sharp intent and a fair amount of effort from designers, as well as organizations. Adding to that, Pandemic has made it an even steeper uphill battle.
Lately, you might have found yourself dealing with a double-edged sword. On one hand, you’re trying to convince organizations the value in user research. On the other, you’re handling the challenges aroused by the remote nature of it. Be it reaching respondents, having a clear plan, prepare for uncertainties or the constant scheduling & rescheduling.
It’s not hard to gauge that there has been an apparent shift in peoples’ perceptions, priorities, choices and biases in the past year. Keeping that in mind, conducting user research is more valuable and critical than ever while building any product and service. And it should be done right. Our research framework should be adopted and evolved to be flexible enough to deal with uncertainties which tag along with working remotely.
The Challenge
Covid-19 has disrupted our lives at an unprecedented level. In the past year, we have seen changes in how we live, work and engage within communities.
For Researchers involved in participatory design and ethnographic research, restrictive measures of the pandemic has brought on new set of challenges in their work and collaboration.
Users can not be approached for physical interviews, due to their rational fear of the pandemic. And Remote Interviews have challenges of their own. Trying to convince people to spend additional hours of their day staring at a screen, taking an additional video meeting post their workday — It’s not the most welcoming idea (Read: Zoom Fatigue).
Even post confirmation, you have to take into account the work involved in walking people through the guard rails of technology, scheduling, research methodology and activities. So it’s imperative that the research is planned and executed in a thoughtful manner, with an empathetic approach towards the communities/people you’re interviewing.
Here are some good practices going forward.
Start by formulating Intent
The first step of your framework should be defining what you want to get out your interviews. At this point, you need to define and document themes or topics you would like to uncover during the course of your interviews.
Next step is to formulate an Intent Statement, which will create a common ground for your discovery process. Your Intent Statement should have a
- Sharp Problem Definition
- An aptitude you would bringing to the solution space.
This would lead to a sharp purpose and clear goals, and would help put together meaningful stakeholders.
The intent statement for our recent behavior study around productivity looked like this:
Reimagining the notion of “work”, to create new rituals, and a system for better ways of working, and a cultural framework that is built to adapt across people, organisations, workspaces, growth horizons and externalities.
We wanted to understand the behavior change around convenience & psychological safety in lives of people working remotely.
Generate and Examine your Sample
You need to understand that reaching your targeted sample for the interview is going to be harder than earlier. So, there might be merit in keeping the sample slightly larger.
Post confirmation of your sample, you need to examine if your sample corresponds to any of these personas:
- People who have relatively free time during the pandemic.
- People who have a need to have their opinion heard, but due to lack of outlets/environmental constraints — they have found this interview to be a voice out for their opinions/unique takes.
- Personal/Professional Acquaintances.
There is nothing wrong with the personas mentioned above. But examining the sample, you might understand who you might be missing out.
It might be the case that you were not able to get positive responses from a certain section of your desired sample. Leaving them out might skew your data, so you need to think of solutions/methodologies which can be made use of, to involve them in your discovery process.
Know who you are talking to
Nothing throws off a participant than a question which doesn’t apply to them. Specifically, when they are taking time off their busy schedule for this conversation.
Make sure you have done a background run regarding a few key points about your participant before starting the interview. Have a clear idea about their pronouns, living situation, educational background, current role in their organization etc. It doesn’t have to be exhaustive. However, it needs to be expansive enough to prevent you from making any unfortunate mistakes and setting the interview in bad taste.
Do A Pre-Interview
Giving participants a phone call, a day before the interview not only helps as a reminder but also sets up the context for the following day.
In any interview, the initial time is often spent on setting up the theme and context — walking them through the process, cluster topics, consents for the usage of research data etc. Pre-interview gives you the opportunity to engage prior to the interview, overcoming the initial resistance/hesitation (if any), and capture trivial information. This gives you ample time during the interview to jump directly into sections of relevance. Use this call to:
- Introduce yourself.
- Provide a summary of the agenda and goals of the interview.
- Extract information about your respondent’s environment, location and their technology setup for the interview. This will leave less room for surprises.
- Get confirmation of their familiarity with the video-conferencing tool. Ideally, the tool should be easy to use, with an option for the participant to login via a link without setting up an account. It should also have support for low network signals.
- Get confirmation on the interview timing.
Post this call, forward them a few sections of the interview questions, possible interview length, etc. Create a calendar invite for the finalized time.
Prepare for the Interview
Make a checklist to stay prepared. Here are some no-brainer tips to have an effective interview.
- Make sure that your devices are charged.
- Keep a secondary source of internet, in case of network drops. You can also use your mobile data as a backup source.
- Find a quiet and secluded place for doing the interview. Check your mic and confirm if you’re getting a clear recording.
- Have a notebook and pen around. Since you’re doing the call over a digital platform, using another tool like a phone/iPad to take notes might seem disruptive to your respondent.
- Keep your questionnaire accessible to you.
- Don’t plan on multi-task during the interview. You should reduce the no. of distractions while on the interview. So keep your phone on mute, with notifications off, and ideally away.
Have a Flow
Your interview should have a clear flow, growing slowly and smoothly in a guided direction.
Participants are more likely to be involved and are comfortable in trusting you with their information if a level of personal connection is established. Start by building a rapport with them.
Ask open-ended and easy-going questions at first, which draws from the experience of your participant. It is also a good practice not to go into yes and no questions early into the interview.
Give the conversation space to breath and expand. Build on it with more complex questions and scenarios which you might want to address and then move it in the desired direction.
Make sure your questions are simple, clear and relevant. The flow for the questionnaire could be around Experience, Environment, Opinion, Patterns, Constraints and then onto specifics. However, this is highly adaptable.
Take Extensive Notes
It’s not easy to resist the temptation of using annotation and recording features of the video-conferencing tools — but that is not an excuse to not take notes. Keep a notebook handy, and take extensive notes throughout the interview.
These notes will also help you keep a readily available inventory of your thoughts, questions, and ideas which might arise during the interview. And these tend to be super helpful during the synthesis process.
Keep it Engaging
Remote interviews are hard to manage. You have no control over the user’s environment, and the commonly used video conferencing tools don’t have an interactive interface to keep the participant engaged for a prolonged duration.
In order to prevent participants from losing interest in the conversation, the interview can be planned with some tasks which have a creative element to it. People are more likely to engage with activities which provides them with a creative agency.
You can also make use of the features available by the conferencing tool. Use screen-share to showcase
- Reference/Examples (Could be textual or audio-visual)
- Questions/Intent/Problem Statements
These will help to ensure that the interview doesn’t get monotonous, and keep participant in the context of the interview with quick recall. Although, one needs to be careful that these do not become overwhelming or distracting.
Avoid Assumptions/Leading Questions
Everyone has been stuck under their bubble for the past year, living within the constraints of their home. That means your window to the world has been through the lens of your screens and your feed. These might add up to the biases you already have, which can easily creep into the interview process.
To mitigate the same, you will have to make a conscious effort to keep them in check.
- Test your questionnaire, pre and post each interview with your team members and get feedback.
- Listen to your recorded conversations as you traverse through the interviews. If you feel that your assumptions might have altered or directed the conversation, Examine and Revise your approach accordingly.
While we were doing our behavior study — our base assumption was that mental health has taken a negative hit for a large portion of the public during the pandemic. Our questionnaire was designed and formatted accordingly. Although talking with a few respondents, we quickly realized that it might not be the case.
During our research, Some respondents felt that pandemic has given a positive direction to their mental health, thanks to support from their families, and organizations. The pandemic has been able to bring them together stronger and help them build meaningful relationships.
This helped us to understand the importance of framing. We took a more neutral approach moving forward, and we could see more diverse & human responses post same.
Take Answers with a Grain of Salt
‘Listening to your users’ might be sincere advice, but it doesn’t translate to ‘trusting every word your users says’. You have to take answers with a good deal of skepticism. As it is not just your biases you need to mitigate during an interview, but your participants’ as well.
If you feel that a few of your questions are going to put your participant in an intimate spot, where a possible answer might pull them away from the crowd. There might be a value in making these questions slightly indirect.
As we know, every human wants to be liked and has a subconscious drive to be part of the group. That might lead them to choose a safer alternative when faced with such questions.
Shifting the pressure away from the participant while dealing with topics/questions regarding abilities, personality, sexual behavior etc. might lead to better results. A research study by Robert J. Fisher says that indirect questioning reduces social desirability bias on variables subject to social influences and has no significant effect on socially neutral variables.
So, Instead of asking our participants of behavior study:
In the past couple of months, Have you been able to finish up your tasks on time?
We used questions like
Do you think the pandemic has affected people’s ability to stick to their deadlines, positively or negatively?
Be Neutral
You have to be careful about how you are responding/conversing during the interview. Don’t be too excited, or react very positively or negatively in what you say or how you act. Watching your body language, and keeping a neutral expression would make sure your mannerism is not affecting or directing the respondent’s answers.
Break it Down
While planning the interview and building the questionnaire — make sure that you have scheduled time allotment for breaks. Generally, these interviews tend to go for long hours. Letting them run without breaks in between might deteriorate the quality of the conversation (eye strain, engagement gap, Zoom Fatigue etc.)
Try to take a 5-minute break for every 40–60 minutes you are spending on the interview. Use this breaks to relax and stretch. Peruse through your notes, synthesize, and examine if you have missed anything or would like to discuss/followup regarding a topic of the conversation.
Also, let your participant know that it’s okay to take a break in the midst if they would like. These could be for making an important/emergency phone call, check up on their kids/parents/food delivery, bio breaks, water reminders etc.
If you feel like the interview is taking longer than the estimated time, please address it. Confirm with the respondent if it’s okay to continue. Otherwise, reschedule for a mutually convenient time.
Closing the Interview
Once you are done with your questions, ask them if there is anything more they would like to add. When both of you have reached the natural end of the conversation, thank them for their time. Let them know about what you would be doing with this research data, and get consent on the usage of the information shared during the interview.
Synthesis
Once these interviews are completed, that is not the end of your research. The observations, findings and notes from the interview need to be organized and synthesized. This process is best done on the day of the interview, while your experiences and perceptions are still afresh.
Put your Research Out. The research process should be a highly collaborative effort. It doesn’t just need to sit on your mind, desk or desktop. Discuss with your team the stories which you found interesting. Let them observe, comment and shuffle through your data to uncover new insights.
Organize. At this point, you can start looking for larger themes encompassing your data. Ideally, the affinity mapping should be done with post-it notes, on a big wall around a bunch of members from your team. But remotely, you can make use of digital tools like MURAL to do your mapping.
This process can be messy. Try to align yourself with the digital tools to make this process smoother. While mapping themes, you should be focusing more on why someone said something rather than what or how.
Some potential ways to group these findings are:
Equivalence. Similar findings across different demographics, users or platforms.
Gaps. Gaps in your findings or insights which can be due to lack of substantial amount of backing information or time spent in dissecting an insight.
Hierarchy. Larger thematic trends which encompass and support several other findings.
To make sure that you’re aligned with your research goals, keep your intent, research guide and project goals in front of you so that your synthesis process is happening with the right background.
Marion Baylé has gotten into the topic in deep detail in her video and post if you’d like to read further about making sense of your research.
Final Thoughts
Since there’s a shift towards remote work, with the looming dread of the pandemic. New Frameworks will have to be adopted and evolved constantly by using mediated forms which can provide equivalent results.
Experimentation with various research methodologies like Epistolary (Asynchronous) Interviews, Mobile Methods, Facebook Groups for Data Gathering etc. are going to be helpful when social interactions are not appropriate or possible.
I hope you found this post useful. How was your experience with remote research and interviews? Do let me know in the comments.
You can check out more of my work at Behance , Dribble or Instagram.
Credits:
Photographs used in illustrations are from Unsplash by MagnetMe, Visuals, Yogendra Singh, Harry Pappas, Ansley Ventura, Vicky Hladynets, Bruce Mars, Bruno Emmanuelle, Mimi Thian, Trust and Tran Mau Tri Tam. Thanks Everyone.
References:
- Just Enough Research by Erika Hall
- A New Make Mantra — A statement of Design Intent
- Cognitive Bias during user research
- User Research Bias: How it Hurts Your App & What you can Do about It
- Avoiding Bias in User Interviews
- Synthesis: How to make sense of your design
- Affinity Diagrams: Learning How to Cluster and Bundle Ideas and Facts
