Best practices for on-video user testing

How do we get people speak up and share feedback on a product through video meetings?

I work for a video conferencing software company called BlueJeans located in silicon valley. Since the product is a software built for facilitating video communications, I conducted a lot of the user testings through video meetings. Compared to other user testings tools like usability testing websites and on-site user testings, on-video user testings leverage the benefits from both and cost less.

On-video user testings are live.

Comparing to clicking tests, navigation tests, and preference tests on usability testing sites, they reveal more details while users are taking the test. Also, an on-video user testing is much easier to set up — it’s no more than just a meeting link. Traditionally, setting up an on-site user testing takes a lot of efforts. The testing schedule needs to be reviewed and approved by marketing, customer relations, and product teams from both companies. Additionally, it requires commutes, setups, and preps for researchers and designers to get product insights. It’s very energy-consuming.

When I first started hosting on-video user testings, I was unprepared and nervous. I like interacting with real people. I react better when I get the full spectrum of information from a face-to-face conversation — I interpret facial expressions, hand gestures, and semantic pauses in users’ responses.

Inevitably, I asked a lot of wrong questions that were off-script and too improvisational. When users requested irrelevant things, I felt unconfident to bring them back to track seamlessly. When users held strong opinions on some specific features and flows, I couldn’t compose a smart answer to inform them that there were comprehensive decisions involved in the development of a product, not only just design decisions.

As I kept practicing my skills on conducting on-video user testings from week to week, I saw some effective practices emerge. Hence I wanted to capture them and share here.

I used scripts for on-video user testings

1. Prepare, prepare, prepare

I realized how necessary preparation is before conducting a user testing. It could take as short as ten minutes to draft a basic structure of the questions to ask and estimate roughly how much time it may take for each section of the test. Building a structure in my head helped me get out of the blank-out moments while remaining a clear thread to continue the conversation. I also realized that a productive discussion is to build upon the information users just provided. Listen, find the critical piece of information, and weave that information into the next question. This method helped me collect a lot of in-depth product feedback, specifically for returning testers who have already participated in product quality surveys and customer interviews.

Prepare and have a structure before testing

Also, having a five-minute rehearsal with my work colleague who assisted me in the testing was very helpful. Outsourcing notes-taking to my peer helped me focus on the questions I wanted to ask. As my peer already understood the structure of the test, the information we collected was filled up quickly under each testing section.

Filling up scripted questions with answers

2. Join the meeting five minutes ahead

I learned this from a sales representative colleague of mine. When he hosts sales video meetings, he hangs out in his meeting room ten minutes ahead. Most people tend to join meeting on time or a bit late, but there is also a group of people who show up ahead of time. If a user entered a meeting and you were already there to say “Hi” — it warmed up the entire conversation immediately. I copied what he taught me, and it worked quite well! Later I realized the magic behind this tip. No matter it is an on-site or an on-video user testing, users want to be seen and welcomed as “people” than “testers”. The more a user testing is close to an everyday conversation we have with our friends and colleagues, the more our users will be engaged in giving us feedback — we all want to be helpful.

Welcome your user as soon as he or she joins your test

3. Introduce yourself to break the ice

As soon as a user joins my testing meeting, I will introduce myself, my colleague, who will be assisting me, and give a brief of what the test will be. It is a quick way to build some credibility before asking for anything. Also, keeping my users informed of what’s next will square way confusions that first-time testers may have.

Introduce yourself and share the brief of the test

4. Focus on listening

Sometimes I have to conduct those user testing meetings by myself. Instead of typing on a keyboard while asking questions, I record the current session so I can focus on listening to what my users say. Listening and understanding the pain points that my users have are more valuable than documenting from word to word. It takes me extra time to rewatch the video recordings and fill up missing spots, but it is relatively easy when I already have the entire picture.

Record your test and focus on listening

5. A quick recap of the test

I keep a Google Doc to track the documentation of my user testings. After the on-video testing, I go back to the document and reorganize the notes. I use bullet points for the takeaways - in case I need to check them later. Also, some of the feedback could be passed on to other teams — product management, engineering and support for feature prioritization, bug fixes and support articles.

A bug caught in my user interview got fixed! 🎉

Closing thoughts

When I first landed on conducting on-video user testings, I had moments that I didn’t know what to do. I looked up for tips on Medium and other sites, but it still took a lot of practices for me to get a real sense of it. Since video meetings become more ubiquitous as a daily communication tool in workplaces, I can see more user testings conducted through video meetings. I’m hoping this article could offer some insights for product designers and UX researchers.

Here’s an easter egg if you read this article to the end. I created a template for anyone who’s starting to host on-video user testings.

Download the template and modify it for your own needs. Leave me a comment if you’re interested in this topic and have thoughts to share!

Free on-video user testing template!

I also found some other great reads if you want to know more about conducting remote user testings:
How to Do User Testing on a Budget

User Testing in Data Visualization

4 Customer Interview Questions You Should Never Ask