Test your products with real users with a portable user testing lab

1. Introduction

User testing is one of the most effective methodologies in user experience design: it allows us to see how actual users interact with what we are designing. Designers, developers and managers are so accustomed to the product they are working on, that important issues become invisible to them. That’s why user testing is always enlightening and allows team members to look at the product with the eyes of new users and understand what needs to be done in order to radically improve the user experience.

When talking about user testing we find a lot of information about lab and guerrilla testing which are very different and opposite approaches. In the last few years working at Net7, I’ve developed a methodology for testing web products that I call Portable Lab User Testing, a lightweight technique that is easy to implement as guerrilla testing and provides detailed results as lab testing. The testing session is run using a screencast application that records how the user interacts with a real product or a prototype; recorded screencasts are then reviewed to obtain qualitative and quantitative results.

In this article, I will guide you through the steps needed to plan and run a user testing session and I will show you that user testing is an easy and fun methodology to adopt. It allows to literally start a dialogue with real users and thus lay the foundations to improve our products and change the way we approach our work as designers.

2. What you need

Let’s see what hardware and software you’ll need:

• a laptop computer (connected to its charger);
• a standard mouse;
• a standard mouse pad;
• the product or the interactive prototype to test;
• a screencasting application like Screenflow;
• a spreadsheet application like Google Spreadsheet;
• a presentation application like Google Slides.

3. Users recruitment

The most important thing needed for user testing are users. Run your testing session with 5 users: this is a decent number that will allow you to collect enough feedback and end up with a long list of to-dos to improve your product. If you have personas defined for your product, select users that correspond to the personas profiles, otherwise just find the average target users that will use the product.

There are endless ways to recruit users and these depend on the type of product or project: you can start by involving your colleagues, family and friends; you can ask your client to do recruitment for you; if your product is targeted to a well-defined niche of users, go where these persons work or meet and get in touch with them. As a last resource, you can apply the same methodology as guerrilla testing and find users in bars and cafè.

Once you have the list of recruited users, plan a 1-hour meeting with each one. Since this testing technique is “portable”, you can be flexible on the location of the session and even reach users at their home or office and set up what you need in a few minutes.

4. Define tasks

It’s time to define tasks. Tasks are what you’ll ask users to accomplish using the product and will put the features you designed under stress. If you previously defined user stories, it’s a good idea to start from these to define your tasks. Since you want each testing session to last approximately one hour per user, I recommend that you select a maximum of 10/15 tasks (depending on tasks complexity).

Tasks should not block each other: users sometimes fail to complete a task and this might prevent you from submitting the following and dependent tasks.

It’s also very important to define when a task is “completed”: this must be unambiguous and clearly observable like “the user clicked on button A”, “the user reached page B” or “the user completed action C”. This will be crucial when you’ll review the user testing sessions to get some quantitative data.

5. Prototypes setup

When I don’t have a functioning product to test, I do Portable Lab User Testing using Invison prototypes built with wireframes or graphic mockups. This type of testing is very effective and users get the feeling they are using a real product. Insert meaningful text and images into your wireframes since some users pay a lot of attention to contents and will get easily distracted by lorem ipsum and demo contents, losing the focus from the assigned task (and wasting precious time).

Be sure that your prototype is structured to include the user journeys that allow the completion of the assigned tasks and also some “side” user journeys that allow navigating other main section of the product.

6. Guidelines document

Before meeting the users for the actual user testing sessions, prepare a Guidelines document starting from this template. The Guidelines will include:

• Objectives: just a reminder of what you’re doing, the overall scope of the activity and what to observe.
• Plan: a roadmap to follow during each testing session.
• Tasks: insert in this section each task with a brief description.
• Users: the list of recruited users. This is essential to remember their names and call them by the first name during the test, to create an informal and relaxed vibe.

Print a few copies of the Guidelines and take them with you when running the tests.

7. Testing setup

Great! It looks like you have everything you need: users, tasks, product/prototype and guidelines. The day of the user testing has arrived, let’s see how to set up your portable laboratory. Any normal desk is fine to run the test, just make some space to fit your laptop and mouse pad. Place a chair in front of the computer (for the user) and one on the side (for you).

Since users will be testing the product/prototype on your laptop it’s very important to make them feel comfortable. Avoid any distraction: distractions means you’re not testing your product but something else (and wasting time). The computer and the operating system must be invisible to the user. Here is some general advice:

1. provide a standard mouse and a mouse pad. You don’t want the user to test your special nerdy mouse. I personally avoid wireless mouses since I don’t want to run out of batteries one second before the test starts;
2. disable operating system navigation tricks like hot corners (macOS), special events happening when clicking the middle mouse button and other things like that;
3. disable all applications that could popup messages and alerts;
4. disable any other special and non-standard feature of your computer or software installed that could distract the user.

Place the laptop on the desk, the mouse pad on the side and connect the mouse. Check that everything works fine. Connect the laptop to its charger.

Launch Screenflow (or you favorite screencast recording software) and configure the recording:

• video recording of the desktop;
• video recording from your front camera;
• audio recording from the built-in microphone.

This is me setting up Screenflow!

Start a test recording: stop the recording after a few seconds and check if everything was recorded correctly (especially the audio). If everything is ok, you’re ready to start!

8. The session

Start by welcoming and thanking the user for participating in the test. Then let the user sit on the chair in front of the computer while you sit on the chair on the side.

Introduce the product you’re testing by providing some very basic information: don’t provide too much info as we usually want to test how users interact with a product they don’t know. If you’re testing a prototype, make clear that this is not a real product, some contents are just demo and only certain sections are accessible.

Explain that you’re not testing the user’s performance, there are no right or wrong answers and it’s not possible to make mistakes; in fact, you’re testing the product. Be empathetic and let the user relax and enjoy the session. Ask to adopt the think-aloud protocol, to be sincere and not worried to offend anyone. Explain that the test is structured in simple tasks that you’ll ask to accomplish; the session will be recorded and for this reason, ask to sign a simple permission form where the user provides permission to be recorded. (As a template for the permission form you can use the template provided by Steve Krug here).

Launch the browser and open the first page where you want the test to start (typically the homepage of a website or the first page of your Invision prototype). Launch the screen recording (you previously tested that the screen recording is working correctly). Once you’re set, leave the control of the computer and mouse to the user and proceed with the test.

In some cases, it’s useful to ask a first general impression: ask the user to describe what’s in the first page, scrolling it but without clicking anywhere. You don’t need to take notes, everything is recorded and you will review the recording at a later stage.

Then start assigning the tasks: start from the first and give the user enough time to complete each task. Users will ask for confirmation they’re doing the right thing, clicking the right button or browsing the right page: be polite and tell them you can’t help, you don’t want to bias the test. Try to be clear as much as possible when describing the tasks: these must be unambiguous and it must be clear (to you and the user) when the task is completed. Users might get stuck and not be able to complete a task, try to encourage them and give them enough time to try to complete it. Go through all the tasks in the lists in the Guidelines document.

After the last task, ask the user for some final comments, impressions and advice on how to improve the experience of the product. This is usually a very informal part of the test where you’ll get a lot of useful suggestions.

Once finished, thank the user, save the screencast and get ready for the next test.

9. Review and present results

After you’ve completed all the user testing sessions, it’s time to move on to the analysis of the results: this phase is essential to make sense of what has been done so far and get a list of actions and to-dos to improve your product.

Open the Google Spreadsheet User Testing Analysis — Template and start reviewing each screen recording video of the tests.

If you asked for general impressions on the first page, insert the most relevant highlights in the “Intro” column. Then, for each task, select if it was completed (“YES/NO” column) and add any relevant comment that will be useful for the final review in the corresponding “comment” column. Insert the user’s final impressions (you asked after the last task) in the “Outro” column: try to be brief and highlight only those aspects that are useful for the final review. The row “Result %” will provide a quantitative value of the completion rate of each task.

After you reviewed all the screencasts go to the “Recap” row and, for each column, summarize the comments found in the cells above (for tasks columns, take into account also the task completion rate): be concise and try to highlight problems and issues emerged. Expect to have fewer recap-notes if the completion rate was high (100%) and much more notes with lower completion rates.

The move to the “Actions” row and “translate” the recap row content in actions and to-dos as a dotted list: these are the actual activities and improvements you are planning to do in order to provide a better user experience. As for the recap, expect to have few or no to-dos for tasks with high completion rate (100%) and a long list for tasks with lower completion rates.

Once you’ve completed the analysis on the spreadsheet, it’s time to show the results to all the stakeholders: let’s see how to prepare a presentation using my Google Slides template User Testing Report.

Start by inserting the product/project name and date in the first slide. Then for each task, you’ll have to prepare two slides: in the first slide insert the highlights you can copy from the “Recap” row of the corresponding task in the spreadsheet. In the second slide, insert the Actions you can copy from the “Actions” row. In both slides insert the task number, the task description and the percentage of completion (using different colors depending on the completion rate, to provide visual evidence of the result).

In the General highlights slide you will insert the general comments you can pick from the “Recap” row of the Intro and Outro columns of the spreadsheet. In the slide Actions to be taken, you will insert the general comments you can pick from the “Actions” row of the Intro and Outro columns.

After the last slide is completed, your presentation is ready!

If you want to be more convincing and have enough budget, you can also edit a short video with the highlights of the various user testing sessions.

Then plan a meeting to present the results to the stakeholders involved. You can split the meeting into two or more sessions, to present the results to different sectors of the company. User testing results can be enlightening and it’s good to share them to a wide audience (including CEO, marketing team, developers). Present the results during the meeting showing the presentation. You can also start the meeting with the short video to engage your audience.

At the end of the presentation, plan a brief discussion to discuss what emerged. Stakeholders might have doubts and concerns about the results; try to explain clearly the methodology (some participants could be completely new to user testing) and decide together how to exploit these results to improve the product. In fact, it’s time to plan your next steps, decide what to do, assign tasks, plan your work for the following weeks defining a solid roadmap.

10. Time to work!

Do you remember why you decided to adopt user testing? To gather some qualitative data to improve your product. Well, you’ve done a great job, you ran the tests and now you got a list of tasks and things to improve. Now it’s time to go back to design and bring that product’s UX to the next level 🚀!