Rapid usability testing for designers

Usability and research methods for lean design teams

Usability testing can quickly and cheaply show how a design solution could be improved, which saves time and valuable resources. Surprisingly, a lot of Designers skip over this step, which is understandable, but can add design debt to the product development process. It’s amazing to me how many Designers ship their products without the slightest amount of user testing.

Remote/rapid usability testing is an great way to gather quick feedback during the design and prototyping phase. Remote testing requires flat designs, an online testing tool (which I’ll explain later), a research method, and research analysis. Small design teams only need to test with 5 users to uncover the majority of usability problems. Using remote usability tools and flat designs, Designers can gather small amounts of data within minutes, even if you’re a design team of one.

Below, I give a overview of how to use this process to conduct remote tests, including a few examples of remote method tests, a standard number of participants, testing schedule, research methods, and a quick overview of how to analyze results.

Using this method, you can:

  1. Gather quick feedback during wireframing or low-fidelity design phase
  2. Identify user pain points and priorities early on
  3. Undergo continuous iterations on designs before development
  4. Make the user experience frictionless and intuitive
Rapid usability testing method

1. Examples of quantitative and qualitative tests

Remote tests can use both qualitative and quantitative methods. Quantitative methods can review interactions, animations, navigation, task times, completion rates, preference tests, and A/B tests. Qualitative methods can include desirability, visual look and feel, and content and messaging. I’ve previously written about readability of notifications based on the amount of time users prefer to see the banner before it disappears and how this test can be done using remote methods.

Example 1: Click test

I wanted to review the usability of a linked list for a financial wellness mobile app. I set up a quick test during the wireframing phase to see if participants understood the usability of the link in low-fidelity.

Research Question: Can users successfully select a connected list item?

Financial wellness app click test

Test Question: Please select how you would continue to the Life Events Survey item.
Research Method: Click Test
Results: Users were able to successfully complete test tasks. 70% of participants selected the Life Events Survey link in the connected list.
Design Recommendation: While the click test meets the benchmark for success, the qualitative feedback suggests that users may be confused about the required actions for this page.

Example 2: Preference test

Preferences tests can quickly tell if users are familiar with a certain UI pattern or as an indicator for additional usability problems down the product development pipeline. However, preference tests are qualitative and can oftentimes provide inconclusive results depending on the test sample size and population. In other words, they should be taken with a grain of salt.

Research Question: For a profile onboarding screen in a health and wellness app, I wanted to discover if users have a preference for form label placement.
Test Question: Which form design is the easiest to understand?
Research Method: Preference Test

Sample usability test for a financial wellness app

Results: The results were statistically inconclusive, meaning that users didn’t have a strong preference. The majority (45%) chose option 2.

Design Recommendation: Clearly written labels are one of the primary ways to make mobile forms accessible. I recommended option 2 for the final design since the user is able to know the significance of the label at all times during usage; it is visible when the field is either in focus, disabled or completed. This placement is especially important for smaller screen sizes so that the form label can be seen for association between content and label.

Example 3: Microsoft usability test (example)

I’ve frequently tested the overall look and feel of projects using the Microsoft Desirability Toolkit which measures users’ response to the aesthetic qualities of an app by asking them to select product association keywords.

In this test, I asked which product keywords best represent these designs.

Designs for Microsoft Desirability Test (client project)
Desirability results

The results were split amongst a variety of keywords. The majority were positive, but it’s clear that the designs failed to have a unified impact.

2. Number of participants

Usability tests can include as few as 5 participants since this test sample size will uncover 85% of usability problems in the design phase. I’ve had the most success with about 20 participants per study. A follow-up study will uncover the remaining 15% of usability errors.

3. Research frequency

I’ve found that planning user research into my week is the best way to ensure that I set aside time for ongoing UX improvements and smaller tests. Usability testing can be a great way to settle disagreements with product and engineering teams. I always plan usability tests as early as low-fidelity sketching to learn about participants’ preferences.

Each week, I plan on conducting a test based on a UX audit concern (such as onboarding) a navigation pain point or other issue. I also block time for feature development usability testing.

Weekly testing schedule

  • Weekly UX audit(1 times)
  • Weekly feature development usability tests (1–5 times)

4. Usability research resources

An amazing resource for remote testing is MeasuringU’s Quantifying the User Experience: Practical Statistics for User Research, which provides an overview of research methods, why they’re important and how you can analyze results for research of any size.

Coursera also offers some excellent courses on statistical analysis (listed below) which are a great starting point for anyone interested in UX research.

Remote usability tools

For rapid feedback, Optimal Workshop, UsabilityHub, UserTesting, and Lookback (mobile specific) are excellent resources.

5. Research methods

I’ve outlined some simple methods to get started with rapid usability testing. All of these can be done with the resources above.

Question Test
Use this method to gather quantitative and qualitative feedback on designs. These tests can include feedback on which order of content is the most engaging, what participants expect to do on a screen, how participants would improve a design, or other exploratory feedback.

“Please select which words best represent the layout. Select all that apply.”

5-second Test
By showing a participant a design for 5 seconds, you can evaluate the readability and findability of content, UX or design elements.

“Please review the following screen and answer a few questions about the layout and design. Which element on the page did you focus on most?”

Navigation Test
Navigation Tests determine the success of transitioning between screens. You can assign participants test tasks as either a single or a multi-step scenario. Ask follow-up questions to uncover additional feedback.

“How would you go through the steps to complete this survey? Why did you click there?”

Click Test
Click Tests can be used to discover if a user can successfully complete a test task. The Designer may be looking for exploratory feedback on a design or there may be successful criteria to determine success. The outcome includes measuring metrics for success rate, qualitative feedback and task time.

“Please select how you would progress to the next screen view. Why did you click there”

Preference Test (Qualitative)
Preference tests can help Designers determine which design the users prefer. These tests can be used to evaluate the visual design of the app to validate one design alternative versus another.

“These are two examples of points that you’ve earned or have yet to earn. Which one do you find the most motivating, clear and captivating?”

A/B Test
A/B tests are used to benchmark designs and evaluate user preferences between UI elements such as screens, layouts or components. Measuring impact includes statistically significant differences between designs in order to determine success.

“Which of these designs shows that you’ve earned new points?”

6. Measuring results

When writing up results, a short and descriptive analysis is the best method to convey statistical significance and next steps. I like to include design recommendations or follow-up tests if the results are inconclusive.

Sample research “report”

Descriptive title of usability research (ex: Survey Button Placement Preference Test)
Research Method 
Research Assumptions 
Success Metrics (time, completion of test tasks, completion rate etc.) 
Results and Conclusion
Was the test successful?
Design recommendations 

What is benchmarking?

Benchmarking is a great way to measure results against a goal. This method can measure improvement and determine the success of the results. However, when determining a goal for task completion rate, context matters. If there are no previous results to compare, aim for a success rate of 70% to 80% or higher. A benchmark of 70% is considered a strong result for publication and is suitable for product design usability research.

What is a p-value and do I need to know that?

User research often requires a metric of success in order to determine how impactful the results are. A p-value indicates how significant the results are and whether the data indicates that it was due to chance. For example, a p-value of .05 indicates that there’s a 5% chance the difference observed between two designs (such as in a preference test) is due to chance.

Measuring “success”

The context of the usability test matters for how a research report will measure success. While a successful completion rate of 70% or higher is desirable, the context of the test will determine the acceptance criteria. For a test measuring a usability issue that is a blocker for the user — such as signing up or providing sensitive personal information — a higher score is required. In some sensitive cases, a success metric of 100% is a requirement.

For rapid usability testing, the majority of tests should use confidence intervals to determine how confident we are in an assumption of a completion rate. Calculating confidence intervals allows us to know more about the precision of our data and the statistical significance of the results.

  • For small sample sizes, the Adjusted Wald Confidence Interval is the best method to discover information about binary categorical data (pass/fail, convert/didn’t convert) on any sample size.
  • A 90% confidence level is the benchmark that is commonly used for product design usability test.

To recap, rapid usability testing allows Designers to gather insight about the effectiveness of their concepts before development work starts. All the Designer needs are a flat design, a research proposal outlining the research methods, a remote testing tool, and research analysis of the results to determine next steps.

Thank you for reading! Please get in touch if you have questions https://twitter.com/kellydern