Three methods to solidify user research and prioritize findings

Developing products or services costs time and resources. Recommendations solely grounded on qualitative insights might not be representative due to small sample sizes. Business owners and stakeholders need data to determine which recommendations help their strategy. This article looks at an often overlooked tool that will improve the robustness of your recommendations; the humble survey.

I previously worked on a series of projects for a big health-tech company based in the US. The healthcare industry tends to lean heavily on evidence-based learnings. Naturally, their request went beyond qualitative data. They asked for hard evidence to prove that the direction we were recommending was the best way forward. We needed to iterate on our approach to ensure our recommendations were objectively defendable.

The problem
The overall purpose of user research is to find opportunities that improve a product. Usually, it begins in the field, talking with users. We uncover problems, behaviors, or needs to generate insights. We use those insights to ideate solutions and define requirements. We then rank and prioritize those requirements to create a roadmap.

Yet, if you look at the underlying reasoning for organizations, it is mainly to reduce risk. Any product development requires investment and a certain level of risk. Stakeholders will need to defend the determined strategy. They need to convince their organization to get a budget, make the investment and manage risk appropriately.

The issue is that most strategies derive from small sample sizes (15 -30 participants). Looking at previous projects, those participant groups even represented multiple roles, target segments, or personas. Separating each group reduces the sample size even further. These low numbers bear the risk that the organization is trying to prevent in the first place.

The solution
Surveys offer a fast and cost-effective tool to quantify uncovered insights. They offer three key benefits:

• Validate if uncovered insights are true or false with robust numbers using a representative sample size.
• Specify the value and priority of results using a representative sample size.
• Qualify the findings with additional insights like user relevance or potential customer satisfaction. Use those qualifiers to build a strategy that crafts a meaningful product or service.

Three survey methods to quantify your insights

Over the past couple of years, I tried various techniques and question formats. The following three methods have proven themselves to be particularly useful. They not only quantify the insights but build on top of them and provide additional nuances to drive product development decisions.

1. The Likert Scale

The Likert Scale is the most versatile method. The goal is to measures peoples’ level of agreement or disagreement with a statement. It can be used to quantify attitude or opinion about a behavior, opportunity, feature, need, or even the execution of a solution.

The survey question is typically composed of five possible responses to a statement. For example: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree.

The Likert Scale is an excellent method to prioritize findings. The scale uses two metrics to qualify the level of agreement or disagreement. These metrics provide additional insight into the strength of respondents’ opinions. You can use those outcomes to inform business strategy.

In the past, I used these two metrics to prioritize for two different intents:

• Kindle excitement: If you intend to launch a new product or service, you might want to focus on the extremes. Especially at launch, you need to create excitement. You want your customers to become fans that spread the word and become your advocates. Focus on the respondents who show ‘highest strong agreement’.
• Scale your audience: If you have an existing product or service and intend to scale to a bigger audience. Focus instead on respondents with the highest overall agreement. The more people agree with your choices, the more customers you can reach.

2. The Probability and Impact Matrix

The probability and impact matrix measures the frequency and impact of a user problem. The question is split into two: How often a particular problem occurs and how significant is the problems’ impact. Each question provides a scale of answers from low to high.

The probability and impact matrix is a great way to understand the relevance of a problem. Ideally, you want to find issues that occur often and have a high impact. However, the other quadrants might be of some relevance to you as well.

Solving problems that occur rarely but have a high impact will increase the user’s confidence in the product. Focusing on issues that arise often but have only a slight impact will help to streamline the process.

You can align each quadrant to a different strategy or goal:

• Quadrant 1 — New Opportunities: Identified opportunities in the 1st quadrant have a high potential to create viable solutions. They could either be a stand-alone product or shift the business in a new direction.
• Quadrant 2 — Increase Confidence: Solutions for problems with a majority of users in the 2nd quadrant act as a safety net. Think about data backups or a parachute in an airplane. You hope you’ll never need them, but it is comforting they are there.
• Quadrant 3 — Streamline the Process: Removing problems in the 3rd quadrant will make the experience smoother. They are like speed-bumps. They won’t cause a crash but will slow users down and build up frustrations over time.
• Quadrant 4 — Irrelevant: Those problems seem to not happen often and don’t bother people too much if they happen. Doing something about those problems might not have a viable business case.

3. The Kano Model

The Kano model, coined by Noriaki Kano, measures the influence of features on customers’ satisfaction. The models’ uses far exceed what I will address in this article. Follow this link for a more in-depth view. For now, we will focus on the feature categorization and the questionnaire methods to determine the category.

The Kano model organizes features into five categories:

• Hygiene Features: Expected features that lead to frustration in their absence, as the absence of hot water in a hotel.
• Performance Features: Features that scale with their sophistication. The higher the proficiency, efficiency, or simplicity, the better they perform on customer satisfaction. Think about the size of your hotel room.
• Delight Features: Unexpected features that will have a positive impact on customer satisfaction. Imagine checking into your hotel and then realizing they have fiber-optic high-speed internet for free.
• Indifferent Features: Features that neither frustrate or delight users.
• Reverse Features: Features that will harm customers’ satisfaction.

Kano established that features degrade over time. A delight feature will become a performance feature over time and a performance feature will become a hygiene feature given some time. Think again about the Wi-Fi in the hotel. Twenty years in the past, having Wi-Fi at all was a delight. Over time we came to expect it, it became a performance feature. The speed and stability determine the customer’s satisfaction. Today it is expected (hygiene). If you can’t get a stable connection, with adequate speed, you will march to the reception and complain. Only the fastest best internet will delight customers once again.

To determine the category, you ask a functional and a dysfunctional question. The participant is first asked how he or she feels about a feature being included in a product or service. Then the question is flipped to the contrary, how would they feel if it was not included. Make sure your participants are aware of the settle difference between these two questions. In the past, it got missed occasionally by some participants, which skewed the results.

Once you have all the answers, you put them into a matrix. Each cell is assigned to a category. If you sum the number of people and cells for one category, you will get an indication of how satisfied your users would be by implementing this feature.

In an ideal situation, there will be a clear majority in one category. Even if this is not the case, you still get an impression of customer satisfaction and where a product or service might be improved. Instead of looking for the highest-scoring category, you look for the highest-scoring features within a category. Your learnings from the qualitative research can help provide insight towards choosing a priority order.

Similar to the other two methods, you can use the outcomes to cater to different goals.

• Cover the Baseline: Sometimes you need to play it safe. For those cases, focus on hygiene features. Execute them as well as possible. This is especially relevant for functional products like an ATM.
• Invest in the Long-term: If you need to build a defendable product (to get budget and stakeholder buy-in) focus on performance features. They scale with your investment, which makes it challenging for others to catch up. Think about the accumulated content of a streaming service, like Spotify.
• Create a Fan Base: If you launch a new product or service, focus on delight features. They will turn customers into fans, which in exchange, will become your best advocates. Think about early adopters of Apple computers. Word of mouth is the most powerful form of marketing.

Keep in mind that successful products/services will require a mix of hygiene, performance, and delight features.

Summary

Qualitative research is great at generating opportunity areas. But in most cases, the sample size of participants is too small to base any decision about which opportunity to pursue. Leadership will need more data to determine which of your recommendations they follow, and how those fit into a product roadmap and a business strategy.

Consider the humble surveys to quantify your findings. It will make your recommendations more robust and minimize the risks moving forward. Qualitative research only provides options; running an additional quantitative analysis can give direction. Surveys are a fast and cost-effective tool to quantify and add depth to your insights. Providing evidence with your insights will enable your stakeholders to make the right decision and manage risks.