80/20 remote user research —even quarantine can’t stop the data

Photo by Callum Shaw on Unsplash

For anyone who had in person user research planned, being in quarantine has definitely thrown a wrench in your plans. But never fear, there are some great remote research options out there, one of which I’m going to outline in this article. It’s called Pareto Principle (80/20 Rule) Based User Research. It’s actually my favorite method in general, even when I’m not locked away!

I conducted this research several times at my previous startup, and got some pretty stellar results each time. Best part? It’s quick, and potentially free!

I wrote a more in depth article that includes the history of Pareto Based Research if you want to know more. This is the abridged version to get you ramped up quickly.

The Process

Step 1: Select a Random Sample of Customers Who Have Opted In To Receive Emails

Do you have a list of your users? Email a random chunk of them who have opted in to receive emails (aka make sure you’re being GDPR compliant) and ask if they’d be interested in joining a “special community of customers who will have the opportunity to impact future changes to the product.” You don’t necessarily have to pay people—just being able to leave their fingerprint on the product is often more than enough motivation to get them involved in the research process.

Step 2: Create your survey

Making a survey is inexpensive using a tool like Survey Anyplace — or possibly even free — with Google Forms.

You’re going to ask exactly two questions in your survey:

1. “If you could change one aspect of our product, what would you change?” (Provide a list of all core product areas and allow only one selection. Do not include an “other” option.)
2. “How would you change it, and why would you make that change?” (Make this question open-ended.)

Step 3: Launch your survey

To launch your survey, you can use a plain old email list, or you can get a subscription to a service like MailChimp. If you’re sending to an email list BCC, just keep an eye on your recipient list. You don’t want your email address to get flagged as spam for all of eternity.

I prefer MailChimp, and here’s why:

• The intuitive dashboard.
• It’s great for tracking open rates, click rates, and other stats.
• You can easily create lists and groups.
• It provides simple campaign templates.
• Unsubscribe/spam rules are handled for you.

Step 4: Analyze your data

After you launch your survey campaign, you’ll be flooded with responses and data. Don’t get overwhelmed—analyzing the data using this method isn’t that intense.

You’ll start by calculating the total responses per core product area.

Part 1: Calculate Totals

• Export your survey data to a spreadsheet.
• Sort it by core product area (the ones in your multiple choice question).
• Calculate totals for how many responses came from each core product area.
• Order them from most to least.

In this case study, my results looked like this (out of 40 functional areas):

Number of responses

Headlines: 26

Editor: 21

Files and Folders : 21

Forms and Surveys: 21

Groups: 17

Calendar: 12

Reports: 8

Remaining core functional area aggregate responses: 26

Part 2: Calculate Product Area Percentages

Now, let’s do some quick math.

Let’s say I had 152 respondents total. Take the total responses for each key area and divide it by the total number of respondents to get the percentage of respondents who identified each key area.

In this case study, my results looked like this:

Headlines: 26/152 = 17%

Editor: 21/152 = 14%

Files and Folders: 21/152 = 14%

Forms and Surveys: 21/152 = 14%

Groups: 17/152 = 11%

Calendar: 12/152 = 8%

Reports: 8/152 = 5%

Remaining 33 areas total: 26/152 = 17%

Using this information I was able to gather some very useful data:

83% of the 152 responses fell into 7 key functional areas, while 17% fell into other functional areas.

The 7 key areas identified make up 18% of the 40 key functional areas. (7/40 = 18% rounded)

So 18% of our key functional areas were causing 83% of our clients’ frustrations. My results wound up being bizarrely close to 80/20! The areas the research identified were shocking; we were expecting completely different results. We made changes in our product based on these results, and then ran the same study the following year. The areas we’d adjusted were knocked out of the list of issues noted by users, and we wound up with another set of data with new areas identified that aligned with the 80/20 rule. We fixed those issues and replicated the study a third year in a row, and once again wound up with similar results. Our team (myself included) was pretty astounded by the consistency of the results year after year.

Step 5: Create a report

Now the fun part: Weaving the data into a simple, skim-able report for stakeholders.

I started the report with a two-sentence explanation that this study revealed that 83% of reported issues were stemming from 18% of our core product areas.

Next I gave them the high-level stats for the seven areas of concern:

Headlines: 17%

Editor: 14%

Files and Folders: 14%

Forms and Surveys: 14%

Groups: 11%

Calendar: 8%

Reports: 5%

Finally, I grouped the detailed user feedback by functional area. For example, under a heading of “Headlines” I provided a bulleted list of all of the detailed feedback customers gave in response to the open-ended feedback question.

It made a neat package that my stakeholders loved. At a glance they could see the big picture, but if they wanted to dive deeper into individual pieces of feedback, they could.

These research findings made a big impact on decisions that guided our annual product roadmap planning. We were able to identify usability issues, areas that needed UX love, and even product gaps based on the research findings.

Final Step: Dig Deeper

Now that your stakeholders are on board, you can move on to the last step: following up with respondents in Zoom to gather additional info and make sure your roadmap is solving the right problems.

Why give this research style a shot?

It’s a trifecta. You obtain a clear view of really powerful data, your clients feel that they are being heard, and the method is simple, cheap, and effective.

If you’re scrambling around trying to figure out how to quickly pivot your user research plans for the coming quarters into a remote friendly format, give Pareto Based Research a shot. I think you’ll be pleasantly surprised—the results really pack a punch.