What’s the worst that could happen? Reducing risk through user research
What’s the best that could happen? What’s the worst that could happen? What’s it going to take to reduce the risk?
If you are about to launch or redesign a product, service or experience ask yourself these three questions early and often. Chances are you will have many decisions to make along the way and will have to make them based on incomplete information. Asking yourself these questions each time you come up against a difficult decision can help you decide how to reduce the risk around that decision. If the chance of a bad outcome is too much to bear, user research is a great way to reduce risk on any project.
User research is the act of gathering data on users’ needs, desires, motivations and behaviors. How much and when to conduct user research is an often debated topic in the user experience design community. There are many types of user research and no single approach for how and when to use them.
Let’s use the following scenario to illustrate how you might use these questions to help figure out how, when, if and how much user research to do.
You have a test coming up on a topic you are not very familiar with. Passing the test is very important to your career. In order to be promoted in your company, you must acquire the certification that comes with passing this test.
- “What’s the worst that could happen?” The worst that could happen is you fail the test and miss an opportunity to be promoted.
- “What’s the best that could happen?” The best that could happen is you pass, get the certification and get promoted.
- “What’s it going to take to reduce the risk?” You’ll have to invest time in studying. You aren’t familiar with the content so the effort required to achieve the reward is high.
In this scenario, being promoted is very important and the effort required to achieve that outcome is high. You want to reduce the risk of a bad outcome and increase the chances of reward, so you decide to make the effort required to improve your chances.
We make decisions like this all the time to reduce risk. We may watch a movie trailer before buying a movie on demand or look before changing lanes on a highway. In many cases, we reduce risk without even thinking about it.
But this kind of risk reduction is not always considered in business settings when looking to redesign or launch a new product, service or experience. Why not?
I’ve been invited into many projects where teams want to skip any form of user research and jump straight to generating and building ideas. It’s usually for one of the following reasons:
- They are under pressure to create and launch something quickly
- They believe they have a clear understanding of users’ problems and needs
- They just don’t think about user research as a step in the process
Let’s take these one at a time.
1.They are under pressure to create and launch something quickly
This one is always a red flag. When I hear this, I generally ask about risk tolerance. How much risk are you willing to accept in exchange for assumed speed? Jumping to an idea, building that idea and launching it might work. But it carries risk. It’s a leap of faith to skip steps that may help you understand the problem you are trying to solve. If your risk tolerance is high and expectations are clearly set that you are taking a chance, I think it’s fine to jump to ideas and skip user research. But there are still ways to mitigate that risk along the way. More on that later.
2.They believe they have a clear understanding of users’ problems and needs
I’ve worked on products where I felt really close to the users. In those cases, I’d spent time speaking with them, observing them and combing through data about how they were using whatever it was. In those cases, I was well positioned to come up with ideas that might work, but still needed to prototype and test to validate assumptions. I’d actually been doing the research all along.
People often refer to Steve Jobs as a visionary genius who didn’t bother with user research. I would argue that Steve Jobs was absolutely a visionary and a genius AND that he was paying attention all the time. He was studying his own experiences and the experiences of people using early computers, early portable music players, early mobile phones. Maybe he just paid attention more than most people and was better than others at developing that base of knowledge about what would meet their needs. That knowledge base was there for him to draw from when creating transformational products like the iPod and iPhone. But not all of us are Steve Jobs.
3.They just don’t think about user research as a step in the process
I see this a lot. Teams will want to jump to ideas. When I ask if they are interested in exploring the problems, they are first intrigued, then have a realization. “Of course! That makes so much sense. Let’s figure out what we are trying to solve first!”
After reading this, if you are interested or just curious about how you might conduct user research to reduce risk, then read on.
There are many types of user research methods and they are not all created equal. Below I’ll go through some of them and note what they are, what they can be used for and most importantly what they can be paired up with to gather the whole story and reduce your risk.
Focus Groups
What they are:
This research tactic may be the most misunderstood and misused of them all, so let’s start here. Focus groups are generally facilitated discussions with groups of people where they gather to talk about a topic, react to an idea, product or service.
What you can learn:
They can be useful at generating new questions or ideas to research further, but they can also be incredibly destructive to a project. Focus groups, much like group brainstorms come with all sorts of built in risks. They can be dominated by a strong personality. They can suffer from groupthink, where group members are influenced by each other thereby reducing the amount of individual thought and contribution. They also require a highly skilled facilitator capable of managing a group of individuals.
The following quote is from the creator of the focus group concept himself, Robert K. Merton.
“Even when the subjects are well selected, focus groups are supposed to be merely the source of ideas that need to be researched.”
What they can be paired with:
Because of all of these risks, I almost never recommend focus groups. Instead, I recommend conducting a series of 1:1 user interviews to learn about people’s experiences. I also recommend user tests to learn about people’s reactions to an idea or product. If you do them, be sure to pair focus groups with prototyping and user testing, along with some additional 1:1 user interviews to help you validate.
The verdict:
Stay away! If you do have a skilled facilitator and are willing to accept the risks of focus groups, then I recommend you listen to Robert K. Merton’s warning and treat the activity as a step towards further research. Be sure to manage expectations with your team and stakeholders that the answers likely won’t come from a focus group. At most, you may be able to generate useful questions. Also, if focus groups are the only research method being used in a project, then sound the alarm.
Surveys
What they are:
Surveys are a tool for gathering answers to questions from people. Oftentimes, surveys are conducted asynchronously, meaning the creation of the survey and the completion of the survey do not happen at the same time. This allows you to gather input from many survey respondents over a period of time. They can be digital, delivered through email or as an intercept pop up on a web site. They can be paper.
What you can learn:
Surveys are a great way to gather input from a group of people without having to directly interact with every person. Surveys should be quantitative and used to help you understand “what” is happening, but they aren’t a great method for understanding “why” something is happening. This is a common mistake. Many people will create surveys that ask people to rank things “on a scale of 1 to 5” but then also ask people to explain why. If you are only surveying a few people, this might work and you may be able to spot patterns, but regardless of the sample size, you lose the opportunity to dig deeper on the why. Much like focus groups, surveys can be misused. Consider this quote from Erika Hall:
“Surveys are the most dangerous research tool — misunderstood and misused. They frequently straddle the qualitative and quantitative, and at their worst represent the worst of both.”
What they can be paired with:
Like spaghetti and meatballs with a fine Chianti, user interviews are perfectly paired with surveys. The surveys will give you a great idea where to focus your questions and the interviews will help you correlate the what with the why.
The verdict:
Use ‘em! But limit the questions to just the quantitative ones. That is, anything that allows the answers to result in structured data. As a pro tip, I recommend adding a final question in surveys that asks if you can contact the respondent to discuss further. It helps you build up a pool of people to interview.
User Interviews
What they are:
User interviews are lightly structured conversations with users with the goal of learning about those users’ experiences. “Lightly structured” is an important detail. You’ll need a plan and a list of the experiences you hope to learn about. But, you don’t want to simply rattle off questions and record answers. Get the stories behind the answers. Dig deep, find out why! This is the most common mistake I see with user interviews. Teams will create a set of questions and then power through the questions, record the answers and be done. Instead, get the stories. And if you do have more quantitative yes/no, or ranking style questions, put them in a survey!
What they can be used for:
User interviews can be used to ground you and your team in the experiences people are having — their needs behaviors and desires. They show you where to look. They help build empathy. And when things get difficult on a project and the team is getting on each other’s nerves, it’s the empathy that will carry you though to the finish line.
What they can be paired with:
Like peanut butter and jelly and a glass of cold milk, pair this tactic up with surveys and then follow them up with user tests after you’ve generated ideas!
The verdict:
User interviews are the godfather of all user research techniques. I don’t think you should ever just pick one user research tactic, but if you had to, this is the one. They will ground you in the empathy you are going to need to keep you focused on the right problems or opportunities in your project.
User Testing
What it is:
User testing can take many forms, but you always need 3 components.
- A learning goal — make sure you are clear about what you hope to learn from the test — usually to confirm or deny an assumption learned from user interviews or other research.
- A scenario for the tester — in order for the tester to provide useful feedback, it’s important to create a real-ish scenario so the tester knows what mindset and context to consider when reacting. Don’t just put something in front of people and ask if they like it.
- Something to test — usually this is some sort of tangible prototype, which is nothing more than an early form of something. It can be a sketch, a digital prototype, a physical prototype, but must be something testers are able to interact with and react to. Describing a concept to someone and asking if they like it isn’t enough. Steve Jobs was often quoted as saying:
“People don’t know what they want until you show it to them.”
What you can learn:
User testing can help you learn about user understanding, help you test your own assumptions, and even settle a feature dispute between colleagues — whatever the goal, user testing is a fantastic way to de-risk a project by learning early on if you are on the right track.
What they can be paired with:
Pair user tests with user interviews. It’s best to do user interviews first to help inform the generation of ideas. Then you can turn those ideas into prototypes and test away!
The verdict:
Do this. Test early and often. Don’t worry about perfection either.
A stone thrown in a lake will result in a splash proportional to the size of the stone; a user test should result in learning proportional to the effort put into your prototype. — Joe Lalley
Card Sorting
What it is:
Card sorting can be done with paper cards or digital ones. In either case, you will note each of your product catalog, content items or menu items on a card — one per card. Then you ask users to group and organize the information in the way that makes sense to them. You can do an “open” card sort where you let the users organize everything and even create categories or a “closed” card sort where you may provide a few top level categories and ask users to place them in the categories they think make sense. It’s kind of like a user test, in that you will set up the test and get out of the way. You don’t help or explain. Card sorting is most often used to help develop the information architecture for a product catalog or a website. For example, card sorting would be a useful tactic for an apparel company looking to organize their products based on gender, brand, size, season or any number of other ways- and wants to make sure the information is organized in a way that makes sense to users.
What you can learn:
A card sort can help teams learn about the way customers organize information in their own minds. It can also be used to determine if the current information architecture makes sense to users.
What they can be paired with:
Pair this with user testing to learn about the entire experience. First do a card sort to understand how users interpret your content. Once you have a sense for that, build that information architecture into a working prototype and test whether or not they can effectively complete tasks on your site. (ie: find a gift for a friend)
The verdict:
This tactic isn’t right for every project or every phase of a project. If you have a lot of information to organize as part of your experience, then a card sort is probably right for you.
Contextual Inquiry/Ethnography
What it is:
This is all about observation. It involves observing people interacting with something in the the way in which they normally would. For example, if you wanted to learn about how people experience a self checkout process at a grocery store you would simply watch people check out. Or if you wanted to learn about the dinner routines of families, you might spend time in their homes observing. The key is to not interfere, but to observe like a fly on the wall. (I’ve always wondered if flies are even paying attention though.)
What you can learn:
It is often used early in a project and for extended periods of time and can help teams learn intricate details of people’s experiences.
What they can be paired with:
This should be accompanied by user interviews and user testing after the team has learned enough to generate ideas that might meet an observed need.
The verdict:
This one can help teams gain a level of understanding about their users that is hard to match, but it requires significant time and effort to do well. I’d recommend doing some light weight observation wherever possible along with user interviews and then moving to ideation, prototyping and testing to gain a well rounded understanding.
A/B testing
What it is:
A/B or multivariate testing is a form of research where you set up dynamically generated options for users to encounter. These options are usually generated by software that can be programmed, for example, to show half of site visitors a blue button and the other half a red button. The button shown is randomized but also split evenly across the user base. The system can also be programmed to “optimize” towards the “winning” option after a certain number of users have encountered both options.
What they can be used for:
A/B tests are quantitative in nature and allow you to test with large groups of users. You can learn what is happening, but not why. Digital products are well suited for this type of testing because web and app technologies are easily integrated with tools to facilitate this kind of testing. Some teams may forgo more qualitative efforts like user interviews and user tests and simply present multiple options and automatically optimize for the winners.
What they can be paired with:
A/B testing will tell you what users are doing or reacting more positively towards, but to learn the whole story, pair this with user testing to understand the what and the why.
The verdict:
It requires some tools and a team to manage the pipeline of tests so the tests don’t trip over each other, but if you have the resources, it can be a great way to manage a product roadmap.
Diary Studies
What it is:
Diary studies involve having users keep some sort of record of their interactions with or around an experience. The tracking methods may be written, audio or video and the users themselves are the ones capturing the research. This tactic shares some similarities with Ethnography in that users interact with the experience in a natural setting, but is different in that the interactions are generally not monitored in real time, but afterwards at routine intervals.
What they can be used for:
Learning how users experience something over a period of time when an experience may consist of many interactions and touchpoints.
What they can be paired with:
Similar to contextual inquiry, diary studies pair well with user interviews and post ideation user testing.
The verdict:
This tactic will generate a lot of unstructured data so it’s important to allocate the time and effort to synthesize the data after or as you capture it. If you have the time and resources, go for it.
Why should you do user research?
Simply put, user research can help you reduce risk. Understanding those who may use your idea, product or service puts you in a better position to create something that will successfully meet the needs of those users. We all have biases and even if you consider yourself a user of something, you probably aren’t well positioned to represent all users. In my career I’ve worked on many many products that I would never use personally that solved problems I have never had. I’ve worked on products that appealed to young children, music genres that I did not enjoy, video games that I would never play, content I would never watch, articles I would never read. In those situations, I had to work hard to understand the experiences of others and only after I’d done that research was I positioned to generate ideas.
Ultimately, it comes down to risk tolerance. Ask yourself: What’s the best that could happen? What’s the worst that could happen? and What’s it going to take to reduce the risk? If you can tolerate the risk of a bad outcome then you may be OK with little to no user research. If you can’t, then give one of the research methods in this article a shot. Good luck!
Part 2: Make the most of your user research and synthesize like a pro
