Three levels of UX Research impact
Evaluating the impact of UX research is hard. Here is a framework that can help.
“John did great work on this user study, but the results haven’t led to any product changes yet. How should we talk about the impact of his work in this perf cycle?”
“Perf” is a shorthand for “performance review” at Google, and yes, it’s that time again to evaluate the impact of our work and determine who gets raises and promotions. As a UX research (UXR) manager, it’s my job to help my colleagues understand how they should look at the impact of UXR, which many found not as straightforward or as quantifiable as some other disciplines.
There are two fundamental challenges in evaluating UXR impact: a long time horizon and limited control of its usage. First, the impact of a study might take some time to realize from its completion. This is especially true for foundational research which is intended to build a baseline understanding of potential users and their needs before the team even starts designing or building a product. Second, UXR might have indirect or little control over how their research gets used by others in the organization. This is usually not an issue for researchers in an agency setting, since impact is realized when the research report is accepted by the client. But for in-house researchers, there is an expectation that research needs to lead to changes in product design or revenue to be considered impactful.
Wrestling with these two challenges initially as an individual contributor and more recently as a manager led me to a mental framework that has helped me better articulate my team’s impact and set the right expectations about what stakeholders can get out of engaging with the UXR team. This framework builds upon my understanding of the core job UXR does — producing knowledge about product users and product designs to support decision making. Breaking down this understanding of the UXR role, I arrived in the following three levels of impact:
- Level 1: research questions answered
- Level 2: decisions supported
- Level 3: user value generated
As the diagram below illustrates, the higher levels of impact not only take longer to realize but also depend on the influence of the UXR team and the organization’s capacity to act on research rather than individual researcher’s expertise and diligence.
Realizing the first level of research impact is relatively straightforward: the researcher needs to answer important research questions and answer them really well. Every business faces uncertainty in their decision making, and UXR helps reduce that uncertainty through collecting, analyzing, and reporting data about users and how they use the product. This uncertainty is usually expressed as questions the team asks or should have asked — sometimes we don’t know what we don’t know.
When research questions are well defined and reflect what the team really needs to know, the first level of research impact can be assessed almost immediately after the delivery of the final research report, based on whether or not stakeholders feel the research questions have been answered satisfactorily and the uncertainty that prompted the research has been reduced if not eliminated.
Picking the right question to answer is as important as, if not more important than, answering the question well. If the research addresses an inconsequential question, its impact will be limited regardless of how well it is carried out. Therefore, it’s very important to make sure all stakeholders are aligned not only on what the research deliverables will look like, but also why and why not this research should be done early in the research planning process.
The second level of research impact is about decisions informed by the research. Those decisions could involve any work across design, engineering, product management, marketing, operations, etc. This means that the stakeholders of the research not only agreed with and accepted the research findings, but they also found ways to apply what they learned in places that could directly add value to the product or the organization. It’s important to note that cancelling an activity that was assumed or planned, because of what the research has suggested is also a legitimate form of research impact.
Nonetheless, a well-done research study doesn’t always lead to explicit or immediate decisions. Sometimes the solution space for a problem uncovered through research can be highly ambiguous, though a correct framing and a rich description of the problem can often pave the way to finding the right solution eventually. That’s why socialization of research is an ongoing and long-term activity. Someone needs to remember that the organization already “knows” this and brings up the knowledge at the right time.
The final level of research impact is realized when the research has generated measurable value to the user through well-made business decisions informed by the research results. It can take months, sometimes years, to confirm the research has had an impact at this level. To do this well and consistently, the organization needs to invest in systematic measurements and cultivating a culture of project landing instead of launching. Good business or design decisions, especially those that can stand the test of time, reflect the quality of research that supported those decisions.
These three levels of impact are related to the career growth of UX researchers. An entry-level UXR should aim to achieve the first-level impact by answering well-defined research questions independently. As the individual gains more experiences and organizational context, they are expected to actively influence and support cross-functional decision making. Finally, a UXR Lead role embraces the responsibility for creating a culture and processes to ensure the whole organization regularly makes sound, data-based decisions.
How do you like to talk about UXR impact? I’d love to hear your thoughts in the comments.
If you’d like to dive deeper into the topic of UXR impact, check out the following articles for complementary insights:
