How do you measure UX? It depends on how you define ‘UX’
“If you cannot measure it, you cannot improve it” is a management mantra that I often hear as a UX designer. But the “experience” seems to be one of those nebulous, immeasurable things. It doesn’t have to be that way.
When you hear “measuring UX”, many people immediately think of the Google HEART framework. With it, you work with the goals you want to achieve, signals that indicate that the goal has been achieved and metrics that indicate success. You measure Happiness, Engagement, Adoption, Retention and Task Success.
Even though Google HEART is a very solid framework, I think that measuring UX is a broader subject than ‘mere’ happiness-engagement-adoption-retention-task success set.
What would it be, then? Different professionals would give different definitions. That definition depends on the role of User Experience in a company/project (aka UX maturity). Measuring User Experience thus has several aspects:
- Usability
- Satisfaction
- Relevance of the product and its functions
- Experience and “delight” of the customer
- Contribution of UX in achieving the business goals
Let’s go through them one by one.
Usability
The purpose of a usability study is to make products more user-friendly through qualitative insights about what works or does not, and why. If you still want to boil it down to a measurement, it is (kind of) possible.
No objective score
Spoiler: it is impossible to measure usability as such. You can only measure the perception of the user-friendliness, whether the customers find something user-friendly (as long as it feels easy and smooth, even it is not so objective). It is important to realize that this score might further be biased by people who have a strong self-monitoring personality trait. They often give inconsistent answers and socially desirable responses.
Since the 1980s, several questionnaires have been developed that are used to measure the perception of user-friendliness. They all contain different formulations and structures, but come down to the same thing: you get an unambiguous number as a result.
Questionnaires you can use
The SUS (System Usability Scale) is one of the oldest and most popular questionnaires measuring user-friendliness. This is due to the fact that it is open-source, that it is widely known, and also that it has been translated into multiple languages. This questionnaire, with 10 statements, has been scientifically tested for validity (whether it measures what it should measure) and reliability (whether it gives consistent results when tested in multiple test rounds).
In 2010 a shorter version of SUS was designed, the UMUX (Usability Metric for User Experience). According to research, it achieves the same results as SUS with 4 just statements. In 2013, a UMUX-Lite was developed, achieving comparable results with mere 2 questions.
Satisfaction
The satisfaction score indicates to what extent the actual performance of the product corresponds to the expectations of the user. A logical solution for it would be to ask questions about expectations beforehand, and follow up with questions about the actual experience after using the product. In practice this is not always convenient. This may be because you want to get the customer through the onboarding as quickly as possible or because you don’t want to tire the customer with a questionnaire twice, no matter how short it is.
Disconfirmation scale
Fortunately, it can be easier. There is some evidence that a Disconfirmation Scale can measure satisfaction just as well as the method I just described. With this method, you ask the customer about their satisfaction with the product as compared to the expectations beforehand.
It is known that people can unconsciously adjust their memories afterwards so that they form a consistent picture with their current thoughts(this effect is called rationalization). But even though that effect may be present in the Disconfirmation scale, that is not necessarily a bad thing. The most important thing is what impression the product has ultimately left on the customer, and what he/she thinks about it. This approach works best for measuring overall satisfaction with the product.
Customer Satisfaction Index
Do you want to evaluate a smaller flow or specific features of a product? Then the Disconfirmation scale is not the most practical method. In general, people are much less aware of their expectations for smaller or transactional parts. Questions such as “Was the onboarding better or worse than you expected it to be?” will provide few meaningful answers.
The Customer Satisfaction Index (CSAT) is a good choice for these types of goals. It measures satisfaction in the “here and now”, allowing you to zoom in on specific activities or parts of the product — checkout flow, delivery, customer service, etc.
To get realistic results you have to make sure that the question is neutral, so not “How satisfied are you with X?”, but “What do you think of X?”.
Net Promoter Score is not a measure of satisfaction!
The Net Promoter Score (NPS) is often used as a method of measuring satisfaction. However, the NPS is a value that measures loyalty and expressed intentions and not necessarily satisfaction (even if there is a correlation between them).
In addition, there is a lot of criticism about the way the calculations of the NPS are done. Jared Spool, expert, researcher and speaker in the field of usability, software, and design clearly summarizes it. There are plenty of reasons not to use NPS, also not for this purpose.
Other parameters
Parameters such as adoption rate, retention rate, and churn rate are also helpful to track. They only give an indirect indication of satisfaction and cannot be viewed as an isolated number: on its own, a “39% adoption rate” means little. You don’t know if it’s the current or past performance that causes this number. Therefore, these measurements are evaluated by period (cohort analysis) so that you can see the trends over time, and they should also be viewed in the context of the industry benchmark. With this you can see whether these parameters are getting better or worse.
Relevance of the product and its functions
Finding out if the product is relevant and matches the needs of the customers is the holy grail of the product development process. You want to know whether what you build matters, and whether a product-market fit has been achieved. In short: “Are we selling something that people need?”.
Unfortunately, there is no silver bullet to easily answer this question, let alone with a hard number or percentage. If it were easy, “No market need” would not be the main reason why 42% of start-ups fail.
In order to estimate how relevant your product is as accurate as possible, you should use a mix of qualitative and quantitative methods. As we are focusing on quantitative options here, here are some methods to obtain indications of whether your product is relevant for the end user.
“How disappointed would you be …”
One way to measure product relevance is described in the book “Hacking Growth: How Today’s Fastest-Growing Companies Drive Breakout Success”. Ask your customers who have recently used the product the following question: “How disappointed would you be if you could no longer use this product?”.
If more than 40% of customers say they would be disappointed, you have reached product-market fit (i.e. your product is useful and relevant). This approach can be used well for products that are still under development and have not yet distilled their Unique Selling Points.
Technology Acceptance Model
Suppose you haven’t developed a product or prototype yet, and/or don’t have a customer base yet. In that case, you have to make do with your own assumptions and the predictions that your target group makes about their own behaviour. The answers to these types of hypothetical questions are less reliable than questions about past behaviour.
Technology Acceptance Model (TAM) is used to measure, among other things, the perception of utility. You can use this to get a sense of whether there is a need for the product. It is necessary to adapt and customize the questionnaire per product and survey based on the model.
Another model with more nuances than TAM is the Universal Theory of Acceptance and Use of Technology (UTAUT). However, this nuance means that you will also have to ask a lot more questions.
A combination of quantitative and qualitative methods is necessary
Because product relevance is a very complex subject, you cannot simply measure it with a questionnaire and express it with a number. Make sure you use multiple methods, for example, customer journey mapping, Jobs To Be Done interviews, different LEAN experiments (pre-selling, landing page, explainer video), etc. Multiple methods together give a clearer picture and a higher certainty of the answer.
Positive customer experience and delight
After the basics of your product are in order, look further. How do you exceed the expectations of the customer and make him / her enthusiastic about your product or organization? Reaching customer delight, that fleeting “extra” becomes the new goal.
What exactly does “delight” mean? That depends on your organization. There is no single definition of delight, and it is interpreted in different ways by different organizations and people. The question is therefore whether your product should necessarily create the feeling of ‘delight’. It is not the ultimate goal of UX. For some industries, I would look for a different desired feeling. For a tax advisory firm, it can be “solid” or “calm.” For a b2b shipping company, “reliable” or “business-like” is an appropriate feeling. Delight can be uncalled for or even backfire in these types of industries.
List of adjectives
One of the easiest ways to measure delight is to present the customer with a number of adjectives that represent the emotions and values you aspire to. Include negative emotions in that list and values that are opposite to what you want to achieve. The customer must indicate how often he or she experiences those emotions with the product.
In many definitions, “delight” implies a positive surprise. Once you have experienced or seen something once, the same is no longer a surprise or delight, but something ordinary that you expect. The KANO model offers further insight into how this effect works.
Role of UX in achieving business goals
Using UX as a way to achieve business goals is the final step in the journey to UX maturity. This also includes measuring the effect of UX on the intended business goals.
How do you go about this?
There is no one-size-fits-all solution for this. Depending on what your company wants to achieve and which products or solutions it offers, you will measure it in different ways.
You start with the business goals. This applies to all organisations: B2B and B2C, governments, NGOs, and for-profit companies. Is this growth? Efficiency? Behavioural change? What role does the product play in achieving that goal? Capture it. You translate this role into specific goals, which you can then further define using the Goals-Signals-Metrics technique of the Google HEART method.
Sounds complicated? Let me give an example. The goals that are not directly expressed in numbers of conversions or in earnings are more difficult to deal with, so that’s the kind of product I’ll describe (it is a fictitious product for a public sector as an example).
Example
Suppose the Ministry of Health, Welfare, and Sport of the Netherlands releases an app that anonymously guides people to quit smoking.
What is the purpose of the Ministry? What are they doing to make this vision a reality?
- ‘Our country is healthy and well’’ is the vision
- One of the steps to get there is to decrease the amount of smokers, among other initiatives via the “Quit Smoking” app
What is the purpose of this app?
- Guiding users in successfully quitting smoking
What are the signs that users might quit smoking?
- The users start on the step-by-step plan
- The users follow the step-by-step plan of quitting smoking
- The users maintain the new habit and do not start smoking again
What are the metrics of the “follow the step-by-step plan”?
- Total percentage of people who indicated that they had not smoked at least 5 times in the last 7 days
- Percentage of people who check in daily halfway through the step-by-step plan
- Percentage of people who complete the step-by-step plan
- These types of metrics need to be measured over time. You are particularly curious about the trend, and not necessarily about the absolute number.
The moment a percentage after the product change has become higher than before, you have a measurement of the impact of the UX/design. If there have been several adjustments in that period within and outside the UX area of the app, the trend is of course less directly attributable to UX. Also, not all product decisions are made by the UX department, so that, of course, makes the attribution harder as well.
Experience metrics and metrics of the effect of UX
You can also see that I do not mention here any examples of conversion rate, satisfaction, task success. These are the metrics of the experience and are instrumental in achieving good experience (and indirectly the business goals). But these scores, even though they are important, do not necessarily mean that the business goal of the product has been achieved.
These measures are difficult to define and require maturity of the organization and clarity from management about what “success” means. Discussions about this help bring the role of UX closer to the business and strategic business choices.
Final thoughts: A score is not an insight!
The methods in this article are quantitative and therefore result in a score. But as useful as they are, they only tell the “what” of the story and must be further supplemented with other research methods to discover the “why”. With these quantitative insights, you can often measure improvements over time or compare the current score with a benchmark of the industry average.
Furthermore, such a score does not directly provide insight into which steps you need to take to improve it. Such a quantitative measurement can best be supplemented with qualitative research that also provides answers to questions such as “why do people think this?”, or “what do our customers expect?”.
A score is just a score. What you do with that is what actually makes a difference.
