Bias in quantitative user research
The perils of bias in UX are well known. When it comes to usability testing, user interviews or other qualitative UX techniques the subject of bias is well discussed. We know that we shouldn’t lead users by asking questions that are more likely to give us the answers that we want to hear. We also know we need to be careful to avoid our own bias when analysing findings of our work. We need to ensure that we see the full picture, and not just focus on the findings that validate our assumptions.
What about quantitative data like website analytics though? Surely that’s black and white, and the data never lies? The data might not lie, but humans certainly do…
Why bias comes into play
While there can be all sorts of issues with the way data is collected in this post I’m going to focus here on where bias can creep in when analysing results.
The following tweet is a great example of what I mean by bias in user research:
By its very nature quantitative data is more measurable. Focusing on the numbers can make it easier to say whether something is ‘better’ or ‘worse’ than an alternative. If you run usability testing on two designs there may not be a clear ‘winner’, but if you A/B test those designs and measure the conversion rates you’ll be able to measure the difference in performance based on metrics like conversion rate (as long as statistical significance is reached).
In reality though the problem of bias is always there. Bias is a part of being human and we need to keep it in mind during all forms of our UX analysis.
Data scientist Andrea Jones-Rooy sums up the situation brilliantly in this article on why she is sceptical about data.
“Data is not fact and fact is often just a hypothesis anyway. We humans design how data is created and we are the ones who interpret data and draw conclusions from it. Therefore, data will always be inherently fallible.
Unless we approach it with a sense of humility and a willingness to acknowledge our opinions as hypotheses to be tested, we will end up using data to entrench the gut instinct behaviours we claim we want it to replace.”
Andrea Jones-Rooy, Social Scientist
Humans will see what they are looking for in the data. In my analytics for UX workshops I run an exercise where I ask two teams to analyse the same data set from Google Analytics and report their findings back to me. The twist is that one team has to report back on positive developments, while the other has to tell me the negative. Despite reporting back on exactly the same data I get two very different sides to the story, both of which are technically correct!
Imagine you’ve just made a major design change to a website and you’re looking at the impact that it’s had. If you notice that this has led to an increase in pageviews (great!) but also an increase in bounce rate (not so great!) then you may well be tempted to only focus on the former when reporting back your findings. This type of behaviour is often exaggerated by managers or stakeholders demanding results and short-term gain. When everyone wants to see signs of improvement it can often be found, one way or another.
The forms quantitative analysis bias takes
We can see why quantitative bias can be an issue, but how does it manifest itself? Keep a close eye out for the following forms of bias when analysing the results of quantitative user research:
Spinning the data
The term ‘spin’ has been used in PR and politics for decades. Essentially it describes the process of knowingly providing a biased interpretation of an event. In this context spinning analytical data means deliberately interpreting the context behind figures in a way to suit your own agenda.
For example, if the data shows that the time users are spending on your website has dropped then you might claim that this is due to improved efficiency, rather than less engaging content. A page with a high bounce rate may be explained away as answering all the users’ questions, rather than something that’s driving people away from the site unsatisfied. These interpretations may be correct, but it’s up to you to find a way to prove that is the case instead of making assumptions with nothing to back them up.
With a bit of effort it can be fairly easy to spin the results to prove your point, whatever it might be.
Cherry picking results
Another common way that bias can creep into analysis is where results are ‘cherry picked’ rather than reported consistently.
For example, if the number of sales of products through your website has dropped, but the conversion rate has increased, then you might be tempted to present the conversion rate figure to your boss while keeping quiet about the drop in sales.
Not comparing like for like
One simple way to show increases in your key metrics is to make unfair comparisons. If your website sells heaters then comparing your winter performance to your summer performance is not the most unbiased way of evaluating your website. Comparing weekdays and weekends can also be a way to, deliberately or otherwise, skew your data to tell a different story.
As well as seasonality you also need to make sure that you’re not taking advantage of, and taking credit for, external factors that you might not be responsible for. For example, if your ecommerce website has a big sale on, this is more likely to be the reason for an increase in conversion rate than the changes you made to your product pages. Be careful not to confuse correlation with causation just because it gives you the results that you want to see.
Report your quantitative findings as consistently as you can and if you’re making data comparisons try to do so in a like-for-like manner.
Only using quantitative data
If you’re guilty of any of the above then you can really compound this by only using quantitative data. If you can spin, cherry pick or otherwise skew this data to fit your story then it can be hard to disprove if you’re also ignoring qualitative data.
For example, if you page has a high bounce rate and you claim that this is due to the information on the page being so perfect that all your users are leaving your website happy, then it’s hard to disprove this assertion with other quantitative data alone. Some quick usability testing may show that users hate your page and can’t get away from it quickly enough, but by deliberately avoiding any forms of qualitative analysis you can hide behind the figures that you want to show.
Data visualisation presentation
Finally, bias can take the form of how the data is presented. Data visualisation can allow us to present the ‘correct’ data in a way that strongly indicates a very particular type of result.
Accidentally using bad data visualisation is one thing, but deliberately bad data visualisation is a very effective way of misleading people. Some examples of how visualisation can be used to manipulate data include:
- Not starting from a zero baseline on bar charts (so small increases can look enormous)
- Connected to the above, using misleading scales is another way to make small changes look very dramatic indeed
- Using carefully selected timescales, e.g. cutting off a line graph just before a big decrease occurs
- Leaving important aspects off labelling in order to only show misleading information e.g. this example, where the headline talks about Trump’s approval rating, but it’s not made clear that this is only among members of his own party
- Grouping data inconsistently e.g. showing data for people aged 20–29, 30–39 and 40+
- Deliberate obfuscation of data using confusing or hard to ‘read’ visuals e.g. showing a pie chart without value labels in order to make it hard to compare the segment sizes within it
There are many ways that data visualisation can be used to further amplify a bias interpretation of your data — further proving that while data doesn’t lie, humans often do.
We see people using the data to meet their own needs all the time. I can look at the data and see a worrying downwards trend while a marketing manager may look at it and pick out the only positive from the data set.
Julian Erbsloeh, Head of Insight at Fresh Egg
With all of this in mind you’d be forgiven for wanting to ignore your analytics but this would be a huge mistake. As I cover in my book, quantitative data can play a huge part in good user research. Instead of ignoring the data let’s have a look at how you can make it more trustworthy.
How to avoid bias in your analysis
The first step in avoiding bias is being aware that it could be an issue. Hopefully by now you’ll have a good understanding of why and how bias might creep into your analysis. Being aware of it makes it easier to spot and easier to avoid in the first place.
Whenever you find yourself checking quantitative data just take a moment to consider whether you might be guilty of any of the biases listed above.
The following tips should help you ensure that you’re analysing and reporting accurately.
- Report consistently
To prevent cherry picking data you should set up solid reporting foundations to ensure consistency. If you’re looking at evaluating the performance of a whole website over time you might want to begin by creating a measurement plan. For more focused projects though you may just want to define your success metrics upfront in order to avoid only reporting the numbers that look the most favourable. Make sure that you agree on how you’re going to be measuring success of a design change before anything is launched.
2. Get others to check your work
It can be hard to identify your own bias so getting another perspective on your data can be very helpful. By involving a colleague in your analysis (and not leading them!) you can get a different, and perhaps more honest, opinion. Asking someone else to evaluate the same data that you have been analysing can really help to challenge your findings.
3. Get to the root of the problem
If bias in your analysis is a consistent issue then you may need to look for any underlying problems. If you are consistently providing deliberately misleading information to back up your points then you need to consider why this is happening. Are you under pressure to deliver impossible results? Are you fearful of being judged unfavourably against others? Do you need to prove something in order to get a bonus, pay rise or promotion? Consider why you are presenting misleading results and think about how best to prevent this from happening. The section below gives some useful tips on the positive side of presenting ‘unsuccessful’ results.
4. Embrace ‘failure’
Learning from your mistakes is a vital part of the job. So much so that UX Consultant and Coach Adrian Howard runs sessions encouraging people to celebrate their failures. Anything that doesn’t go to plan is an opportunity to learn and improve. There’s no need to allow your bias to show the results that you want to see, embrace changes or tests that show a downturn in your analytics and see them as a lesson learned rather than a failure.
5. Learn from experience
By keeping a close eye out for bias in your work, and also in the work of others, you’ll learn how to spot common issues. When it comes to analysing your data think carefully about mistakes that you, or your colleagues, may have made in the past and look out for any signs of repeating those mistakes. Even if you’ve ‘got away with’ using misleading techniques in the past you need to be thinking about whether it’s right to continue using those techniques in the future. It’s likely that the long-term impact of using bias analysis will lead to negative results such as wasting money investing in the wrong areas.
Don’t overlook the data
If you’ve got this far you might be considering giving up on quantitative analysis entirely and focusing all your efforts on qualitative techniques instead! This would be a huge mistake though as it means you’d be overlooking a large amount of vital information about your users and their behaviour.
Whether you’re trying to identify problem areas on your website, or you want to get started quickly with some initial user research, analytics data can be very powerful.
This kind of data can also be incredibly useful for measuring the impact of your design changes, so overlooking it is very short-sighted.
Using a mixture of quant and qual is vital for great user research, just make sure that you’re doing everything you can to avoid allowing your own bias to creep in.
What Next?
I hope you’ve found this post helpful. It’s surprising how often bias is used in quantitative analysis, but how rarely this is discussed. Following some of the tips in this post should help you to minimise your own bias and allow for more accurate and useful analysis.
If you want to know more about how to use analytics data to inform your UX decisions then you might want to buy my book, come along to one of my training workshops or get in touch to find out how I can help.
