Grading predictions about UX this year
Plus: how to make better predictions, and why you should
Life would be easier — if a bit less interesting — if we could reliably predict the future.
As it is, though, jokes about the unreliability of weather forecasts are common enough to induce groans and eye rolls. Over the past several decades, sophisticated statistical models have given new life to election and sports forecasting — but no one has yet claimed Warren Buffett’s offer of $1 billion for a perfect NCAA March Madness bracket.
Even so, it’s a kind of holiday tradition for companies and thought leaders across the industry to publish predictions for the year to come.
But has anyone taken a close look at how those predictions ended up panning out? To find out, I took a look back at articles projecting themes and predictions for tech and UX in 2022.
Some of the predictions, and the results
Last December, many in the tech world predicted 2022 would be a banner year for mainstream applications of cryptocurrency and web3.
Instead, one of the more notable stories in tech this year was the collapse of the NFT and crypto bubbles. Bitcoin prices are down to almost one-third of their value from one year ago following turmoil with the stablecoin Tether and the now-bankrupt exchange FTX. Although this crypto winter may well reverse and turn into another bull run in the years to come, for 2022, I rate this one a miss.
On the other hand, others felt that 2022 would be a big year for artificial intelligence.
And with the hype surrounding DALL-E 2, ChatGPT, and similar projects like Make-A-Video, this prediction has borne itself out in one sense. But have we yet reached a tipping point where AI is regularly solving user problems beyond a few limited cases? Insofar as UX practitioners are specializing in AI interaction, we’re still in early stages. Nevertheless, I tentatively rate this one a hit.
A number of experts also predicted another year of incredible growth for UX research and design.
Although I agree that greater growth and demand remain the field’s long-term trajectory, 2022 was largely characterized by an economic recession whose impact has been disproportionately felt in tech and UX in particular. There’s evidence that the field continues to grow at a dampened pace, but a modest increase doesn’t match the tone of these predictions. I rate this one a miss.
Lastly, many observers predicted greater investment in accessibility across the industry.
Beyond anecdotes, this prediction’s difficult to empirically evaluate as data on organizational spending are mostly confidential. One could approximate it — perhaps by conducting a comparative keyword analysis of accessibility terms appearing in UX articles between this year and last, or a similar review of accessibility-related job titles among professionals on LinkedIn. The larger problem, though, is that this prediction fails to operationalize how it might be right or wrong — so I rate this one too squishy to call.
In short, predictions for 2022 turned out, as one might have expected, to be a mixed bag: some hits, some misses, and some that can’t be called.
Researchers should make predictions
Even if we have a mixed track record, making predictions is part of our job description as UX researchers.
We are routinely predicting that our organizations, products, and users will have better outcomes based on our research. If we aren’t confident in those predictions, we should say so. Otherwise, our work is either reckless or purely academic.
Plus, working in tech means keeping an eye on metrics and other KPIs.
As we head into Q1 2023, many organizations will conduct performance reviews and set OKRs for the quarter and year to come. Setting individual, team, or organizational OKRs is ultimately a kind of prediction about what we intend to achieve. Since a meaningful part of our compensation and job evaluation is based on how we meet those predictions, we are highly motivated to make them as accurate as possible.
There’s also a strong social expectation that experts make predictions in their field.
Anyone claiming to understand what’s happening in their area of expertise should know the trends well enough to suggest what will happen — even if they’re wrong. Some predictions, like Moore’s “law,” turn out to be self-fulfilling and a boon for the industry. And other predictions will be wrong in the short term, but prove right in the long term.
Making better predictions
To make your predictions better, make sure your predictions can be wrong.
A falsifiable theory has the possibility of being disproven — and this quality has been a cornerstone of good science since Karl Popper first introduced the standard in 1934. For example, the claim that “all swans are white” is falsifiable, as observing a single black swan would disprove it. Likewise, a good prediction should state, or at least imply, the conditions under which it would be untrue.
Next, acknowledge that a range of future outcomes are always possible.
Weather, election, and sports forecasting at their best are probabilistic predictions. We plan our outfits and days differently for a 10% chance of rain versus a 60% chance. Similarly, we can distinguish a hot take from a sure bet when forecasters provide their own confidence level on a similar scale of 0 to 100% chance.
Finally, give the outcome of your prediction some consequences.
A great example of this comes from prediction markets, or communities devoted to forecasting on a number of topics. Some, like Metaculus, work by wagering reputational points, and others, like PredictIt, involve buying and selling shares in a prediction using real money. Similarly, friends and colleagues can play fantasy sports for bragging rights, while others might prefer financial reward from sports betting.
We probably won’t see fantasy UX games any time soon, but we can still hold ourselves accountable.
FiveThirtyEight regularly conducts post-mortems after elections to evaluate and improve their forecasting models. Writers Scott Alexander and Matt Yglesias both “pre-register” their predictions for the year to come and then grade their predictions over the past year.
Putting falsifiability, probability, and accountability into regular practice will make us all better forecasters.
Summary
Experts routinely share predictions for the year to come. How did those turn out in 2022?
- As a whole, they were a mixed bag. Some, like greater attention to the UX of AI products, were hits. Some, like continued explosive growth in the field, were misses. And others can’t be called because the relevant data are confidential.
- Nevertheless, UX researchers should continue to make predictions. Our work and recommendations are a kind of prediction about the success or failure of our organizations and products. Predicting OKRs is also part of how we’re evaluated.
- Good predictions are falsifiable, probabilistic, and hold us accountable. Even when they turn out to be wrong, getting into a regular practice of making good predictions will make us better at seeing what lies ahead.
The past several years are full of examples of both unprecedented events and the unfortunate consequences of failing to anticipate them. Let’s all spend some time giving serious thought to the possibilities of the year to come, and how those may affect the work we do as UX practitioners.
If you’ve found this article useful, sign up for Depth by Drill Bit Labs, a free bimonthly newsletter on UX research and digital strategy.
