UX Measurement #3

Seven heuristics for identifying proper UX instruments and metrics

TL;DR: To properly measure UX, you need what’s called an instrument. One example is UEQ, the User Experience Questionnaire. However, there are plenty of instruments out there that supposedly measure UX — often promoted by companies more concerned with sales and marketing than accuracy and reliability — but don’t really. So, how can we identify the good ones? That’s exactly what this article helps with. It gives you 7 heuristics to easily identify metrics that actually measure UX and shows how to apply them in 2 short case studies (UEQ — good; and QXscore — bad).

This is the third in a series of articles on user experience measurement:

1. Conversion rate & average order value are not UX metrics
2. So, how can we measure UX?
3. ☞ Seven heuristics for identifying proper UX instruments and metrics

Created with Microsoft Designer. Prompt: “A researcher looking at a range of different metrics, colored drawing.”

In the two previous articles of this series, we have first learned that metrics such as conversion rate, average order value, or Net Promoter Score are not suitable to measure user experience (UX) [1] reliably. The second article then explained how UX is a latent variable and, therefore, we must rely on research instruments and corresponding composite indicators (that produce a metric) to measure it [2]. Now, the logical next question is how we can identify those instruments and metrics that do reliably measure UX. This boils down to what is called construct validity and reliability, on which we will give a brief introduction in this final article, before deriving easily applicable heuristics for practitioners and researchers alike who don’t know which UX instrument or metric to choose.

Construct validity refers to the extent to which a test measures what it is supposed to measure. In the case of UX, this means that the instrument or metric should measure the concept of UX as it is understood in the research literature and not, for example, only usability. One good way to establish construct validity is through factor analysis.

Construct reliability refers to the consistency of a test or measure [4]. Put differently, it is a measure of how reproducible the results of an instrument or metric are. A good way to establish…