A framework to define your product metrics

Have you ever felt that you created a great experience out in the world and a lot of people use it but you’re not sure how to explain to stakeholders what the impact is? Perhaps spent hours digging through historic data to try prove that your design/idea made a difference?

It’s one thing to survey your customers and get qualitative validation on your feature, but it’s an entirely different beast proving it as a cold hard statistical fact.

In this article, I won’t be going into the low level mathematics, instead, I’ll try to provide a high-level framework which might help you choose and define metrics for your product.

Before I begin, I’m going to start with some assumptions about you as the reader:

1. You as the UX Lead, Product Designer, UX Designer, Product Owner, Product Manager(or any other swanky titles) are responsible for tracking the success of your project — in opposed to having a research department or a scientist in your basement.
2. The company you are working for is or has already been tracking data on its product and services. If not, maybe this article prompts you to get a first database going.

External vs. internal metrics

No doubt somewhere in your career you’ve heard the phrase “as long as it moves the needle” (probably from someone who has vested interests in that needle) … When they talk about the needle they are usually talking about some business metric; the number of leads, revenue, turn around time etc. We’re gonna call these External Metrics.

Then there are the metrics of your product (or experience), the things that tell you about how people are using what you designed; adoption, click-through rates, engagement, we’re going to call these Internal Metrics.

With that in mind:

Every project, product or feature should always START with an external metric in mind…

This is extremely important because it connects what we do as tech to the things business cares about, but it also, it will add some higher stakes to your product, encouraging you to ruthlessly prioritise and thoroughly research before you build anything.

But there is a second part to my previous twitter-ready quote:

… but please, for the love of god, don’t make the metric ‘sales’, ‘revenue’ or ‘profit’

All companies exist to be profitable even the non-profit ones stimulate some sort of income to sustain themselves. Every external metric should ideally cause profit, for example, increasing the number of properties a real estate company has on their books should increase the number of sales.

So when we put all of this together we get something that looks like this:

Metrics Pyramid?

Notice how all the metrics seem interconnected. I want you to imagine for a second, hundreds of number sliders.

Network of sliders

When you move one slider it affects each other slider differently, some move forwards, some move backwards. Imagine a complex relationship existing between each slider and every other slider.

This network effect is how metrics (in my experience) tend to behave. When you affect one metric like “number of leads” you might get a negative effect somewhere else (maybe more dropoffs in a later process or a spike in manual tasks). Everything is connected. Rainbow emoji. 🌈

What you are trying to discover is the optimal “slider” to affect an external metric and in turn, affect profit.

Choosing a good external metric

Enough theory. Let’s use the example of a specific external metric I worked on with Zaheera Ismail:

Zaheera is the UX Lead on a project to improve Leadhome’s support centre technology and processes and we needed to define some metrics to guide the direction of her project. One of the areas we were looking at was the time it takes to resolve a case (cases are how the customer support agents keep track of communications with clients).

For this project, we could have easily gone for Reduce the average time it takes to complete a case but after looking at the current metric closely, we realised that 27% of cases are resolved within the same hour they are created, this is inspired a more considered metric: Increase the percentage of cases resolved in the first hour.

So we changed the semantics, big deal… what does that matter? Well… by simply changing the goal it changes your approach. By confining your metric and being more specific it becomes easier to experiment and design for. In this project, it became about identifying and prioritising the cases that should be solved faster.

So remember:

Always try to make your chosen business metric more specific to the experience you are trying to create… and also again … please don’t make it ‘sales’, ‘revenue’ or ‘profit’.

Let’s say you’ve grappled with the business logic within your organisation and found a potential metric for improvement.

First, I want you to make peace with the fact that external metrics by their very nature are exceptionally difficult to affect so your next mission is to try to establish a strong connection between some internal metric you can affect and the external metric you would like to influence.

Choosing Internal Metrics

Now sometimes these connections are apparent like conversion rates between steps in your online wizard and the number of leads. But sometimes they need to be shown to be true. Like engagement with social proof and the number of leads.

In statistics, this is called causation and it’s notoriously difficult to prove. I would suggest doing a small trail experiment that costs little to no money to prove the relationship. (e.g. in the before-mentioned example, use an A/B test with social proof on a single low converting step—see what happens.)

In the case of Zaheera’s project, she focused on improving the accuracy of how the system prioritises different types of cases that had a higher likelihood of success.

She found that prioritisation could be greatly improved through a series of needfinding interviews with Leadhome’s support agents.

The logic went something like this, if we increase the prioritisation accuracy (internal metric) then that will, in turn, affect the percentage of cases resolved in the first hour (external metric) and ultimately increase profit.

For interest sake, Zaheera’s work increased the percentage of customers converted in the first hour from 27% to 36%.

Conversion before and after Zaheera’s project

The TL;DR version

1. Start with an external metric that has potential for improvement
2. Find a series of internal metrics that have a connection with your external metric (…and prove the connection)
3. Design a feature/product/service that drives your internal metric
4. Watch your external metric to see if you were successful

I really didn’t want this article to sound so wolf-of-wall-street-esque, there is definitely more to creating great experiences for customers than cold hard numbers and profit, but I often see projects fail because they lack focus. As product designers/owners/titles-titles, we are often in the details, this dropdown, that button, this colour…

Usability can always improve through iteration, but a well-researched feature can make a massive difference in your organisation, even inspire a whole new revenue stream.

UX / Product designers are qualitative researchers at heart, empathy is an amazing tool, but numbers are too, and the combination of the two is extremely effective.

I’m thinking of creating a more tutorial style article on this subject, let me know if that’s something you guys would be interested in.