Product managers: Not all metrics are good metrics

How to use the right product metrics, and use them right.

In the world of Product Management, we’re often told that we need to be more data-driven. Sometimes, leaders from across the business will ask us to back up our decisions with some sort of metric. Where’s the data that proves this hypothesis? However, metrics are highly context-dependent. Data must be used carefully, to inform decision-making rather than blindly drive it. When we automatically default to leaning on the only data available without first understanding context, it can lead to a negative outcome.

Let’s look at two example metrics that are often captured in digital products: “number of clicks” and “time spent on page”.

To the layperson, a longer time spent on a page could be equated to higher adoption. And in the short term, that could very well be the case. But when you zoom out, more time spent on a page could also be a symptom of a challenging user experience. How do we benchmark the optimal time a user should be spending on a given page? And do we even care?

The same could be said about the number of clicks. Driving down clicks is usually used synonymously with “simplifying” or “streamlining” the user experience. But in contrast, the number of clicks isn’t a golden rule for product design. What if your product requires a lot of data entry to initially onboard the user? Would a single page with 5 forms and 1 submit button mean it’s more simple? Or would 5 unique prompts, each of which requires a click of “next”, reduce the cognitive load and streamline the experience? The answer usually lies somewhere in the middle (“it depends”). As I said, context is everything.

These two examples can be chalked up as vanity metrics. They look good on paper but in reality, they don’t paint a picture of success or provide any actionable insights. They’re measurements, but they’re not necessarily measuring outcomes. In most cases, the time a user spends on a page is completely useless in helping prioritize the next feature to build.

In the worst case, vanity metrics act as a Red Herring — diverting attention in the wrong direction, potentially resulting in a negative impact. It’s the job of the Product Manager to be an…