Aga Szóstek, PhD is an experience designer with over 19 years of practice in both academic and business world. She is an author of “The Umami Strategy: stand out by mixing business with experience design”, a creator of tools supporting designers in the ideation process: Seed Cards and the co-host in the Catching The Next Wave podcast.The traps of standardized measures

Numbers are objective, aren’t they? Or at least they give an impression of objectivity… It is easy to start believing that this objectivity is true. While, in fact, any strategic number is still a subjective representation of a given goal.

Have you ever asked yourself where does the idea of measurement come from? In a very interesting book “The tyranny of metrics” Jerry Z. Muller explains that the main reasons behind our present culture of being metrics-driven is to eliminate the bias of human judgement. Makes sense, right? People are fallible, so let’s find something to secure us from this fallability. We can easily fool ourselves into believing that numbers are objective while people are not. But is it really so that the numbers that serve as KPIs are truly objective? Or do we simply objectivize the subjective data?

The nature of measurement

The obsession with measurement comes from the statement allegedly attributed to Peter Drucker: — “What doesn’t get measured, doesn’t get done.” Such a statement implies that people predominantly respond to incentives. It also means that if an incentive is not attached to a certain goal, people won’t do it. Hence that when things go wrong (in any business sense imaginable) organizations tend to increase the measures as a way to increase punishments (and more rarely awards). In other words, they serve as the proverbial stick-n-carrot.

Often such measures are “dressed up” as tools for increasing transparency. But what often happen is the decrease of trust and motivation among the employees. And here we are in a vicious circle — the less trust and motivation, the tougher metrics leading to even less trust and motivation. This is the moment when the measurement turns from a tool to an obsession, indeed, becoming an end in and of itself.

Obsession with measuring

If you think about it many companies set their goals in terms of financial and performance measures (such and such level of EBIDTA, such and such growth over time, etc.). There are two aspects worthwhile noting here. Obviously such measures reflect the expectations of a certain group of stakeholders (namely, investors). Therefore they are inevitable biased towards the numbers set by these groups. But do they truly reflect the needs of other stakeholders: customers, partners, employees, etc.?

Another crucial aspect is this. Business schools teach generic managers who hold belief that all business is similar, therefore all can be measured (and managed) in a similar manner. As a consequence, many businesses use standardized metrics not realizing that these metrics lead to the market averaging syndrome: the offering of one company becomes the reflection of the offering of any of its competitors. It is a result of the fact that the standardized targets become the entrained pattern of behavious based on the unchanging set of expectations. In other words, people at the organization stop questioning things and keep on doing the same over and over again.

Sure, standardized metrics gives the managers the perception of control of what happens at the company. It gives them the tools to provide extrinsic award and punishment system. It leads to finding the shortest and easiest way to improve the bottom-line. It provokes the thinking: — what can I do to get me the best result? — rather that thinking — what can we do to built the best value?

On the top of all of this, there is one more aspect that is worthwhile noticing: with an obsession of measuring stuff, how much time becomes dedicated to the measurement itself versus to actually doing things? Many managers recognize that they spend more time reporting results than working with their teams to deliver better outcomes. Sounds familiar?

Closing the gap

I would like to discuss one more facet of strategic measurement. Often it looks like this: after a strategic goal(s) are set the company is supposed to do whatever they can to close the gap between today’s state of affairs and the envisioned target. It means that people focus on outputs and outcomes as a way to report progress.

Let’s first look at the difference between outputs and outcomes. Output is defined as: “the quantity or the amount produced”. In other words you can measure it by looking at how many features have been launched, how many tests have been performed or how many promotions have been offered. Measuring anything by output is the best way to get a lot of work done. But not necessarily the important work. Rather the work that is easily measured.

Tom Gilb tells a story of visiting Bell Labs once back in the 1970-ies. The engineers there talked about how they measure quality of service. It has turned out that their measure was to count a number of bugs in 1000 lines of code (a very much output-focused measure). But, if you think about it, is one bug as crucial as the other? This is the output-based measurement.

On the other hand, outcome is typically defined as: “final product or an end result”. In other words, you look at how much money the promotions generated or how many customers the new features attracted. In Tom’s story, the outcome-focused measure would be to check how often the customer fails to be able to make a phone call per 1000 of tries.

Another comparison between outputs and outcomes could be done with sports. An output is the number of actions performed by players. The outcome is the final result of the game. Output measures focus on what you have been doing, outcome measures validate how well you do what you do. They are also easy to measure, however, the question is this: is it important to measure what’s important or what’s easy?

Obviously, neither outputs nor outcomes are to be ignored: they represent what you have been doing. They also help to figure out what should be done better. But the metrics based on outputs and outcomes can be manipulated and gamed. You can see it with colleges lowering the criteria for passing exams to increase the number of graduates, policemen recording crimes as smaller than they were to lower the crime rate or hospitals extending treatment in order to avoid repeated visits.

My absolute most favourite story of cheating the desired outcomes comes from a Polish bottle factory from the early 1980-ies. The factory was making bottles of a first and a second sort. At some point the management decided that there were too many second-sort bottles and assigned financial bonuses as a reward for reducing the number of such bottles. One month in, no more second-sort bottles were produced. Fantastic, right? Bonuses were given, people were happy and the management could only congratulate themselves for a target well set.

A year later a client showed up interested in the second-sort bottles. One of the managers went down to the work-floor to ask his employees what would it take to get the second-sort bottles produced again. The answer was simple: — Nothing. They just needed to stop destroying the second-sort bottles.

What happened was this: once the employees heard they needed to decrease the number of second-sort bottles, they simply started breaking them. Nothing else changed. They reported no more second-sort bottles but was it what the management really wanted? I don’t think so. Yet, it was the easiest way to achieve the expected result. To close the gap. To deliver the desired outcome.

False objectivity

Standardized metrics are often seen as a silver bullet for productivity. Like I mentioned earlier, they offer the false perception of objectivity. They feel as “hard numbers” — something you can rely on. It is so easy to forget that metrics are biased too, particularly those focused on the outcomes. That they are a sort of wishful thinking. That it is easy to fall into a trap of having one group of stakeholders (often investors) to dominate all the actions within the organization leading to the compromises that might not be the intention at all.

Furthermore, standardized measures seem to take a toll on the way the organization functions on every level. They reduce cooperation, as everyone in in it for their own gain. They impact innovative thinking as they make people focus on closing the gap short-term not willing to risk losing a bonus. And, finally, they kill motivation as they leave little space for outside-the-box thinking and doing things that are more risky.

Becoming value-driven

Let me be very clear here: measurement of itself is not a wrong thing. My rant is not about having a measurement in the first place but about making it a goal or a target. It is about discarding the Goodhart’s Law:

“When a measure becomes a target, it ceases to be a good measure.”

It might be even appropriate to paraphrase that law and, following Marylin Strathern, say that: — “When a measure becomes a target, it ceases to be a measure at all.”

The metrics should be set up not to command and control people but to inspire and focus them. It should represent a great vision for the organization and at the same time it should help to define the steps to move towards that vision. It should support both short- and long-term thinking. But without the defined outcome. Having a defined outcome will make employees develop a tunnel vision and miss out on the less obvious opportunities that might pop up along the way.

In other words, a good measurement should focus on value you want to deliver. To all your crucial stakeholders. Not just the investors. Not just the customers. Not just the employees.

___________________________________________________________

Aga Szóstek, PhD is an experience designer with over 19 years of practice in both academic and business world. She is an author of “The Umami Strategy: stand out by mixing business with experience design”, a creator of tools supporting designers in the ideation process: Seed Cards and the co-host in the Catching The Next Wave podcast.