Why in many cases micro-metrics are more important than macro metrics
If you validate all experiments by macro metrics (conversions, average check, signups, searches, etc.), then you can miss valuable insights. Not every change in your product can be described by a “big” metric that shows the success of your business. This article is based on our own experience working on some projects in the past.
The goal of the article is to share our experience and vision why in some situations the micro-metrics are more important than macro metrics
Let’s imagine that we decided to test our website search. We decided to make changes in the algorithm which shows search suggestions to users when they start typing the search query. This change is not too noticeable for the users.
Our macro metric is — the share of transactions made by users using the site search.
Analyzing the experiment data, we noticed that there is no statistical difference between the control and experimental groups.
This could be the end of the data analysis and the article, but let’s dive deeper into what this change might also affect:
- Users are more likely to use the suggestions because they are more relevant to their search
- The time between typing a search query and browsing search results reduced
- Returning users are using search more frequently
As you can see, now we have three new micro metrics that are not related to product economic indicators. Not every change in the product aimed at maximizing profits. User experience is also extremely important and good UX may affect repeating purchases, user loyalty and many more.
Metric Priority Search retention rate: priority — 1
The time between query and browsing search results: priority — 2
Suggestions usage: priority — 2
Revenue: priority — 4
1 — Highest priority, 4 — Lowest priority
Revenue has the lowest priority because the Revenue metric is not relevant to this experiment. The goal of this study is to optimize a user experience.
Let’s analyze one of the micro metrics — The time between query and browsing search results.
It’s obvious, that the type of distribution for the data is not normal. After applying the nonparametric Mann-Whitney criterion, we get a p-value = 0.003.
Test variant B time decreased by 30%. We can conclude that the new search is more convenient for users and they switch from query to results quicker.
Product growth is not limited to pushing up macro metrics. if you focus on the “big” macro metrics, you can miss valuable insights as well as a large number of great hypotheses. Looking deeper into the metrics helps you better understand your users and their behavior.
Originally published at awsmd.com.
