A/B testing to drive product adoption
The power of running business experiments
In 2018, while at KPIT, we shipped a mobile app designed to improve the safety of heavy commercial vehicle drivers. As the product manager, one of key metrics I was tracking was the average number of traffic violations. This was important as it had a direct bearing on the safety of the ecosystem, which was our key value proposition. To reduce the avg. number of traffic violations, we tried out a few experiments. One of the experiments involved a segment of drivers viewing a small clip about the ensuing journey before they began driving. The clip would highlight points in the journey where the traffic could be expected, locations of toll booths, rest stops, fuel stops and points where the route was particularly risky. We observed that for the drivers who viewed the short video before setting out, the 30-day traffic violation rate came down by a whopping 60%. This simple A/B test gave us insights that helped to increase product adoption.
A/B testing is an experiment done to determine whether a new product design (UX, messaging, feature, etc.) can bring about an improvement in a metric that translates into product success. Different variations of the product are released to different cohorts of users and the groups are then monitored for impact on the chosen metric. Here is how Netflix, a strong proponent of A/B testing, defines the process:
“The general concept behind A/B testing is to create an experiment with a control group and one or more experimental groups (called “cells” within Netflix) which receive alternative treatments. The different cells are then monitored to determine which treatment (design) is most successful in moving the needle on the chosen metric. The specific metrics could be streaming hours or user retention.
How can companies democratize A/B testing
Controlled A/B testing is a great tool to curate better customer experience. However, companies often shy away from experimentation (if ain’t broken, why fix it?) and even the ones that do experiment, typically run maybe a few dozen tests a year rather than few thousands of tests. Instead, if a company were to experiment more often, it would increase the likelihood of learning something new about its user behaviour. A few handful of experiments may not lead to any insights and the company may incorrectly conclude that A/B testing is not useful.
The HiPPO effect
Employees avoid testing ideas thinking they may fail and it would be seen as a wastage of resource. This can lead to people deferring to the Highest Paid Person’s Opinion. To democratize testing, it is vital that data should trump gut feel. All decisions should be backed by data and not based on personal opinion alone. Systems and resources should be put in place to facilitate experimentation. Anyone should be able to conduct a test not just data analysts and product managers. At Facebook, all new employees are trained on the so that everyone is aware of the nature of data available, how to access it and use it to test a hypothesis.
A/B testing at Booking.com
Booking.com is famous for being one of the most A/B tested sites on the planet. They typically have billions of landing page variations live. That essentially means, more landing page combinations than there are humans on the planet! Even two customers in the same city are unlikely to see the same landing page. Booking.com runs minor, isolated tests on literally every element on their website. This allows them to measure the impact of each individual element and how users engage with that element.
Good design at Booking.com isn’t just about making something that seems to be better. It’s about measuring results, understanding the impact on the full customer experience, and actually making things better.
The rigorous testing ensures that empirical evidence helps guide product design choices. It is clear that Booking.com has facilitated a culture of experimentation pan organisation. Any employee can launch an experiment, which could potentially impact millions of customers, without seeking manager approval. If the experiment doesn’t work, then it isn’t looked at as a failure but rather an iteration away from a more conclusive experiment. To reduce the possibility of experiments overlapping, a central search repository is maintained which has a history of all past experiments, the user base and the learnings gained. Thus, while there is a centralization of knowledge, the testing itself is decentralized.
“If it can be a test, test it. If we can’t test it, we probably don’t do it.” — Stuart Frisby of Booking.com
It would be erroneous to think that a majority to the experiments carried out are successful. Infact, only 10% of experiments lead to product design changes. Consider the following two examples.
Hypothesis: A more personalised message (‘B’ mentioning the name of the proprietor) would get more customers to leave a review.
Result: ‘B’ saw an increase in the amount of reviews by 20%. Product design change.
Hypothesis: The classic ‘hamburger icon’ in test A would see an increased click-through-rate.
Result: Both ‘A’ and ‘B’ saw similar levels of user interaction. No product design change.
It’s not just the technology companies that benefit from A/B testing. Even companies without digital roots regularly turn to product experiments to continually improve their service.
Experimentation at Walmart
Walmart serves more than 250 million customers across 27 countries. Even a small change in the either the website or the store layout could potentially impact millions of customers. The experimentation team at Walmart has built a culture of experimentation across the company. They use experimentation to identify design choices, customer discounts and product recommendations. Store Number Eight at Walmart was famous as the store where founder Sam Walton would try out new ideas. Walmart conducts more than a hundred tests per month for its online customers through its ‘Expo’ platform designed for A/B testing. These online and offline experiments have helped Walmart to improve the experience for its legion of customers.
Things to watch out for when A/B testing
While A/B testing can undoubtedly drive product adoption and engagement, there are a few pitfalls that should be kept in mind.
Short duration of experiment. For A/B testing to be successful, testing has to be run for a sufficient period of time. A change in user interface may show an initial increase in user engagement but over a period of time may actually reveal less desirable behaviour. In the case of a financial planning product, customers may not use the app on a weekly or monthly basis. Thus, restricting the study to a few weeks may not accurately gauge the impact on the entire market.
Testing only for a certain segment. An A/B test may show a positive reaction across one customer segment but may totally turn away another segment. At times, this may be intentional — you may have consciously chosen to optimize only for a particular segment. But otherwise, A/B testing should be designed to capture the value for the different segment of customers. This can also help companies come up with product innovation for different market segments. Netflix brought out a mobile-only plan for India, a market where a majority of customers consume content on mobile platforms.
As the joke goes, A/B testing actually stands for ‘Always Be Testing.’ In today’s digital age, well thought out product experiments should be the standard operating procedure. A/B testing can help anchor product design and strategy on solid data rather than them being a function of intuition and guided by hierarchy. Using A/B testing, organisations can iterate rapidly and if not win then fail early and pivot.
Experiment fast, fail often and believe in data. For data never lies. — Maria Klokner
I hope this article was helpful. If it was, feel free to follow me on Twitter where I share thoughts and articles on product management and leadership. You can also check out my website https://komenco.in/ for more such articles.
