How to overcome product local maximum

When you have a new product, you can find a lot of significant hypotheses to test and grow your product. However, with every passing month and year, you will soon realize that you must test more and more specific and even minor hypotheses, especially if the growth of your business slows down significantly.

In this article, we will review how to determine the moment when your business is in need of crucial changes at which to take that inevitable leap of growth. Let’s look at a popular product, booking.com as an example and work through it as a case study.

Booking.com product has existed since 1996, and I am sure that those who work for the company test a lot of hypotheses every day. We assume that booking.com team launch around 300 different experiments daily. With such a mature product it could be difficult to find something new and original that would provide additional and substantial growth. With a product like booking.com, in general, even a small increase such as a hundredth of one percent in global metrics and also micro-indicators can be meaningful.

Let’s imagine that booking.com team decided to make significant changes in the design or even launch a completely updated product. For the purposes of this case study, we will describe how we would approach this process.

Step 1 — Quantitative analysis

It is essential to understand how a product works and where its weaknesses lie. First of all, we would explore the product funnels for each segment to learn where the product pain-points are. Next, it is just important to look at the user journey to determine any non-obvious behavior scenarios. This would make it possible to understand what funnel steps can be optimized and whether the core audience would be hurt with the new update.

Also, we would machine learning classification algorithms, such as a decision tree and random forest algorithm, to name a few. This would give us an understanding of the most significant factors affecting the key metrics in various segments. We will review how we use machine learning algorithms for UX analysis in future articles.

At this stage, we can now identify the problems and develop hypotheses, but we would not immediately get an answer as to what causes these problems in the product.

Step 2 — Qualitative user research

On the previous step, we received data indicating where the product has problems. At this next stage, we need to get qualitative feedback from users and learn what exactly causes most of the issues.

We would choose the following scenario for user research:

Phase 1 — Moderated, in-person Usability Research

Having conducted a series of studies with different types of users, we will have successfully gained valuable insights into what exactly the users found to be incomprehensible and unusable about the product. Based on our analysis of the quantitative and qualitative data, we now would form our primary hypotheses.

Phase 2 — Remote Testing

After the first phase, it makes sense to make prototypes and run remote usability tests, where users will be able to complete tasks and give feedback without a moderator’s supervision. The pros of these types of usability studies are that they allow us to collect a generous amount of respondents in a short period, as well as quickly and cheaply test the prototype.

Step 3 — A/B Testing

Before implementing any new design or code, we would run experiments to test the boldest hypotheses. Moreover, we would first look at the segment of “loyal” customers, because, most likely, they form a fundamental part of the product economy. Of course, we would not forget about the new users, too, but the main point here is that we should analyze these two groups separately.

In the results of the study, we expect to get the following insights:

• What hypotheses make a negative impact. If the obvious to us hypothesis in the first and second phases showed a negative impact, it makes sense to return to step 2 and chat with users again.
• We would learn the hypotheses that do not impact users at all — if a complex hypothesis somehow does not affect the audience, this in itself is a reason to abandon it. The implementation and support of such hypotheses could potentially be quite expensive, and it would not benefit the business.
• A solution that would reflect positively on the users and increase business metrics.

The process of creating and developing a product requires unbiased attention. The product is not someone’s personal ambition; it is what secures company growth. If the manager or designer bases decisions on opinions or tastes, while ignoring audience insights, the outcome most likely will be disappointing. Consequently, data and research are always needed, and the method outlined above will help create a truly useful and highly-demanded product.

Originally published at awsmd.com.