Why and how to run a fake door test — a UX case study

Martina Augustin
UX Collective
Published in
4 min readJan 1, 2020
Photo courtesy of pixabay.com

Have you ever clicked on an option and only got a message saying “Sorry, this feature is not available yet?”. Did it perhaps upset you and you wondered why the option has been displayed when it’s not active? You probably came across what is known as fake door test. In this post, I’ll show you how we used this method in Notino. We wanted to test whether to invest in the costly implementation of the multi-language website.

Notino is the largest beauty-specialist online retailer in Europe. We currently have online stores in 24 countries and we are rapidly expanding to new markets. In 9 countries you can spoil yourself in our luxurious perfumeries. Yet the majority of orders come from the websites, so we invest a lot of effort in their optimization.

Notino UX team
We’re shy people.

The question

You can find a localized Notino website in most of the countries we’re doing business to. In summer 2019 we entered the Swiss market with notino.ch localized in German and the sales were satisfying. However, Switzerland is a country with multiple official languages. So we wondered, could we do even better with the website that would serve more language variants?

Insights from Google Data Studio

At first, we did a little bit of desk research and tried to look at the data we already had. We created a report in Google Data Studio sourced from Google Analytics showing

  • how many users visit notino.ch
  • what is the language variant of their browser
  • what is the count of purchases and conversion rate.
Google Data Studio report showing a count of conversions and conversion rates from various browser language variants.
Count of purchases and conversion rate — September 2019

The data show that people with other browser language variants were not a majority and their conversion rate did not vary much. Before dropping the idea of multi-language website we decided to give it one more shot — with a fake door test.

Sprichst du Französisch oder Italienisch?

With the fake door test, you can measure the interest in a product or feature without actually coding it. You create the visual part of the feature and show it to the customers, but the feature is not active. In our case, it meant having a language variant switcher on the website. We had the website in German, so the options in the select box were French and Italian. Since the Romansh population is only about 0.5% we omitted it from the test. As KPIs, we wanted to measure how many people click on the switcher and which variant they choose. And whether this action has an impact on purchase conversion rate.

How to prevent people from being upset

One of the best practices when designing a fake door test is to avoid leaving the user frustrated or upset. Having this in mind, we considered a pop up with a purchase discount displayed to anybody who would click on the switcher. But we trashed the idea soon. We were afraid that people would just click on the switcher to get the discount.

We used rapid ideation to quickly brainstorm what to display to the user. The beauty of the technique is to generate as many ideas as possible during a short time. We used dot voting to choose the right idea. As you can see in the picture below, we decided to tell the truth.

Here you can see the final design that we put to the test.

The fake door test we run in Switzerland

Here’s the original version of the modal window we had translated to French and Italian.

Modal window with the message

And after two weeks…

The whole test was implemented via Weblayers in Exponea and we used Exponea funnels for checking the results. We let the test run for two weeks as the traffic on the website was pretty high.

So what was the demand for other language variants?

Exponea funnels with the results of the fake door test
7.10.2019–21.10.2019

Nothing special, right? In total, the click rate of the switcher was just 3,5%. You might notice that the sum of the clicks on French and Italian doesn’t equal clicks on the switcher. It’s because some people might have clicked on the switcher only to see the options, but then didn’t choose any language. Also, it seems that not having a preferred language variant doesn’t stop people from buying. The purchase conversion rate after the fake door test was even slightly higher than the average.

The fake door test is a fast and cost-effective method of how to measure peoples’ interest in your new feature or product. Thanks to this test we found out that adding more language variants to notino.ch would not scale up the number of conversions. Preparing and launching the test took us only about a day and a half. Comparing to the preparation of a multi-language website, it was time well invested.

Have you ever run a fake door test? Share the findings with us in the comments!

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Published in UX Collective

We believe designers are thinkers as much as they are makers. Curated stories on UX, Visual & Product Design. https://linktr.ee/uxc

Responses (2)

What are your thoughts?

Focus on the problem instead of the solution is vital, and it is definitely a matter of empathy.
It's also a matter of business. Companies don't set aside budgets for solutions. Solutions have no inherit value. Instead, companies have budgets to fix…

--

I think you’ve got minimalism a bit wrong. But great post!

--