Five uncomfortable truths about A/B testing (and what you can do about them)

Photo by Rodolfo Clix from Pexels

Do you manage the testing and optimization program for a big company or are you looking to start one? Here are five obstacles you’ll likely encounter and how you can move past them to build a thriving optimization program for your organization.

1. You’ll have more losing tests than winners.

Photo by Stephen Dawson on Unsplash

…and that’s okay. The best part about a/b testing is what you learn from each experiment. The insights you are able to draw from each test you run are extremely valuable. Over time, you’ll start to learn more and more about how your customers think and why they buy. That’s where the real magic happens!

Solution: Increase your success rate by creating tests based upon problems you’ve observed and data you have collected. If you’re hard-pressed to find purposeful a/b test ideas, here’s 10 ways you can generate more a/b test ideas.

2. There will be winning treatments that you absolutely hate.

Photo by icons8.

If you are collaborating with several designers on each a/b test, it’s inevitable that there will be designs and copy that you may not feel strongly about. Perhaps it was something a coworker designed or wrote and it feels as though it isn’t as compelling as another test candidate. Unfortunately, this is the nature of a/b testing.

You have to keep in mind that you aren’t in the business of proving who is right and who is wrong. Rather, you are in the business of finding out how to get customers to open their wallets more often, and perhaps you’re trying to get them to spend more too. A/b testing is merely a means to an end.

Solution: Don’t let yourself get emotionally attached to your experiments. Remember the reasons why you’re testing in the first place:

• You’re trying to learn more about what makes your customers tick
• You’re trying to identify and take advantage of new revenue opportunities with the traffic you already have
• You’re trying to identify any areas where you are leaking revenue

3. There will be times when you are approached to run a test with a weak hypothesis… or none at all.

This is more typical when you have people involved who haven’t spent the time learning about the correct way to conduct an a/b test… and that’s okay.

Solution: Instead of shutting out an idea completely, take this opportunity to educate them on how to form a proper test hypothesis and emphasize why it’s important to have one.

In case you aren’t familiar yourself, there are three parts to a solid a/b test hypothesis. The template below is how you’ll structure the statement for each and every test you prepare and launch.

“If [the variable], then [the result], because [the rationale].”

The variable is the part of the website (copy, design element, imagery, etc) being added, edited, or removed to reach the outcome you’re seeking.

Examples of variables:

• Headlines
• Copy
• User interface elements (CTA buttons, navigation elements, etc)
• Hero/callout banners
• Imagery

The result is the outcome you’re predicting. This is important because it’s comprised of the metric(s) you will use to measure the success of the experiment.

Examples of results:

• More conversions
• More signups
• More emails collected
• Increased average order value
• Increased revenue per visitor

The rationale is all of the research you spent time gathering and why you feel strongly about running the test in the first place. It serves to validate your hypothesis. The best way to go about this is to provide any data you have observed and analyzed to support your hypothesis.

You can fuel your rationale with data from:

• Qualitative/quantitative user tests
• Heatmaps
• Scrollmaps
• User session recordings
• Surveys and polls

4. Tests will break at times... especially if you’re testing from within a large organization.

Photo by Markus Spiske on Unsplash

If you run a website that is supported by a large software development team and there are constant deployments or changes happening to the website, you’ll probably have a few tests break. The good news is this can be avoided, but you’ll have to be proactive in communicating with the teams that touch the code base.

Solution: Communicate planned a/b test launches well in advance with the engineering leadership in your company. Try to plan launches on days where deployments aren’t happening or see if “code deployment freezes” can be agreed upon in regular intervals.

If this is unavoidable, just make sure you know and document what’s changing and where, so you can adjust your test if it becomes necessary. And make sure you keep an eye on your tests constantly to make sure they are still functioning as expected. You don’t want a broken test to impair the user experience!

5. Despite all of the data and winning tests in the world, you’ll still have those few that can’t leave their biased opinions and gut feelings out of the decision-making process.

Us CROs call them HiPPOs (AKA Highest Paid Person’s Opinion or the Highest Paid Person in the Organization). A HiPPO tends to lead with the gut. You know who they are… they seem to know what’s best because they’ve been “doing this stuff for years.” Or they think they the winning test candidate isn’t the right direction to go, because “none of the competitors are doing it.” Their unwillingness to try something different can leave dollars on the table, even when data supports your case.

Solution: Author and Digital Marketing Evangelist of Google, Avinash Kaushik, wrote about this very topic and suggested six strategies for handling these HiPPOs:

1. Create an experimentation/testing program. Hey, whaddya know? If you’re reading this, you’re probably already there… or you’ve at least contemplated the idea!
2. Capture the “voice of the customer.” Remember when I mentioned using surveys, polls, and other qualitative methods? This is the kind of data that you should be gathering to form your hypothesis for tests.
3. Create benchmarks against your competition. You need to know where you stand among the biggest players in your marketplace.
4. Leverage competitive intelligence. Find where your competitors are at their weakest and use this information to support your cause.
5. Hijack a “friendly” website. Convince a friend or coworker to let you optimize a side project or personal website as a test case, and use any data and findings as support.
6. Hit up an industry expert. These guys lead the industry for a reason… and there’s a greater likelihood that they’ve dealt with the very same types of people.

If you want to view the full article, you can do so here. I highly recommend giving it a read if you’re facing similar problems.

It also helps if you can take their perspective. The executives at the top are held to a very high standard and face a lot of pressure; they are expected to perform with extreme precision and are ultimately held accountable for the direction a company goes. If you want to win their support, you should make it a point to provide as much customer data and research to support everything you do.

Thankfully, not all leaders fit the HiPPO profile. Many today are very data-driven, open, and receptive to taking calculated risks for the chance at more conversions and greater revenues.

Closing Thoughts

Despite the difficulties in kickstarting and managing a testing program, it can be a powerful and incredibly lucrative part of your organization. When performed correctly, a/b testing and conversion optimization can have a tremendous impact on your organization’s ability to: 1) learn more about your customers, 2) increase conversions, 3) capture more leads, and 4) generate greater revenues. Once you start to see growth from your conversion optimization efforts, you won’t look back!