A/B Testing

A/B testing is a well-known method to compare different versions of an offering and test which version or features work best. Typically, different offer variants are shown to groups belonging to the same customer segment and then compared according to their appeal.

A/B testing is an appealing experiment to test a large variety of different solution-related hypotheses; while the most widespread use is to optimize your website or newsletter for conversion, this test format can also be used to analyze the appeal of different value propositions or offer compositions. In this format, your test group is split into two randomized groups. Each group receives one variant of what you want to test, combined with a call to action that they need to follow to show their interest. At the end of the test, you analyze the effectiveness of each variant to convert test participants to follow the call to action.

Helpful Tips

Get a large enough group: A/B testing is a statistical experimentation tool. For it to give you significant answers, the pool of participants needs to be large enough. To understand what the sample size needs to be, you have to set a baseline conversion rate and a minimum detectable effect, meaning the minimum improvement you are willing to detect with your test. Use this tool to calculate what that means in your specific case.

Segment your respondents: When doing an A/B test, try to gather as much information as possible on your respondents. You can use this data to segment after the test and see, for which segments a variant worked particularly well. This helps you to get a better understanding of your results and to identify early adopters

Make it simple: In order to understand what triggered the change in conversion, the two variants need to be either completely different or very similar. Otherwise, there will be ambiguity as to why the conversion changed.

Combine with other formats: An A/B test needs to be used together with other test formats, such as Landing Pages, (online) Flyers, Prototypes or (Blog) Posts. To get the right traction and find test participants, you can combine it with Online Advertisements.

Kiva.org, a non-profit lending company, conducted an A/B test to improve the number of donations from first-time visitors. They A/B tested their website with and without additional information about the borrower and found out that donations increased by 11.5% when they displayed this additional information.

How to Guide

Prepare the options you want to test. Identify and recruit participants. Set up success measures for the interactions.

Split your participants into two groups and let each group see and use only one variant (either online or in person with prototypes). Measure the success and feedback during the test.