Split Testing - A/B Testing

The easiest path to validated learning

delivery-measure-and-learn loop position

No. People

2+

Time

Several days

Difficulty

moderate

Participants

Product Owner, Data Scientist, UX Designer, DevOps Team

This practice is part of: Product Ownership.

What is it?

This is a randomised experiment in which we compare and evaluate the performance of different versions of a product in pairs. Both product versions are available in production (live) and randomly provided to different users. Data is being collected about the traffic, interaction, time spent and other relevant metrics, which will be used to judge the effectiveness of the two different versions based on the change in user’s behaviour. The test determines which version is performing better in terms of the target outcomes you have started with.

You can compare two or more different versions of an application/product as long as you always do that in pairs. The pairs need to be identical except for a single characteristic which will vary between the two versions.

The team should take care with regards to:

  • Differentiating the users, e.g. new users from returning users (cohorts).
  • Comparing results - the results may not always point a significant difference between the versions and may either need to be run longer or different combinations of pairs should be used
  • Running the test always simultaneously as multiple factors may vary with time and influence the results

Why use it?

Simple to apply, fast to execute and often conclusions can be made simply by comparing the conversion/activity data between the two versions. It can be limiting as the two versions should not differ too much and more significant changes in the product may require a large number of A/B tests to be performed.

This is one of the practices that allow you to “tune the engine” as per The Lean Startup by Eric Ries.

The practice provides an objective criteria for important decisions on the product design, features, behaviour, etc. It is used as a way to enhance the product performance, the user experience and the results the product produce for your organization.

Split Testing - Multivari Testing

Design of Experiments

Further information

The Ultimate Guide to A/B Testing, Paras Chopra, Smashing Magazine

The ABC of A/B Testing, Cara Olson, Marketing Land

Blue-green Deployments, A/B Testing, and Canary Releases

Improve this practice
View all practices