Image by Author | Ideogram

A/B testing is a solidly established experimentation method to compare two versions of a system or variable, such as a computer algorithm, a predictive analysis model, or even the design of a website and a personalized advertising campaign. The two versions are compared in terms of any chosen metrics or performance indicators depending on the specific domain. The significance of A/B testing in decision-making lies in its ability to enable data-driven evaluations.

This article provides a concise guide to understanding A/B testing key concepts and applications.
 

A/B Testing Concepts

A/B testing can be defined as a controlled experiment in which two versions of a variable, algorithm, or system, are compared against each other. The goal is quite simple: to determine which performs better based on a predefined metric. A/B testing is fundamental behind the process of fully launching new versions of major corporate websites, mobile apps, forecasting models, and so on, and it helps ensure newer versions of a system, etc., are only released if they perform as appropriately as (or even better than) the currently live version.

The central mechanism behind A/B testing lies, therefore, in considering two core versions of the system or aspect being benchmarked: the control version (or control group) vs. the variant version (or variant group).

• The control group can be deemed as the baseline condition or “stable version”, that is, the existing version of the product or systems users are being exposed to, before considering further modifications into a newer version. Importantly, having a well-defined control group is the first step towards ensuring that any observed shifts in user behavior may be attributed to system modifications rather than external factors.
• Meanwhile, the variant version constitutes the updated version that includes the intended changes at the experimental phase. These could include a redesigned user interface, a new product recommendation algorithm, or a novel pricing strategy for a hotel booking platform.

With these two versions at hand, we are ready to understand the key part of A/B testing: randomly assign users to either one of the two defined groups (control group vs. variant group), which is typically done by redirecting a percentage of the total of users to the variant group. We then measure how the changes significantly impact performance metrics or indicators, e.g. engagement levels, prediction accuracies, conversion rates, revenue, and so on. How to measure whether the changes in performance among versions are significant? I bet you know the answer, as it is everywhere on the blog website you are reading right now: statistical hypothesis testing.

Let me give you a practical example: in an e-commerce scenario, an A/B test could be undertaken to compare two layouts of a product page on the online store website: one with a standard “Buy Now” button (control version) and another with a more prominent button that looks more visually flashy (variant version). If measured metrics for users routed to the variant group lead to a significantly higher purchase rate, then the company should make the decision to implement this newer product page version, leaving the current one behind.

Wrapping Up

A summary of best practices for performing A/B testing is:

• Ensure proper design of experiments.
• Do not peek at results too early: wait until a sufficient number of user interactions from groups are completed.
• Be cautious when interpreting findings, avoiding instinct-based conclusions.
• Apply the testing process iteratively for continuous system improvement.

Other salient applications of A/B testing, besides e-commerce and website design, include ad performance analysis in marketing, experimentation on new features in mobile apps, comparing machine learning models, and clinical trials in healthcare.