Docs
Search K

Appearance

A/B Testing Guide ​

This document describes how to use and test the A/B testing framework in Open Library.

1. Configuration ​

Active experiments are defined in openlibrary/core/experiments.py under the ACTIVE_EXPERIMENTS dictionary. There are two supported formats:

Simple Configuration (Classic) ​

If you only need to define variant weights, map them directly under the experiment name:

python
ACTIVE_EXPERIMENTS = {
    "AB_Testing": {
        "control": 25,
        "a": 35,
        "b": 35,
    }
}

Advanced Configuration (Modern) ​

If you need additional rules (such as limiting the experiment only to logged-in users), structure the variants under a "variants" key and add options alongside it:

python
ACTIVE_EXPERIMENTS = {
    "AB_Testing": {
        "variants": {
            "control": 25,
            "a": 35,
            "b": 35,
        },
        "authorized_only": True,  # When True, guest/unauthorized users are automatically put into the control group
    }
}

2. Usage Examples ​

Inside HTML Templates / Templetor Macros ​

To conditionally render markup based on a user's variant bucket:

templetor
$code:
    variant = ctx.get('experiments', {}).get('AB_Testing')

$if variant == 'a':
    <div class="layout-a">
        <!-- Variant A layout -->
    </div>
$elif variant == 'b':
    <div class="layout-b">
        <!-- Variant B layout -->
    </div>
$else:
    <div class="layout-control">
        <!-- Control layout -->
    </div>

Inside Python Controllers / Handlers ​

To conditionally execute backend logic (works in both web.py and FastAPI routes):

python
import web

variant = web.ctx.get("experiments", {}).get("AB_Testing")
if variant == "a":
    # Execute Variant A logic
    pass
elif variant == "b":
    # Execute Variant B logic
    pass
else:
    # Execute Control logic
    pass

Inside Frontend JS (jQuery, Vue, Lit) ​

To check variants client-side, query the global window.getExperiment helper or import it:

javascript
// Anywhere in your scripts:
const variant = window.getExperiment("AB_Testing");
if (variant === "a") {
  // Enable variant A layout
} else if (variant === "b") {
  // Enable variant B layout
} else {
  // Enable control layout
}

// Or import as an ES module in modern bundles:
import { getExperiment } from "./experiments";
const variant = getExperiment("AB_Testing");

3. Testing & URL Overrides ​

For development, QA, and debugging, you can force a specific experiment variant using URL query parameters.

The format is: ?experiment_[experiment_name]=[variant]

Examples ​

  • To force Variant b of the AB_Testing experiment: http://localhost:8080/?experiment_AB_Testing=b
  • To force the control group: http://localhost:8080/?experiment_AB_Testing=control
  • Overrides also work across multiple parameters: http://localhost:8080/?experiment_AB_Testing=a&experiment_AnotherTest=control

NOTE

URL overrides will only apply if the specified variant name exists in the configuration of the active experiment. Invalid override variants or incorrectly formatted parameters are safely ignored, falling back to the user's deterministically assigned bucket.