Skip to content

Experiments and the bandit

Targeting, assignment and traffic allocation, and how to read the results: probability of being best, expected loss and the sample-ratio check.

What an experiment is#

An experiment compares the page as it is — the control, created automatically and called “Original” — with one or more variants, on the visitors and pages you choose, against a primary goal.

StatusWhat visitors get
DraftNothing yet. Variants can be edited freely and previewed.
RunningEligible visitors are assigned and measured.
PausedEveryone sees the original page. Collected data is kept, and visitors exposed earlier can still convert.
CompletedThe experiment has ended. If a variant was promoted as the winner, it is served to everyone the experiment targeted until the experiment is archived.
ArchivedNothing. The results stay available.

While an experiment is running or paused, its key and its allocation mode are frozen, so the meaning of the collected data cannot change underneath it.

Targeting#

Page rules decide where an experiment runs. A URL must match at least one include rule and no exclude rule; an experiment without include rules runs on every page.

MatchMatches whenExample
exactThe URL equals the value. The query string is ignored unless the value has one./pricing
prefixThe URL starts with the value./blog/
containsThe URL includes the value.utm_campaign=spring
glob* matches within one path segment, ** across segments, ? one character./products/*/reviews
regexThe regular expression matches the URL.^example\.com/(de|fr)/
  • A value that starts with / is compared with the path. Any other value is compared with the whole URL without its scheme and without www. (example.com/pricing).
  • Matching is case-insensitive, except for regular expressions. Trailing slashes do not matter: /blog/ and /blog are the same page.

Audience rules narrow who is eligible. Every condition you set must hold: device class, new or returning visitors, UTM source, medium and campaign, referrer host, query-string parameters, and country. Country is only known on the server, so it applies to assignment through the server SDK.

Traffic is the share of eligible visitors who enter the experiment at all, up to 100 percent. The rest see the original page and are not counted.

How visitors are assigned#

Assignment is deterministic. The visitor's id and a seed that is unique to the experiment are hashed to a number; that number first decides whether the visitor is inside the traffic share, then which variant they get according to the current weights. The same visitor always lands in the same place, in the browser and on the server, without any lookup.

The result is then sticky: it is remembered in the visitor's browser (the most recent 50 experiments), and the first variant a visitor was exposed to is also what the server records. When weights change later — which the bandit does constantly — visitors who already have a variant keep it. Only new visitors follow the new weights.

Because each experiment has its own seed, being in variant B of one experiment says nothing about which variant a visitor gets in another.

Fixed allocation and the bandit#

Fixed
Weights stay as you set them — an even split unless you change it — for the life of the experiment. This is the classic A/B test: the cleanest measurement, at the cost of sending a full share of traffic to a losing variant until you stop it.
Bandit
The experiment starts with an even split, and Avenue Lift then re-weights it on a schedule so that more new visitors get the variants that are converting. You give up less to losers while the experiment runs. Included in the Starter, Growth and Agency plans.

The bandit, plainly

Think of each variant's true conversion rate as unknown, with a range of plausible values that narrows as data comes in. The bandit gives each variant a share of new traffic equal to the probability that it is the best one — the same “P(best)” you see in the results. A variant that is probably best gets most of the traffic; one that is probably worst gets little; while nobody knows, traffic stays spread out. This method is called Thompson sampling.

Four safeguards keep it honest:

  • Warm-up. The split stays even until the experiment has 100 visitors per variant and 10 conversions in total. Early noise does not get to steer traffic.
  • Exploration floor. Every live variant keeps at least 5% of new traffic, so a variant that had an unlucky start can still recover.
  • Control floor. The original page keeps at least 10%. Every lift is measured against it, and a starved control makes every comparison noisy.
  • No churn. A re-weighting that would move no variant by at least 1 percentage point is skipped.

The bandit only looks at the primary goal. For a revenue goal it is the goal's conversions that drive the split.

Reading the results#

Results are computed per goal, with unique visitors as the unit. The statistics are Bayesian: instead of a p-value they answer the questions you actually have, as probabilities you can read directly.

Visitors and conversions
Unique visitors exposed to the variant, and how many of them converted on the goal at least once.
Conversion rate
Conversions divided by visitors, with a 95% credible interval: given the data, the true rate lies inside that range with 95% probability. Wide intervals mean little data.
Lift
The relative improvement over the control — +10% means the variant's rate is one tenth higher than the control's — also with an interval. An interval that includes zero means the variant may still be worse.
Chance to beat control
The probability that this variant’s true rate is higher than the control’s.

P(best): probability of being best

The probability that this variant has the highest true conversion rate of all variants in the experiment, the control included. Across the variants it adds up to 100%.

It is computed by simulation: from what has been observed, Avenue Lift draws 10,000 plausible sets of true conversion rates and counts how often each variant comes out on top. A variant at 80% won 8 of every 10 of those simulated worlds.

  • With two variants, P(best) of the challenger is the same as its chance to beat control. With more, it is stricter: the variant has to beat everyone.
  • It says how likely a variant is to be best — not by how much. A variant can be 95% likely to be best while being better by a hair. That is what expected loss is for.
  • The same counts always give the same numbers. Results do not jitter when you reload the page.

Expected loss: what you risk by choosing it

Suppose you ship this variant today and it turns out another one was truly better. Expected loss is how much conversion rate you would give up, averaged over all the simulated worlds — counting zero for the worlds where this variant really was best. It is expressed in percentage points of conversion rate.

An expected loss of 0.05% reads: “if I pick this variant and I am wrong, I expect to lose about 0.05 percentage points of conversion rate, on average.” Small enough numbers mean the decision is safe even if it is not certain: the variants are either clearly ranked, or so close that choosing the wrong one costs next to nothing.

Read the two together

P(best) tells you how sure you can be. Expected loss tells you how much it matters. A winner is recommended only when both agree: high probability of being best and little to lose.

Revenue goals

For a revenue goal the metric is revenue per visitor: total value divided by all exposed visitors, converting or not. P(best) is then the probability of having the highest revenue per visitor, and the recommendation follows it.

When a winner is called#

Every result carries a recommendation. These are the rules, in order, with their default thresholds:

SituationRecommendation
Any variant has fewer than 500 visitors or fewer than 10 conversionsKeep running — not enough data to say anything.
The sample-ratio check fails (fixed allocation)Keep running, and investigate. Nothing is promoted on numbers that cannot be trusted.
A variant reaches P(best) of 95% with an expected loss of at most 0.1%Promote that variant.
The control reaches P(best) of 95%Stop: the original page wins.
Every variant is below a P(best) of 1%Stop: no variant can plausibly beat the original.
Anything elseKeep running — no clear winner yet.

Independently of the overall recommendation, an individual variant whose P(best) falls below 1% is flagged as a clear loser that can be stopped while the others continue.

The visitor minimum and the win probability can be changed per site, in the autopilot settings. Whether anything acts on a recommendation without you is up to autopilot.

Promoting a winner

Promoting ends the experiment and, if the winner is a variant, keeps serving it to everyone the experiment targeted. This is a bridge: it holds the improvement in place until the change is built into your site, after which you archive the experiment. You can also end an experiment in favour of the original, or without a conclusion.

The sample-ratio check#

If you ask for a 50/50 split and get 5,000 visitors on one side and 4,300 on the other, something is broken — and whatever broke the split has probably also skewed the conversion numbers. This is called a sample-ratio mismatch.

Avenue Lift compares the visitors each variant actually received with what the configured weights predict. When the difference is larger than chance can reasonably explain (p below 0.001), the experiment is flagged, no recommendation is made, and autopilot will not act on it. The check runs for fixed allocation; with the bandit the traffic shares move on purpose, so there is no constant split to compare against.

Usual causes, roughly in order of frequency:

  • A variant that redirects, or breaks the page, before the visit is recorded.
  • The snippet missing on some of the targeted pages.
  • Caching or bot traffic that reaches one variant more than another.
  • A variant paused or changed midway through the experiment.

Segments#

Results can be broken down by device, country, referrer, UTM source, and new versus returning visitors. Use segments to understand a result, not to find one: with enough slices, some will look like winners by chance alone. A segment finding is a hypothesis for the next experiment, targeted at that audience.