Analysis
Reading results
Conversion rate, lift, 95% intervals, p-values, significance, sample size and why you should not stop early.
Open an experiment and select the Results tab. You see a summary, a table with one row per variation, and a chart of each variation's 95% interval. The dashboard combines all environments. To see one environment only, call the results API with ?environment=prod (a filter in the dashboard is coming later).
The results table#
| Column | Meaning |
|---|---|
| Visitors | Unique visitors with at least one event for this variation. |
| Conversions | Unique visitors with at least one conversion. |
| Conversion rate | Conversions divided by visitors. |
| Lift vs control | Relative change: (variation rate minus control rate) divided by control rate. +20% means the variation converts 1.2 times as often. Shown as a dash when the control has no conversions yet. |
| 95% interval | The range the true conversion rate very likely falls in, given the data so far. |
Confidence intervals (Wilson)#
Each conversion rate comes with a 95% Wilson score interval. Wilson is used instead of the simpler normal approximation because it stays accurate for small samples and for rates close to 0% or 100%. For example, 50 conversions from 1,000 visitors gives 5.0% with an interval of about 3.8% to 6.5%.
Intervals get narrower as visitors grow. Overlapping intervals do not prove there is no difference, but they are a sign that the test is not done.
P-values and significance#
The platform runs a two-sided, two-proportion z-test between the control and the best-performing variant (the one with the highest conversion rate). The p-value is the chance of seeing a difference at least this large if the variations were really the same. The result is called significant when p is below 0.05, which matches 95% confidence.
Minimum sample
Significance is only tested once both the control and the best variant have at least 100 visitors. The dashboard shows Collecting data until every variation has at least 100 visitors.
What the summary means#
| Summary | What to do |
|---|---|
| Collecting data | At least one variation has fewer than 100 visitors. Wait. |
| Variant is outperforming the control | The best variant converts better and p < 0.05. This is your candidate winner, if the test has run long enough (see below). |
| The control is outperforming every variant | The best variant is significantly worse. Consider stopping and keeping the control. |
| No significant difference yet | The data cannot tell the variations apart. Keep running, or accept that the effect is too small to matter. |
A/B/n tests
With three or more variations, only the best variant is tested against the control, and there is no correction for multiple comparisons. Treat a borderline p-value (say 0.03 to 0.05) with extra caution when you have many variants.
How many visitors you need#
The smaller the change you want to detect, the more visitors you need. These numbers are per variation, for 95% confidence and 80% power, using the same formula as the platform's statistics library.
| Baseline conversion rate | Detect a 10% lift | Detect a 20% lift | Detect a 30% lift |
|---|---|---|---|
| 2% | 80,683 | 21,110 | 9,799 |
| 5% | 31,235 | 8,159 | 3,781 |
| 10% | 14,752 | 3,843 | 1,776 |
Divide by your daily visitors per variation to estimate the duration. Run for at least one full week either way, so weekday and weekend behaviour are both included.
Inconclusive results#
Many tests end without a significant result. That is still useful: the change did not make a big difference, so you can ship whichever version is simpler or cheaper, and move on to a bolder idea. Do not keep a test running forever hoping it will flip.
Peeking and stopping early#
The z-test assumes you look at the result once, at a sample size you chose in advance. The dashboard updates continuously, and if you stop the first time p dips below 0.05, your real false positive rate is much higher than 5%.
- Decide the sample size before you start, using the table above.
- Check that the experiment is working, but do not make the decision until you reach that size.
- If the result is significant early, keep going to your planned size anyway.
Visitors not split as expected#
If the number of visitors in each version is far from your traffic split, the Results tab shows a warning. For example, a 50/50 test with 5,500 visitors on the control and 4,500 on the variation. A gap this large rarely happens by chance, so something is usually wrong: a redirect that drops visitors, a slow or broken variation page, or the snippet missing on one page.
The check runs once a test has at least 100 visitors and flags a gap only when the chance of it happening by luck is under 0.1%. Fix the cause before trusting the results. If you changed the split during the test, the warning can appear for that reason alone.
Heatmaps#
On paid plans, every running test collects heatmaps automatically for each variation, from a sample of visitors. Open the Heatmaps tab on the experiment, pick a variation and a device, and switch between the click map, scroll map, confetti (each click as a dot), an overlay on the page and an element list with rage and dead clicks.
Heatmaps record where people click, which element they clicked and how far they scroll. They never record keystrokes, form values or page text. Individual clicks are kept for 30 days; daily totals are kept for the life of the test.
Sharing results#
At the bottom of the Results tab, select Create share link to get a read-only link you can send to anyone, even people without an account. The page shows the test name, each goal and the numbers for every version. It does not show your page addresses, goal settings or account details, and search engines are asked not to index it.
Select Turn off to stop the link working at once. Creating a new link afterwards gives a new address, so the old one never comes back.
Results by API#
The same numbers are available from GET /api/experiments/{id}/results. See the REST API.