Skip to main content

A/B Significance & Sample Size

What it does

The A/B Significance Calculator runs in two modes. Pre-test: given a baseline conversion rate and the minimum lift you want to detect, it returns the sample size required per variant, so you know whether your traffic supports the test before you start. Post-test: given the visitors and conversions for two variants, it returns the relative lift, the p-value via two-proportion z-test, and a verdict on statistical significance at 95% and 99% confidence.

Common situations

You’re about to launch an A/B test on landing page copy and need to know whether the traffic volume will produce a conclusive result. The pre-test mode tells you: at 2% baseline and 20% minimum detectable lift, you need ~5,000 visitors per variant. If you’re getting 1,000 visitors per week, the test takes 5 weeks per variant.

A test has been running for a week and the variant is showing a 15% lift. The post-test mode tells you whether that’s statistically significant. At low baseline rates and modest sample sizes, it usually isn’t yet. The calculator settles “should we call this test or keep running” objectively.

You’re discussing test results in a meeting and someone says “the new variant is winning”. The calculator’s p-value tells you whether they’re right or whether the lift is plausibly noise. P-value 0.04 = significant at 95%; p-value 0.31 = inconclusive.

You’re planning a test of an expensive change (new design, new tech stack, complete rewrite) and need to know whether the test is statistically feasible at your scale. Pre-test sample size shows whether you have enough traffic; if you don’t, the test isn’t possible at the resolution required.

You’re auditing a vendor’s A/B test results that claim significance. Run their numbers through the calculator; it’s common for vendor reports to claim significance at sample sizes that are statistically inadequate for their stated baseline.

What you need to know

A/B testing aims to determine whether a change to a page (or campaign, or email, or anything else) actually moves the metric you care about, against the baseline. The challenge is that random variation in conversion rates creates “lifts” that aren’t real. At typical conversion rates and small samples, the variation can be substantial.

The two-proportion z-test addresses this by calculating the probability that an observed lift is the result of random variation rather than a genuine difference. The output is a p-value: the probability that you’d observe a lift this large (or larger) if there were no real difference between the variants.

p < 0.05 means there’s less than a 5% chance the lift is random noise. Conventionally this is “statistically significant at 95% confidence”, the threshold most teams use to call a test.

p < 0.01 means less than 1% chance of being noise. Higher confidence; appropriate for high-stakes decisions or businesses with regulatory requirements.

p ≥ 0.05 means the result is inconclusive. The lift might be real, or it might be noise; the data doesn’t distinguish. Either keep running (if you can) or accept the result is unclear.

The pre-test sample size calculation:

The required sample depends on baseline rate, minimum detectable effect (MDE), statistical power, and significance level. The formula is:

n = [(z_α × √(2 × p̄ × (1 – p̄)) + z_β × √(p1 × (1 – p1) + p2 × (1 – p2)))² ] / (p2 – p1)²

Where p1 is the baseline rate, p2 is baseline × (1 + MDE), and z_α and z_β are critical values for significance and power respectively.

The intuition: lower baseline rates need much bigger samples (because the effect is buried in low-rate noise), smaller MDEs need much bigger samples (because finding a 5% lift is harder than finding a 50% lift), and higher confidence requirements need bigger samples.

A practical example: at 2% baseline and 20% MDE (looking for a 0.4 percentage point lift to 2.4%), 95% confidence and 80% power, you need ~5,000 visitors per variant. At 5% baseline and same parameters: ~1,800 per variant. At 0.5% baseline (typical for high-consideration B2B): ~20,000 per variant.

The post-test calculation runs the two-proportion z-test on actual data to determine significance. The calculator returns p-value, relative lift, and verdicts at both 95% and 99% thresholds.

What the calculator doesn’t capture:

  • Multi-variant tests (more than two variants). Running multiple comparisons inflates the false-positive rate; corrections like Bonferroni need to be applied. The calculator handles two-variant tests only.
  • Sequential testing. Calling a test the moment p crosses 0.05 inflates the false-positive rate (the “peeking problem”). Wait until your pre-determined sample size is reached, then call.
  • Bayesian methods. Some teams prefer Bayesian A/B testing which avoids p-values and reports probability the variant is best. The calculator uses frequentist (z-test) methods.
  • Non-conversion metrics. If you’re testing revenue per visitor or other non-binary outcomes, t-test is more appropriate than z-test. The calculator handles conversion rate (binary outcomes) only.

Frequently asked questions

What’s a good sample size for an A/B test?

Depends on your baseline conversion rate and the effect size you want to detect. Use the pre-test calculator with your actual baseline. The honest answer for most teams: more than they typically run. Tests called at 500 visitors are usually inconclusive even when they look like they have a clear winner.

When can I call a test?

After reaching the pre-determined sample size and seeing p < 0.05 (or your chosen threshold). Calling early, when p first crosses the threshold but the planned sample isn’t reached, inflates false-positive rates. Pre-commit to the sample size and wait.

Why is p-value 0.06 not significant?

It is significant at 94% confidence, but conventional thresholds are 95% (p < 0.05) or 99% (p < 0.01). The boundary is arbitrary; 0.06 means there’s a 6% chance the result is noise. Often the right call at 0.06 is to keep running rather than declare a winner.

What’s statistical power?

The probability of detecting a real effect when one exists. 80% power means the test has an 80% chance of detecting the specified MDE if it’s real. Higher power needs bigger samples; 80% is the conventional default.

What’s MDE (minimum detectable effect)?

The smallest lift you care about detecting. If you only care about lifts of 20% or more, MDE = 20%. If you’d act on 5% lifts, MDE = 5%. Smaller MDEs require dramatically bigger samples, so be honest about what’s actionable rather than overspecifying.

Can I keep running the test if it’s not significant yet?

Up to your pre-determined sample size, yes. After that, accept the result and decide based on what you have. Continuing past the planned sample to chase significance (“data peeking”) inflates false-positive rates and produces unreliable conclusions.

What if the test shows a negative lift?

Same logic applies: significance is significance regardless of direction. A statistically significant negative lift means the variant is actually worse than control. The calculator handles both directions equivalently.

Why does the calculator say “no real difference detected” when one variant is clearly winning?

Because the sample isn’t large enough to be confident the lift is real rather than random variation. Visual lift can be misleading at small samples; the math is what tells you whether the lift is genuine.

Common problems

Problem: Test running for 2 weeks shows 30% lift but p-value is 0.4.

Sample is too small for the baseline rate. At low baselines (1-2%), even large-looking lifts need thousands of visitors per variant to reach significance. The variant might be better, but your test can’t yet tell. Either keep running or accept the result is inconclusive.

Problem: Pre-test calculator says I need 50,000 visitors per variant.

Your baseline is low and your MDE is small. Either lower the MDE expectation (look for 30% lifts instead of 10% lifts), accept higher confidence threshold (90% instead of 95%), or recognise that the test isn’t statistically feasible at your traffic volume.

Problem: P-value flips between significant and not significant as the test runs.

Normal. P-value drifts as data accumulates; this is the “peeking” problem. Don’t make decisions based on day-to-day fluctuations. Wait until the planned sample is reached.

Problem: Variant A wins on metric 1 but Variant B wins on metric 2.

Multi-metric tests need pre-specified primary metric. If you measure multiple things and any of them are significant, the false-positive rate inflates. Decide upfront which metric matters most; report secondary metrics as secondary, not as alternative winners.

Problem: Test ran to planned sample, p = 0.08, looks like B is winning but isn’t significant.

Choose your action based on what you know. If you’re certain B is theoretically better and the cost of being wrong is low, ship it; the data doesn’t disprove your prior. If you’re uncertain, the right move is to accept the result is inconclusive: neither significantly better nor worse.

Tips

  • Calculate sample size before launching tests. Tests that aren’t statistically feasible at your traffic volume waste time and produce unreliable conclusions.
  • Pre-commit to sample size. Adjusting based on what you see during the test is data peeking.
  • Don’t run more than 2 variants if your traffic is limited. Multi-variant tests need bigger samples and proper correction.
  • Use 95% confidence as the default; reserve 99% for high-stakes decisions where false-positive cost is significant.
  • Report results honestly. “Inconclusive” is a valid result; pretending an inconclusive test is a winner is worse than no test.

Related tools in this suite

Part of the Paid Search Suite, this calculator handles the statistical side of testing. The Conversion Rate Calculator provides the baseline rate input for the pre-test calculator. The RSA Tester is upstream: A/B testing different ad copy combinations is a primary use case for this tool.

What this looks like at scale

For a single test, the calculator is sufficient. For an organisation running many tests across pages and campaigns, A/B testing infrastructure should be part of the analytics stack: Optimizely, VWO, or Google’s experimentation tools, with proper test management and post-test analysis built in. The systems we build often include this kind of testing infrastructure for clients running data-driven optimisation programs. For the concepts behind statistical testing, the Knowledge Center covers the underlying theory.

Take it further

If your team runs many tests but has trouble drawing reliable conclusions from them, the underlying issue is usually test design: sample sizes too small, multi-variant tests without correction, or peek-and-call practices. Talk through the practice and we can scope what testing discipline looks like.