What It Actually Means
It is the closest thing marketing has to an experiment, and its appeal is obvious: instead of arguing about whether the green button works better, find out.
The Problem Most Businesses Have
You probably do not have enough traffic, and this is the part that gets skipped.
Detecting a genuine difference requires a certain number of outcomes in each group, and the smaller the difference the more you need. Detecting a change from a 2% conversion rate to 2.4%, which would be a substantial commercial improvement, needs thousands of conversions per version.
A site with two hundred visitors a week and eight enquiries cannot run that test. It can run something that produces a number, and the number is noise. One version will be ahead, the tool will draw a chart, and the result would reverse if you ran it again.
Businesses at this scale routinely make real decisions on that basis, and it is genuinely worse than deciding on judgement, because it carries false authority.
How To Tell If A Result Is Real
Three checks, none requiring statistics.
Was the sample size set before starting? A test should have a target number of conversions decided in advance. Watching until it looks significant and stopping is the most common error and it manufactures positive results reliably.
Did it run for whole weeks? Behaviour differs by day. A test running Tuesday to Friday compares a version to a different audience, not to a different design.
Would you bet on it repeating? The honest gut check. Most small-site test results would not survive being run again.
What To Do Instead
At low traffic, the productive alternatives are unglamorous and more reliable.
Test large changes rather than small ones. A fundamentally different page can produce a difference big enough to detect. Button colours cannot.
Fix the obvious. Most small sites have several problems that do not require testing: a form with too many fields, a buried action, a page that does not match its ad, something broken on mobile. Removing those is worth more than optimising anything.
Watch real people use it. Five recorded sessions of somebody trying to complete an enquiry will tell you more than a month of underpowered testing, and it points at causes rather than at outcomes.
Ask the people who did enquire what nearly stopped them. Cheap, and consistently informative.
What To Ask
- How many conversions do we get per week? This decides whether testing is available to you at all.
- What sample size was set, and was it reached? If nobody set one, the result is not evidence.
- How long did it run, and did it cover whole weeks?
- Are we testing something big enough to detect?
Where It Genuinely Works
At sufficient volume this is one of the most valuable tools available, and for businesses running substantial paid campaigns the traffic often exists even when the site overall is small, because it is concentrated on one landing page.
That is the realistic route in for a smaller business: test the page taking campaign traffic, where volume concentrates and the commercial stakes are clearest, and leave the rest of the site to judgement and obvious fixes.
More terms are in the glossary.
