A/B test calculator.
Find out whether the gap between two versions of a page is real or could be chance (whether it's statistically significant), and how long your test needs to run.
- Free
- Runs in your browser
- Nothing is sent or stored
A converts at
2.00%
B converts at
2.40%
B against A
+20.0%
Version B looks ahead, but it could still be chance.
If A and B really converted the same, a gap this big would still turn up 5.4% of the time. That's over the 5% line for 95% confidence, so keep the test running.
B's conversion rate is likely somewhere between −0.01 points and +0.81 points against A's (95% range). That range includes no difference at all.
Details for the statistically minded
- Test
- Two-sided two-proportion z-test
- z-score
- 1.928
- p-value
- 0.054
- Significance level
- 0.05
How long should it run?
Work out the visits each version needs before you start, then let the test run that long. Stopping as soon as one version looks ahead is the most common way to crown a winner that isn't.
- Visits for each version
- 21,109
- Visits in total
- 42,218
- Run it for
- 4 weeks
22 days of traffic, rounded up to whole weeks so every day of the week is counted the same. Check the result once, at the end.
At 95% confidence, set above, and 80% power: the chance of spotting a real rise of that size.
Where tests go wrong.
The calculator can only judge the numbers it's given. These are the habits that make a result look surer than it is.
- Stopping when it looks good Checking every day and stopping the first time B crosses the line finds a lot of winners that aren't. Set the length first and check once, at the end.
- Too few visits for the change Small rises need a lot of traffic. At a 2% conversion rate, a 10% rise needs about 80,683 visits for each version. Test bold changes on your busiest pages first.
- Part weeks and sale periods Weekend and weekday shoppers behave differently, and a sale changes everyone. Run whole weeks, and keep big promotions out of the test.
- Counting the wrong thing A change that gets more clicks can still lose orders. Judge the test on orders, or revenue, not on clicks.
Once a version wins, put a figure on it: the conversion rate calculator shows what each rise in your rate is worth in orders and revenue.
See what the winner is worthCommon questions.
About significance, sample size and how long to run a test.
How do I know if my A/B test is significant?
Enter each version's visits and orders above. If a gap that big would turn up by chance less than 5% of the time when the two versions are really the same, the result is significant at 95% confidence, the usual line.
How long should an A/B test run?
Until each version has the visits you planned for, and for at least two full weeks so every day of the week is counted the same. Work the visits out before you start, with the planner on this page.
What sample size do I need for an A/B test?
It depends on your conversion rate and the smallest rise worth finding. At a 2% conversion rate, finding a 20% rise takes about 21,109 visits for each version; a 10% rise takes about 80,683.
What does the p-value mean?
The chance of seeing a gap at least this big if the two versions really converted the same. Below 0.05 is the usual line for 95% confidence. It isn't the chance that B is better.
Should I use 90%, 95% or 99% confidence?
95% is the usual choice. 90% reaches a verdict sooner but crowns more false winners; 99% is stricter and needs more visits. Pick one before the test starts, not after you've seen the result.
Why does it say “could be chance” when B looks better?
Small samples swing a lot. Until each version has enough visits, a gap of this size turns up by chance too often to call. Keep the test running to the length you planned.
Do you see the numbers I enter?
No. The calculation runs in your browser and the figures never leave it. We only record that the calculator was used, not what was typed.
Know what's worth testing.
A CRO audit gives you a ranked list of the changes most likely to move your conversion rate, with a way to measure each one.