Free tool
A/B Test Calculator
Three calculators in one. Decide whether a test is worth briefing, paste in live data to see where you are, and project when the 85% or 95% threshold becomes reachable. Covers both conversion rate and revenue per visitor.
Surface
Implied baseline conversion: 3.20% at 8,000 visitors/week.
Plan
Stats
✓ Projected ~512 orders/arm by week 4 (floor: 100)
✓ Happy zone: 250+ orders/arm
✗ Smallest detectable lift by week 4: ~18% vs expected 10%
You can only detect 18% by week 4 but expect 10%. Test a bigger swing, run longer, or pick a higher-traffic surface.
Detectable lift by week
| Wk 1 | Wk 2 | Wk 3 | Wk 4 | Wk 5 | Wk 6 | |
|---|---|---|---|---|---|---|
| orders/arm | 128 | 256 | 384 | 512 | 640 | 768 |
| min. detectable | 37% | 26% | 21% | 18% | 16% | 15% |
| 2% | · | · | · | · | · | · |
| 3% | · | · | · | · | · | · |
| 5% | · | · | · | · | · | · |
| 7% | · | · | · | · | · | · |
| 10% | · | · | · | · | · | · |
| 15% | · | · | · | · | · | ✓ |
| 20% | · | · | · | ✓ | ✓ | ✓ |
| 25% | · | · | ✓ | ✓ | ✓ | ✓ |
| 30% | · | ✓ | ✓ | ✓ | ✓ | ✓ |
| 40% | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| 50% | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Size on orders, call on your locked primary metric. 100 orders/arm is the callable floor, 250+ is the happy zone. Two-proportion z-test at your significance and power settings.
If the expected lift is real: how confidence builds
Bayesian probability variant beats control over time, assuming your expected 10% lift is real. Dashed lines mark the 85% and 95% thresholds.
A reminder before you ship the win
Hitting 95% in this calculator is necessary, not sufficient. Three things worth checking before you call a test:
- Sample size was pre-committed. Peeking daily and stopping at the first crossing inflates false positives from 5% to 20 to 30%.
- Test ran for at least two full business cycles. Weekday and weekend traffic behave differently. A three-day test captures one of them.
- Segments are hypothesis generation, not validation. Slicing by device and re-running significance on the slice is a false-positive factory.
This is the discipline behind our A/B testing service. Optimising for revenue, not just conversion rate? Try the revenue per visitor calculator.
A/B test calculator FAQ
- How long should an A/B test run?
- Long enough to reach the sample size your effect needs, and at least two full weeks so weekday and weekend behaviour both land in the data. Paste your live numbers and days running into the calculator and it projects the additional days needed to reach 85% or 95% confidence at your current rate.
- What sample size do I need for an A/B test?
- It depends on your baseline conversion rate, the lift you want to detect, and your significance and power settings. Smaller lifts need far more traffic. The pre-test tab shows the per-variation sample size and whether you can realistically reach it in your test window.
- What is statistical significance in A/B testing?
- It is the probability the result is not down to chance. 95% significance means a 5% false-positive risk. This tool reports probability-to-beat-control so you can watch confidence build over time rather than reading a single pass or fail.
- Can I calculate revenue per visitor, not just conversion rate?
- Yes. The revenue-per-visitor tab handles continuous revenue data, which is driven by the coefficient of variation rather than a simple rate. It is the metric that tracks profitability, since a test can lift conversion while dropping average order value.
- When can I call an A/B test a winner?
- When you pre-committed the sample size, reached your significance threshold, and ran across at least two business cycles. Stopping at the first threshold crossing while peeking daily inflates false positives from 5% to 20 or 30 percent.