Skip to content
Impact Conversion

Experimentation · October 2026

How much traffic do you need to run an A/B test?

Most stores ask this question the wrong way round. They look at their traffic, decide it feels like plenty, and start testing. Three weeks later the dashboard says "not significant," someone suggests running it another fortnight, and the test quietly dies in month two.

The traffic you need doesn't depend on how busy your site feels. It depends on three things: how many people buy, how big a lift you're trying to catch, and how many versions you split them across. Get those right and you can tell, before anyone writes a line of code, whether a test will ever give you an answer.

1. Count orders, not visitors

Visitor counts flatter you. A test with 80,000 visitors per arm sounds enormous until you notice it holds about 1,600 orders each. The orders carry the signal. Most visitors are just noise around them.

That's why the cleanest rule of thumb counts orders. At 95% confidence and 80% power, each arm needs roughly:

  • 400 orders to detect a 20% lift
  • 1,600 orders to detect a 10% lift
  • 6,300 orders to detect a 5% lift

Those numbers barely move with your conversion rate. A store converting at 1% and one converting at 5% both need about 1,600 orders per arm to catch a 10% lift. The 1% store just has to buy five times as many visitors to get them.

So the first question isn't "how much traffic do we get?" It's "how many orders does this page see in a month?"

2. Halving the lift quadruples the sample

Look at the list again. Going from a 20% lift to a 10% lift doesn't double the orders you need. It quadruples them. Going to 5% quadruples them again.

Here's the same thing in visitors, for a store converting at 2%, across both arms:

  • To detect a 20% lift: about 42,000 visitors
  • To detect a 10% lift: about 161,000 visitors
  • To detect a 5% lift: about 630,000 visitors

Most changes don't produce 20% lifts. A new headline, a reworded button, a different hero image: if these move anything, they usually move it a little. That's the trap. The tests that are easiest to dream up are the ones with the smallest effects, so they need the most traffic you don't have.

3. Know your minimum detectable effect before you launch

Every test has a blind spot. Lifts below a certain size are too small for your traffic to tell apart from luck. That threshold is your minimum detectable effect, or MDE. It's the smallest real improvement your test can reliably catch.

"Reliably" has a precise meaning here. At the standard settings (95% confidence, 80% power), a test sized for a 15% MDE will catch a true 15% lift about four times out of five. A true 5% lift, sitting well under the line, it will mostly miss. The result comes back "not significant," and everyone concludes the idea didn't work, when the test was never able to see it.

That's why MDE is the most useful number in testing. It tells you what the test can and can't answer before you spend a month finding out.

You work it out backwards from your traffic. Take the visitors the page gets, assume you'll run for about four weeks, and ask how small a lift you could detect. At a 2% conversion rate, with all of that page's traffic in the test:

  • 20,000 visitors a month: MDE of about 30%
  • 50,000 a month: about 18%
  • 100,000 a month: about 13%
  • 250,000 a month: about 8%

Rather than read it off a list, put your own numbers into our pre-test calculator. Enter your weekly traffic, conversion rate and the lift you expect, and it shows your MDE week by week, plus a go or no-go on whether the test can reach it.

Then hold your idea up against it. If your MDE is 18% and the change might realistically move conversion by 5%, the test can't succeed. Most likely it finds nothing. Sometimes luck pushes it over the line and you ship a "win" that was never there, and that's the expensive outcome.

There are only three ways to shrink your MDE: more traffic, a longer test, or a bigger change. Section 5 covers what to do when none of those is enough.

4. The traffic that counts is the page's, not the site's

Sitewide traffic is the number on the dashboard. The number that matters is how many people actually see the change.

A product page template test borrows most of your buyers, because nearly everyone views a product. A test on one collection page gets a slice. A test inside a submenu gets the sliver of people who open the submenu, which is usually fewer than you'd guess. Before you size a test, check how many sessions reach the element you're changing, then how many of those go on to buy.

Two more things eat into the number:

Extra variations. Each one adds a whole arm's worth of traffic. An A/B/C test needs about 50% more visitors than an A/B test to reach the same verdict, and it gives you more chances to be fooled by luck.

Revenue per visitor. If you're calling the test on revenue rather than conversion rate (you should, usually), expect to need more traffic, because order values swing around far more than a yes-or-no purchase does. The revenue-per-visitor tab in the same calculator works this out from your own order data.

5. What to do when the numbers say no

Plenty of tests won't pass this check. That doesn't mean you do nothing.

Make the change bigger. If you've got four small fixes for the product page, bundle them into one variation. You'll learn less about which fix did the work, but you'll get an answer.

Move the test to a busier surface. The same idea on the product template or the cart often has the traffic that a single landing page doesn't.

Run longer, with rules set up front. Six to eight weeks is fine if you decided that before launch. Stopping the day the dashboard turns green isn't. That habit is how a 5% false-positive rate turns into 20% or 30%, which I've covered in why most A/B tests are statistically broken.

Ship it on research, and label it. If surveys, recordings and support tickets all point at the same broken thing, fix it. Just write down that it wasn't tested, so nobody quotes it as a proven win later.

If you're running a smaller store, the New Zealand version of this argument goes further into what's worth testing at low volume.

The rule of thumb

  • Size on orders, not visitors: about 400 per arm for a 20% lift, 1,600 for 10%, 6,300 for 5%.
  • Halve the lift you're chasing and you need four times the sample.
  • Work out your MDE before you brief the test: the smallest lift your traffic can catch. If your idea won't beat it, don't test it.
  • Every extra variation costs a full arm of traffic.
  • If the numbers say no, make the change bigger, move it somewhere busier, or ship it on research and say so.

If your test roadmap is full of tests that never reach a verdict, the problem probably started before launch. Run your numbers in the pre-test calculator before you brief your next test. Or book a 15-minute call and we'll go through which of your pages can be tested and which can't.