You are testing twenty creatives and calling noise a winner

01What does statistical significance actually mean in an ad test?

It means one narrow thing. If the two ads were truly identical, a gap this large would turn up less than one time in twenty by chance alone. That is the whole claim. It is not a 95% chance your ad is better, it is not a promise the gap will hold next month, and it says nothing at all about how big the real difference is.

The half nobody says out loud: even when a difference is real, the measured size of it is inflated by the act of picking the biggest one. The top of a noisy list is high partly because it is good and partly because it got lucky, and only the first part comes with you into next month.

02The multiple comparisons problem, in plain language

Run one comparison at a 5% threshold and you accept a one-in-twenty chance of being fooled. Run nineteen challengers against a control and the chance that at least one of them fools you is 62%. Compare all twenty ads against each other and there are 190 pairs. Wikipedia's own worked example puts a hundred independent tests at roughly a 99.4% chance of at least one false positive.

In practice an ad test is worse than that, because you are not running twenty separate hypothesis tests. You are looking at a leaderboard and taking the top row. There is no threshold to correct, no test being run, and no record of the nineteen comparisons your eye just performed. The correction never happens because the comparisons were never counted.

A leaderboard always has a top row. That is a property of leaderboards, not of your creative.

The sentence we now open every test review with

What that does to a real test

We ran the simulation rather than describing it. Twenty ads, every one with the identical true conversion rate of 2%, a thousand visitors each, two hundred thousand repeats. There is no better ad in this world. There is nothing to find.

import numpy as np
rng = np.random.default_rng(20260901)

# 20 identical ads. Same true rate. 1,000 visitors each.
k, n, p, reps = 20, 1000, 0.02, 200_000
x = rng.binomial(n, p, size=(reps, k))
best = x.max(axis=1) / n              # the ad you would have crowned

print(best.mean() / p - 1)            # 0.434  -> a 43% lift out of nothing
print((best >= p * 1.2).mean())       # 0.991  -> it clears +20% almost every time

# Now give one ad a real 20% edge and spend the SAME total budget.
def true_best_wins(k, total, p=0.02, lift=0.20, reps=200_000):
    n = total // k
    rates = np.full(k, p); rates[0] = p * (1 + lift)
    x = rng.binomial(n, rates, size=(reps, k))
    return (x.argmax(axis=1) == 0).mean()

for k in (4, 5, 8, 10, 20):
    print(k, round(true_best_wins(k, 20_000), 3))   # .822 .735 .536 .446 .229

The leader averaged a 43% lift over the truth. It cleared a 20% lift in 99.1% of runs. If you had been watching that leaderboard you would have found a convincing winner every single time, written a note about what made it work, and rolled it out. Widen to forty ads and the leader's phantom lift goes past 50%. Give each ad five thousand visitors instead of a thousand and it drops to 19%, which is smaller and still entirely fictional.

03Fewer arms beats more arms on the same money

The second half of the simulation is the part that changes what you do on Monday. Give one ad a genuine 20% edge, hold the total budget fixed at twenty thousand visitors, and vary only how many ways you split it. Splitting twenty ways, you crown the genuinely better ad 22.9% of the time. Splitting four ways, 82.2%. Same money, same real effect, more than triple the odds of finding it.

This cuts against the advice everyone gives, so state the boundary honestly. Volume genuinely works: shipping more distinct swings is the largest controllable lever in feed advertising, and Motion measured that across 578,750 creatives. Volume over time is not the same thing as volume in one test. Ship many creatives across many weeks; run few of them against each other at once. The argument for the first is in how many creatives your budget can actually teach you.

What genuinely different looks like on one brand

Three arms, not three edits

Argument: a cue in your eveningArgument: it belongs with the good part of the dayArgument: privacy, stated plainly

Three statics for the same house wellness-patch brand. Different argument, different world, different energy, different point of view. Any two of these can lose without telling you anything about the third, which is the property that makes an arm worth its share of the budget.

04How many conversions does a creative test actually need?

Roughly 420 conversions per creative to detect a 20% relative lift at 80% power, and roughly 1,600 per creative for a 10% lift. Widen the difference you are hunting and the requirement collapses: a 50% relative difference needs only about 64 per creative, which is the figure the calculator in how many creatives your budget can teach you runs on. What surprises people is that none of it moves much with your conversion rate. At a 1% baseline you need about 1,631 conversions per arm for a 10% lift; at 5% you need about 1,562. Your conversion rate changes how much traffic those conversions cost, not how many you need.

One honest caveat on that last output. It assumes the comparisons are independent and that you would test every pair, and neither is quite true, so the real figure sits lower than the one shown. What pulls it back up is that a leaderboard applies no correction at all. Treat it as an order of magnitude.

05Why checking every morning makes it worse

Evan Miller's demonstration is the clearest one in circulation. Test a 50% conversion rate against itself, check for significance after every single observation, stop the moment you see 5%, cap it at 150 observations. The real false positive rate is 26.1%, not 5%. His table for less extreme peeking is just as instructive: to hold a true 5% after ten looks, you need to report 1.0%.

06What do you do when you cannot afford significance?

Most accounts cannot, and pretending otherwise is how people end up trusting a number they should not. Four moves, in order of how much they buy. Widen the effect you are hunting: stop looking for a 10% lift and go looking for a 40% one, which is a fifteenth of the sample and is also the only kind of difference a genuinely new creative idea produces. Then screen on the metric you have thousands of instead of hundreds, which usually means hook rate and the triage order for fixing one that is below benchmark.

The third move is to stop calling it a test. Platform delivery is an allocation algorithm that shifts budget toward whatever is working right now, which is often the right tool and never a defensible verdict about which creative is better.

The fourth is the one people find hardest. Say it out loud: we chose under uncertainty and it may be noise. Then behave accordingly - keep the alternatives alive, expect the winner's number to fall, and do not build a theory of your customer on top of it. A wrong belief about why an ad worked outlasts the ad.

07Leaving the losers running is sometimes the cheaper choice

Turning arms off is what makes the noise permanent. At twenty arms and a thousand visitors each, the ad you crown is the genuinely better one 22.9% of the time in our simulation, which means the other 77% of the time the ad you switched off was the better one, and you will never learn that because it is off.

There is a real cost to holding losers open, and it is delivery, not money - budget follows performance, so a genuinely weak ad starves itself. Weigh that against the cost of rebuilding a creative you already paid for, which is the argument in what failed renders actually cost. And before you conclude an ad has stopped working at all, rule out the boring explanations first, which is what telling creative fatigue apart from a bad week is for.

Test. Then believe the answer only if you could afford to run the test that produced it. The cheapest fix is also the least popular one: fewer arms, further apart, read once, on a date you wrote down before you launched.