Interactive demo
Why your A/B test is lying to you
Almost every experiment dashboard updates daily, and almost everyone watching one has stopped a test the moment it turned green. That single habit quietly turns a 5% false positive rate into something closer to a coin flip's worth of wishful thinking. Below, two identical variants — no difference whatsoever between them — and a test that will still tell you one of them won.
Everything here is simulated, and that is the point. The question is not what one experiment did, but how a decision rule behaves when run thousands of times against a truth we control. Conversions are drawn as real Bernoulli trials and significance is a standard two-proportion z-test — about 340 lines of plain JavaScript, no libraries.
The experiment
One experiment, day by day
Watch the p-value wander as data accumulates. Every dot below the dashed line is a day on which the dashboard would have shown a significant result — and someone watching would have been tempted to call it.
Press play to watch the p-value move as data arrives.
Now do it a thousand times
One experiment proves nothing about a decision rule. So run many, against the same known truth, and compare two policies on identical data: testing once at the end, versus checking every day and stopping at the first significant result.
Called a winner — fixed
—
Called a winner — peeking
—
Inflation
—
Median effect reported
—
Run the simulation to compare the two decision rules.
So plan it instead
The fix is not clever statistics, it is deciding the sample size before you start and then not looking early. Here is the size you need to detect a given effect, from the baseline conversion rate set above.
Two-sided test at α = 0.05.
Visitors needed per arm
—
— days at the traffic set above