The Null World
A p-value is one counted number — and almost no one, including the people who report it, can say what it counts. Here is the whole thing, made of nothing but coin flips: assume nothing is going on, spin that world thousands of times, and count how often blind chance does what you saw or more. That fraction is the p-value. Watch it assemble itself.
The null-world simulator
You flipped a coin 100 times and got 60 heads. Does that prove it's biased — or is a fair coin just doing what fair coins do?
A fair coin lands this far from an even split — or further — about 5.7% of the time.
That number you watched build — the orange fraction — is the entire definition: the probability that a fair coin produces a result at least this extreme. Written out, p = P( data this surprising given the null is true ). The simulation just counts it; the exact binomial sum (in green) is where the count is heading.
The default already hides the first surprise. 60 heads out of 100 feels obviously rigged — yet a fair coin does that or better about 5.7% of the time, a hair over the sacred 0.05 line. Nudge the slider to 62 and it slips under. The line is a convention someone chose in the 1920s, not a fact about the coin. Drag the dials and feel where "significant" actually lives.
Three things a p-value is not
Every misuse below is in textbooks, press releases, and papers. Each one is undone by a number you can recompute here.
Misreading 1 — the inverse
Not the probability the coin is fair
The p-value runs one direction: it assumes the coin is fair and asks how surprising your data is. The thing people want — the chance the coin is fair given the data — runs the other way, and you can't flip a conditional probability without a prior. (This is the exact inversion that Bayes' theorem exists to fix.)
How far apart are the two directions? A much-cited calibration (Sellke, Bayarri & Berger, 2001) gives a lower bound: even being as generous as the mathematics allows to the "it's biased" verdict, a result sitting at p = 0.05 still leaves the coin fair at least ≈ 29% of the time. Not 5%. The p-value is not your error rate.
Misreading 2 — significance is not size
Not a measure of how biased the coin is
A barely-bent coin becomes "statistically significant" if you flip it enough. Significance measures detectability — effect size tangled up with sample size — never the size of the effect alone. Hold a real bias fixed and just add flips:
The effect never changes; only n does. A 52% coin needs about 2,400 flips before it crosses 0.05; a 51% coin, nearly 10,000. A tiny p means a detectable effect, not a big one.
Misreading 3 — the silence
p > 0.05 does not prove the coin is fair
A non-significant result is not a verdict of "no effect." It can equally mean "real effect, not enough flips to see it." Absence of evidence is not evidence of absence — and usually you cannot tell the two apart from the p-value alone.
Concretely: a coin that truly lands heads 55% of the time, flipped 50 times, fails to reach significance ≈ 92% of the time. The bias is real and sizeable; the experiment is just blind to it. Reading that silence as "the coin is fair" gets it backwards nine times in ten.
That 92% is one point on a surface, and it has a name. Turn the dials on it below, along with the thing this card left out: what the significance filter does to the results that do get through.
The world where the coin is bent
Everything above lives in the null world, where nothing is going on. Here is the other one, drawn against it.
"Missed 92% of the time" is a fact about the experiment, not about the coin, and the quantity it reports is called power: the chance a study can see the thing it is looking for. Below, the fair coin's outcomes (blue) are drawn against a coin that really is bent (violet), with the region that gets called significant shaded orange. Move the bend and the number of flips, and watch the two worlds come apart.
The power instrument
The coin truly lands heads 55% of the time. You flip it 50 times and run the same test as above. How often do you catch it, and what do you report when you do?
Two things are worth watching, and only one of them is usually taught.
The overlap. Power is simply the share of the violet curve that falls inside the orange. Drag the flips down and the violet slides back under the blue: the experiment loses the ability to tell the two worlds apart, and a non-significant result stops carrying information at all. That is card 3, made operable.
The gap between the two yellow lines, which is the part almost nobody shows. The solid line is the coin's real bias. The dashed line is the average size of the bias reported by the runs that reached significance, drawn on the coin's true side (size only, because a small share of those runs point the other way and are counted separately). Significance is a filter, and a filter that only passes the luckiest runs must overstate what it passes. Gelman and Carlin gave the two failures names: Type M, the exaggeration of magnitude, and Type S, getting the sign itself wrong.
At a true 52% and 100 flips the numbers stop being subtle. Power is 5.0%, barely more than the 5% you would get from a coin with no bias at all. The surviving results overstate the bias by 6.25 times. And 12.2% of them report that the coin favours tails: one "discovery" in eight points the wrong way. Set the dials there and look.
Both problems have one cure, and it is the cure card 2 was already asking for. Flips. Here is what 80% power actually costs, computed exactly:
| a coin that is truly | flips for 80% power | exaggeration there |
|---|---|---|
| 70% heads | 49 | 1.112× |
| 60% heads | 199 | 1.120× |
| 55% heads | 786 | 1.124× |
| 52% heads | 4,927 | 1.124× |
| 51% heads | 19,648 | 1.125× |
The last column is the quiet one. Get to 80% power and the exaggeration nearly vanishes, whatever the effect: it is not the size of the bias that corrupts the estimate, it is the shortage of flips. An underpowered experiment is not merely a weak one. It is a machine that fails most of the time and lies about the size of the effect when it succeeds.
The check — show the working
Both p-values on the simulator are recomputed live, exactly. The green figure is the two-sided binomial tail — the probability mass of every outcome at least as far from an even split as yours — summed from exact binomial probabilities built up by the recurrence P(k+1) = P(k)·(n−k)/(k+1) (no normal approximation, no library). The orange figure is the Monte-Carlo estimate from the histogram you watched; it converges to the green one, which is the point: a p-value is a counted tail-fraction.
The three anchor numbers — ≈29% (Sellke–Bayarri–Berger), ≈2,400 flips for a 52% coin, ≈92% miss-rate for a 55% coin at n=50 — are each recomputed from first principles, offline, by verify-p-value.mjs (notebook in research/p-value/), and the verifier also drives this page headless to confirm the default reads 0.0569 and that the live exact value matches its own binomial sum to 6 decimals.
The power instrument. Power, the exaggeration ratio and the wrong-direction rate are exact sums over the rejection region, not simulations. Because the two-sided p falls as an outcome moves away from an even split, the rejection region is always one critical half-width, found once per setting; power is then the bent coin's probability mass inside it, the exaggeration is E[|p̂ − ½| | significant] / |p − ½|, and the wrong-direction rate is the share of that mass landing on the far side of ½. Those are Gelman and Carlin's Type M and Type S. The 80% power thresholds in the table are the smallest n reaching 80%, which needs care: power rises with n but not monotonically, because the rejection region moves in integer jumps, so a binary search alone overshoots (for a 55% coin it lands on 807 when the true answer is 786). research/p-value/power.mjs brackets by doubling, binary-searches, then scans downward, and the reported values do not move when the scan window is widened to 2,000.
This page shipped on 2026-06-24 with a real arithmetic defect, found while the power instrument above was being built, and it is worth stating plainly because the page is about not being fooled by a number. The binomial was built in linear space starting from 0.5n. That value is exactly zero for n ≥ 1075, because the smallest positive double is 2−1074, so every probability derived from it collapsed to zero and the p-value printed 0. Card 2's flips slider runs to 6,000, which put 247 of its 300 positions inside the broken range, and a zero reads as p < 0.05, so the badge said significant. At the slider's maximum a 50.5% coin was called significant when its true two-sided p is 0.446. The card's own claim that a 52% coin needs about 2,400 flips was correct and was the one place the reader could not check it, because the instrument showed p = 0 there.
The recurrence is now carried in logs, where nothing underflows, and is checked against exact BigInt rational arithmetic (no floating point at all) at every n the sliders reach. The old verifier could not have caught this: its large-n assertion was that a 52% coin at n=6,000 comes out significant, which the broken zero satisfied. The check now compares the page's live value against an independent reference across the whole slider range instead of asserting a verdict.
The conventions, named honestly. "Two-sided" has more than one definition; with a fair-coin null (p₀ = ½) the distribution is symmetric and they all agree, so the figure is unambiguous — we say so rather than hiding the choice. The 29% bound assumes a point null and equal prior odds and is a minimum over a class of alternative priors (real posteriors are often higher). The "2,400 flips" figure assumes you observe exactly the expected count, the cleanest case; real samples scatter. These are conventions of inference, not measurements of the world — flagged so you can see exactly what is claimed.