How many samples are enough?

Short answer: it depends on which number you care about! The median calms down after about 100 samples. The 10th percentile, the number that tells you who's having a rough time, needs 2,000 to 3,000 before it behaves as well. That's 20 to 30 times the samples for two numbers computed from the exact same data, and the reason why is one of my favorite little bits of statistics. I'll use speed tests as the example, but the same math applies to signal readings, latency pings and pretty much anything you measure one sample at a time.

Let's assume there's a city broadband office, and someone there sends over a screenshot of 12 speed tests from one neighborhood. Three of them are under 25 Mbps. The email asks one question: is this neighborhood underserved?

To answer that properly, we need a city where we already know the truth. So let's give this one an internet personality we know inside and out. Most tests come from homes on a fast connection, centered around 250 Mbps. About 15% come from a slow tail centered around 25 Mbps, think old DSL or a laptop two walls away from the router. Across millions of tests, the true median is 226 Mbps and the true 10th percentile is 34 Mbps. Knowing the right answer is the whole trick, because now we can measure exactly how wrong a small sample gets.

Here's the experiment. Grab 10 random tests from that city, work out the median and the 10th percentile, and write them down. Throw those back, grab another 10, and do it again, 2,000 times in all. Then repeat the whole thing with 20 tests, 30, and on up to 5,000. This is called a Monte Carlo simulation, named after the casino, because it's basically rolling dice over and over until a pattern shows up. My laptop rolled more than 22 million simulated tests for this post and didn't complain once.

The median settles fast

With only 10 tests, the median is all over the place. In 90% of my draws it landed somewhere between 154 and 310 Mbps, and the typical miss was about 14%. Same city, same internet, wildly different answers. That's why one speed test, or even a handful, can't tell you a place has bad internet.

At 100 tests things calm down a lot. The middle 90% of draws sat between 202 and 251 Mbps, and the typical miss dropped to under 5%. By 1,000 tests it's 218 to 234 Mbps, off by about 1.4%. Pretty cool how quickly that tightens up!

The 10th percentile takes its time

The 10th percentile needs about 30 times the samples

Typical miss from the true value, by samples per draw. 2,000 draws at each size.

Median10th percentile

Simulated city: 85% of tests from a fast group (median 250 Mbps), 15% from a slow tail (median 25 Mbps). Typical miss = median of |estimate − truth| ÷ truth. Code: data-stories/sample-size, seed 42.

The 10th percentile is a lot jumpier. With 10 tests, the typical miss was 168%. That sounds impossible until you think about what 10 tests means here. With only 15% of tests coming from the slow tail, about one sample in five (0.85 to the 10th power, roughly 20%) has no slow tests in it at all. Those samples never see the people having a rough time, so their 10th percentile lands way up in the fast crowd.

It keeps wobbling for a long time. At 100 tests the typical miss is still about 25%. At 1,000 it's 8%. It takes about 2,000 tests to get close to the steadiness the median had at 100.

That makes sense once you say it out loud. The median gets to use the whole crowd. The 10th percentile lives in the thin end of the distribution, so most of your tests tell it nothing new. You can watch it happen:

10 samples per round

Each round grabs fresh samples from the same city and leaves a tick where its median and 10th percentile landed. A tight pile of ticks means a number you can trust. Download speed, Mbps (log scale).

This round's samplesMedian10th percentile

Median · middle 90% of rounds–true value 226 Mbps

10th percentile · middle 90% of rounds–true value 34 Mbps

Simulated city, same settings as simulate.py: 85% of samples from a fast group (median 250 Mbps), 15% from a slow tail (median 25 Mbps). Up to 40 rounds shown.

Different questions need different amounts of data

So "how many tests is enough?" really depends on the question. Here's what our city says for a few common ones. "Steady" here means the typical miss is around 5% or less.

The question	The number to watch	Samples for a steady answer
What does a typical test look like?	Median	~100
How fast is the fast end?	90th percentile	~100 to 200
Who's having a rough time?	10th percentile	~2,000 to 3,000
What share of tests come in under 25 Mbps?	A share (7.5% here)	~1,000 (lands between 6% and 9%)
Is town A faster than town B?	Median in each	~300 per town
Did the slow end get worse?	10th percentile, before and after	~2,000 per period

Comparisons are where people get caught out. Let's assume a second city whose median is about 10% lower (204 vs 226 Mbps). With 100 tests from each, you'd put them in the right order only 86% of the time. At 300 each it's 97%. Asking the same thing about the slow end is much harder. If the slow tail gets 20% slower, the 10th percentile only drops from 34 to 27 Mbps, and 300 tests per city gets the order right just 75% of the time. That's one wrong call in four, delivered with total confidence. It takes about 2,000 per city to reach 96%.

Slicing eats your sample, too. Say you have 1,000 tests and want to see them by hour of the day. That's about 42 tests an hour, so each hour's median is off by around 7% and each hour's 10th percentile by something like 40%. Slice by day of the week and hour and you're down to about six tests per slice. At that point the chart is mostly a very pretty random number generator.

Why the 10th percentile is so stubborn

Time to put on the nerd glasses. If formulas aren't your thing, skip to the end and I promise I won't take it personally.

First, the square root. The wobble in almost any estimate shrinks with the square root of the number of tests. I think of it as the universe's tax on certainty. Four times the tests only halves the error, so going from 100 to 400 tests buys the same improvement as going from 1,000 to 4,000. That's why the curves flatten out: every extra bit of steadiness costs more tests than the last one did.

error \propto \frac{1}{\sqrt{n}}

Second, how crowded the data is where you're measuring. Statisticians have a handy approximation for the standard error of any percentile:

SE ({\hat{q}}_{p}) \approx \frac{1}{f (q_{p})} \sqrt{\frac{p (1 - p)}{n}}

Here p is the percentile as a fraction (0.5 for the median, 0.1 for the 10th), n is the number of samples, and f(q_p) is the density of samples right at that value. Density is the part that matters. If lots of samples sit near a value, a few extra barely move it. If the data is thin there, each new sample can shove the estimate a long way.

In our city, the median sits in the middle of the fast crowd, where tests are packed in tight. The 10th percentile lands in the thin valley between the slow tail and the fast crowd, right where the two groups barely overlap. It's the loneliest spot in the whole distribution. The p(1 − p) part actually helps the 10th percentile a little (0.09 against 0.25 for the median). Density doesn't. Divide the standard error by the value itself and multiply by √n, and you get 0.68 for the median and 3.84 for the 10th percentile. So for the same number of tests, the 10th percentile is about 5.7 times as uncertain.

Put those together. Because error shrinks with √n, shrinking it 5.7 times takes 5.7 squared, or about 32 times, the samples:

n_{p10} \approx n_{median} \times {(\frac{3.84}{0.68})}^{2} \approx 100 \times 32 = 3,200

That's right where the simulation landed, which made me happier than it probably should have.

There's a plainer way to see it, too. Only about one test in ten falls below the 10th percentile. With 100 tests, roughly ten of them are doing all the work. With 10 tests, it's one.

Real data makes all of this harder. The math assumes every sample is an independent draw, and real samples cluster. I think about it the way I think about cell coverage: pepperoni on a pizza. You want the pepperonis to touch with no overlap, so the whole pie is covered. Stack ten of them in one spot and you've still only covered one bite. One household running several tests in a row is a stack, and so is one busy spot holding a big share of an area's tests (I ran into exactly that in Texas). Statisticians call the number of samples that really count the effective sample size, and stacks shrink it. So I'd treat the numbers above as a floor.

What I took from it

Before I trust a number, I ask which number it is. A median from a few hundred tests is usually solid. A 10th percentile from the same few hundred is a decent first guess, and I'd show its range before I'd show it as a single value.

The real world adds its own wrinkles. Tests aren't spread evenly across homes or hours, and a real city's slow tail might be bigger or smaller than the 15% we assumed. So treat 100 and 2,000 as the shape of the answer, and run the simulator with your own numbers before you lean on them.

Back to that email with the 12 tests. In the city we assumed, only 7.5% of tests come in under 25 Mbps, and three slow tests out of 12 still shows up about one time in 18. So my reply would be friendly and a little boring: those three tests are worth a look, and they aren't a verdict yet. Collect a few hundred tests across different days and times of day, and then we can talk about the slow end with a straight face.

Run it yourself

Everything is in data-stories/sample-size. Run python3 simulate.py and it rebuilds the table behind the chart. No downloads needed. Try --seed 7 and you'll get different draws with the same overall shape. python3 use_cases.py reruns the numbers in the table above.

Sources: the city is simulated. Its settings live at the top of simulate.py.