Here's a puzzle. You have 3,500 speed tests from one town. Last time we found out the median settles down after about 100 tests, so 3,500 should be total overkill. And yet the answer you get is off by 54%.
How? Nobody lied, no test was broken, and every number in the pile is real. The problem is who did the testing.
The group chat problem
Picture a group chat of ten friends picking a restaurant. Nine of them vote once. The tenth sends forty messages that just say "TACOS." Count messages and tacos win in a landslide. Count people and it's a completely different dinner.
Count messages, and tacos win
Ten friends pick a restaurant. Nine vote once. One of them really wants tacos.
Speed tests work the same way. Let's assume there is a town of 1,000 homes, the same town from How many samples are enough? Most homes are on a fast connection around 250 Mbps, and about 15% are on a slow one around 25 Mbps.
Now give the town a month of testing. About 230 homes never run a single test, and most of the rest run one to three. Then there are 51 homes that run 40 tests each.
Those 51 are our taco friends: the power-user households. Let's assume there is a house with a gamer, a home office and someone who runs a speed test every time a video buffers. (No comment on whether that's my house.) Let's also assume these homes pay for a faster plan, about twice their neighbors' speed, because people who care about speed tend to buy more of it.
Count the messages, get the wrong dinner
Those 51 homes are 5% of the town. They ran 59% of the tests. Here's what that does:
| What we count | Median | Share under 25 Mbps |
|---|---|---|
| Every home in town (the truth) | 241 Mbps | 9.0% |
| Every test, one vote per test | 372 Mbps | 4.0% |
| Every home that tested, one vote per home | 253 Mbps | 8.3% |
Count every test and the town looks 54% faster than it really is, while the slow end shrinks to half its real size. Tacos win.
51 homes, 59% of the tests
Every house is one of the town's 1,000 homes, sized by how many tests it ran this month. Switch to counting homes and every home gets the same say.
Here's the part that surprised me. More tests don't help. Remember the wobble from the last post, the one that shrinks as you add samples? This is a different beast. I ran the town 2,000 times, and the every-test median came out high every single time, by 59% on a typical run. Adding more tests just adds more taco messages. Statisticians call this bias, and it loves to hide behind a big sample size.
One home, one vote
The fix is the group chat fix: count people instead of messages. Squash each home down to one number, its own median, then take the median across homes.
That gives 253 Mbps against a true 241. Across the 2,000 runs, the one-vote median landed within about 4% of the truth 90% of the time.
Wait, is that actually allowed?
I asked myself the same thing. It can look like I just picked whichever rule gave the nicer answer. So here's why statisticians really do it this way, simple version first.
The trick is to ask what your question is about. "How fast is the internet in this town?" is about homes. So the home is the thing you count, and tests are just how you peek at each home. Statisticians call that the unit of analysis. When your samples arrive in clumps, forty from one house and two from another, it's called cluster sampling.
Two ideas make it work.
Idea 1: turn down the loud ones. A home with 40 tests had 40 times the chances to land in your pile. So each of its tests gets 1/40 of a vote, and every home ends up with exactly one vote no matter how chatty it was. That's called inverse probability weighting. In our town it gives 244 Mbps, right next to the truth.
Idea 2: copies aren't new information. Test number 40 from the gamer house tells you almost nothing that tests 1 through 39 didn't. It's the pepperoni again: forty slices stacked on one spot still only cover one bite. Statisticians measure this with the design effect, and in our town it says 3,483 tests are worth about 800 independent ones. We had 767 homes. So the math lands right where common sense does: you have about as many real samples as you have homes.
That second idea has a catch. 767 homes is plenty for a median, but the last post said the 10th percentile wants a couple thousand, and forty tests from one house won't get you closer.
One more thing so I don't oversell it. One vote per home is right when the question is about homes. If the question is "what does a typical speed test look like?", every test should count. The mistake is mixing them up, taking a number counted by tests and reading it as a story about homes.
Nerd corner
If you want the actual formulas, here they are. Skip ahead and nothing bad happens.
Each test i gets a weight of one over m, the number of tests from its home. This is the Horvitz-Thompson idea, if you want something to search for.
m̄ is the average number of tests per home (4.5 here), and ρ is the intraclass correlation: how alike tests from the same home are compared with tests from different homes. Ours is 0.95, so the design effect is about 4.3, and 3,483 ÷ 4.3 is about 800.
The tests nobody ran
There's one more way samples go missing, and it's the sneakiest. You can't run a speed test when your internet is down. A day offline leaves zero trace in the tests, so every test you have comes from a moment when things worked well enough to run one.
This is survivorship bias, the famous World War II planes story: the bullet holes you can count are on the planes that made it home. I dug into exactly this with Hurricane Beryl, where the gap in the data was the story.
And those 233 homes that never tested? If they're quiet because everything works, great. If they're quiet because they gave up, they're the ones you most want to hear from.
The takeaway
The last post was about how many. This one is about which ones. Before I trust a pile of samples now, I ask how many homes are behind it and whether a few of them did most of the talking. A huge pile from the wrong places just gives you a very confident wrong answer.
Next up, deciding what counts as one vote turns out to be a normalization question, and that gets its own post. Those power-user homes are interesting enough that I'll come back to them too.
Run it yourself
Everything is in data-stories/sampling. Run python3 sampling.py and it builds the town, runs the tests and prints the table above, plus the weighted median, the design effect and the 2,000-town check. Try --seed 7 for different draws with the same shape, or change POWER_SHARE at the top and watch the every-test median drift.
Sources: the town is simulated. Its settings live at the top of sampling.py, with the same speed mix as sample-size/simulate.py.