Is It Real, or Just Luck?

They each pull up their tips from the last eight nights to settle it. Maria does have the higher average — but Sam has a point too. Eight nights isn’t many. Couldn’t a gap that small just be the luck of the draw? How would you ever know?

Here’s every night, side by side. Maria’s smiley nights run a bit higher — her average beats Sam’s by $4.25 a night.

Let’s take Sam’s side for a moment and assume the smiley does nothing. If that’s true, then the labels “smiley” and “no smiley” are meaningless stickers — every night’s tip would’ve been the same no matter who earned it.

So let’s test that world. Toss all sixteen nights into a hat, shuffle, and randomly deal eight to a pretend “Maria” and eight to a pretend “Sam.” Measure the gap. That gap is pure luck — we just made the labels up. Do it thousands of times and we map out exactly how big a gap luck alone tends to cook up.

Then the only question left: is Maria’s real $4.25 gap a normal, everyday luck-gap… or a freak one that luck almost never produces?

Before you shuffle: what’s your gut say? Is the smiley real, or did Maria just have a good week?

Each shuffle is one imaginary world where the smiley does nothing. Build up a few thousand. The bars show every luck-gap; the teal line is Maria’s real one.

Look at that shaded slice — the shuffles where blind luck produced a gap as big as Maria’s. It’s tiny. Out of thousands of “it’s just luck” worlds, only a sliver ever reached $4.25.

That fraction has a name: the p-value. It’s simply how often luck alone fakes a result this big. When it’s small, luck is a poor explanation — so something real is probably going on. Maria wins the argument.

Start with the thing we pretended to believe: the null hypothesis. It just says the smiley does nothing — Maria’s nights and Sam’s nights are drawn from the same pile, and the labels are decoration. Everything we did was a way to give that boring world a fair hearing.

To measure the argument we need a single number — a test statistic. Ours is the obvious one: the observed difference in group means, $\Delta = \bar{x}_\text{smiley} - \bar{x}_\text{none}$, Maria’s average minus Sam’s. That came out to $4.25. Big numbers are suspicious for the null; near-zero numbers are comfortable for it.

Now the clever part. If the labels truly don’t matter, then any relabeling of the sixteen nights is just as legitimate as the real one — every shuffle is equally likely. So we shuffle, recompute the difference, and write it down. Repeat thousands of times and the pile of fake gaps traces out exactly what pure chance tends to produce. That pile is the null distribution.

The p-value is then just a count: of all those shuffled worlds, what fraction reached a gap as big as the one we actually saw?

If this smells like a t-test, it’s a cousin — but a more honest one here. The t-test assumes the data follow a tidy bell curve; the permutation test assumes no such thing and simply resamples the data you actually have.

You just ran a permutation test — and the exact same shuffle settles far bigger arguments. Did the new checkout button really boost sales, or was it a good week? Did the new drug actually beat the placebo? Did the redesign lift sign-ups? Every time, it’s the same hat: pool the results, shuffle the labels thousands of times, and ask how often luck alone could fake what you saw. No formula required — just the courage to shuffle.