Ever tested something and felt sure it worked — but the numbers said "not so fast"? That gap between what we feel and what the data shows is where hypothesis testing lives. And honestly, it trips up more people than they'd like to admit And it works..
The short version is this: knowing when do we reject a null hypothesis is the difference between spotting a real effect and fooling yourself with noise. Let's talk about how that actually works in practice, not just in a textbook.
What Is Rejecting a Null Hypothesis
Picture a court case. No difference, no effect, no change. Even so, it says nothing interesting is happening. Practically speaking, the defendant is assumed innocent until proven guilty. Also, in stats, that "innocent" default is called the null hypothesis — usually written as H₀. The alternative — H₁ or Hₐ — is what you're really hoping to find: the new drug works, the funnel change boosts signups, the teaching method helps Practical, not theoretical..
You'll probably want to bookmark this section The details matter here..
Rejecting a null hypothesis means the evidence was strong enough that we stop pretending "nothing's going on" is plausible. That's why we don't prove the alternative is true beyond all doubt. We just say the data makes the null hard to defend.
Here's the thing — rejection isn't a verdict of "the null is false." It's a verdict of "we don't have room for it anymore given what we observed."
The Null Isn't Your Enemy
A lot of beginners talk about the null like it's a villain to defeat. It isn't. The null is a yardstick. Practically speaking, it gives you a baseline so you can measure surprise. Without it, every bump in your dashboard looks like a miracle.
What "Reject" Really Means
In plain language, reject means: the result you got would be rare if the null were true. In practice, rare enough that you'd rather bet the null is wrong than believe you just hit a 1-in-20 unlucky streak. That "rare enough" cutoff is the famous significance level.
Why It Matters
Why does this matter? Even so, because most people skip the logic and jump to "p < 0. Here's the thing — 05, ship it. " That's how teams waste months on changes that never moved the needle.
In practice, getting the rejection call wrong cuts both ways. Reject a null when you shouldn't — that's a false positive, or Type I error. You launch a "winning" variant that was actually random luck. Don't reject when you should — that's a false negative, or Type II error. You kill a real improvement because you didn't listen to the data Simple, but easy to overlook..
Turns out, the cost of each mistake depends on the field. A pharma company shipping a fake drug hurts people. A blogger A/B testing headline emoji mostly wastes an afternoon. But the mechanics are the same.
Real talk: understanding when to reject keeps you honest. It's the line between "I found something" and "I found something worth believing."
How It Works
So how do we actually decide? Here's the path most tests follow.
Set Your Significance Level First
Before you look at results, pick α (alpha) — the max risk of a false positive you'll accept. 05. So that means you're okay being wrong 5% of the time if the null really is true. Some fields use 0.But you choose it up front. 01. Some use 0.The default is 0.10. Not after you peek.
Short version: it depends. Long version — keep reading Not complicated — just consistent..
Run the Test and Get a p-Value
The p-value answers one question: if the null were true, how likely is a result this extreme (or more)? A tiny p means "this would be weird under the null." A big p means "eh, happens all the time No workaround needed..
Look — a p of 0.03 says: assuming nothing's happening, a difference this big shows up 3% of the time by chance. That's less than your 5% line. So the null looks shaky Most people skip this — try not to..
Compare p to Alpha
This is the actual decision rule. If p ≤ α, reject H₀. If p > α, fail to reject H₀. In practice, notice the wording — we never "accept" the null. We just don't have enough to ditch it Most people skip this — try not to..
Check the Test Statistic Too
The p-value comes from a test statistic — a t-score, z-score, F, chi-square, whatever fits. So each has a cutoff from a distribution. If your statistic passes the critical value, same conclusion: reject. The p-value is just the smooth version of that old critical-value method Small thing, real impact..
This is where a lot of people lose the thread.
Don't Ignore Power and Sample Size
A clean rejection needs enough data. Still, small samples make everything noisy. You can do everything right and still fail to reject because your test was underpowered. Worth knowing: a non-significant result is often "we don't know," not "nothing there But it adds up..
One-Tailed vs Two-Tailed
If you only care about a change in one direction, a one-tailed test gives you more sensitivity — but you must commit before seeing data. Two-tailed splits the alpha on both sides and is safer for "did anything change?" questions. Most people use two-tailed by default Small thing, real impact. Took long enough..
Common Mistakes
Here's what most guides get wrong. Practically speaking, 05 is a magic gate. Here's the thing — they act like p < 0. It isn't.
Mistake 1: Treating "fail to reject" as proof the null is true. No. It means your data didn't clear the bar. The effect might be small, your sample tiny, your test wrong.
Mistake 2: Peeking and re-running until significant. You run a test Monday, not significant. Wednesday, still no. Friday it dips under 0.05 so you stop. That's p-hacking. The real error rate is now way above your alpha.
Mistake 3: Ignoring effect size. A result can be "significant" and meaningless. Huge sample, tiny lift — statistically reject the null, practically who cares? Rejecting tells you something happened, not that it matters.
Mistake 4: Using 0.05 like a law of nature. Fisher never meant it as a bright line. It's a convention. Some contexts deserve stricter, some looser.
Mistake 5: Believing rejection means causation. You rejected "no difference" in your experiment. Great. But if the design was a messy observational dump, the cause is still a guess Small thing, real impact. Which is the point..
I know it sounds simple — but it's easy to miss these in the moment, especially with a dashboard breathing down your neck.
Practical Tips
What actually works when you're standing in front of real data?
- Pre-register your plan. Write the hypothesis, the alpha, the test, the sample size. Then run it. You'll save yourself from "oops I tested 12 segments" syndrome.
- Report the confidence interval, not just the p. If the interval for the lift is 0.2% to 4%, you rejected the null and you know the size. If it's 0.001% to 0.003%, congratulations, but maybe don't celebrate.
- Use power analysis before you start. Tools exist. Plug in the smallest effect you'd care about and see how many users you need. Then wait for that many.
- Look at the raw distribution. A single p-value can hide weird skew, bot traffic, or a broken tracking pixel. Rejecting a null on garbage data is just expensive garbage.
- Say "we rejected H₀" out loud. If that sentence feels weird given the context, pause. It should feel like a specific, modest claim — not a victory lap.
And honestly? The best analysts I've read treat rejection as a starting point. "Okay, null's out — now what's the real size, and does anyone care?
FAQ
When do we reject a null hypothesis in plain terms? When the p-value is less than or equal to your pre-chosen significance level (usually 0.05), meaning the observed result would be too unlikely if the null were true.
Can you reject the null if p is exactly 0.05? Yes. The rule is p ≤ α, so 0.05 with α = 0.05 is a rejection. But it's the thinnest edge — don't act like it's a slam dunk Not complicated — just consistent..
What if I reject the null but the effect is tiny? You've shown a real difference, statistically. Whether it's useful is a separate question. Always check effect size and context before deciding what to
do something about it. Statistical significance is the floor, not the ceiling.
Does a non-significant result mean the null is true?
Nope. It just means you didn't have enough evidence to reject it. Maybe there's no effect, or maybe you needed more data. A non-significant result is inconclusive — not a verdict The details matter here..
What's the one thing I should always check first?
The confidence interval. It tells you the range of plausible values for the true effect. If it includes zero, you can't reject the null. If it doesn't, you can — but more importantly, it shows you how big the effect might actually be Simple, but easy to overlook..
Final Thought
Hypothesis testing is a tool, not a oracle. It can tell you whether something looks real or is likely due to chance, but it's up to you to decide whether that something matters Not complicated — just consistent. That's the whole idea..
The goal isn't to get more asterisks in a results table. It's to make better decisions with uncertain data. Sometimes that means rejecting a null hypothesis. Often it means holding off on conclusions, collecting more data, or asking a different question entirely Most people skip this — try not to. Surprisingly effective..
So before you click "publish" on your next analysis, take a breath. Check your assumptions, question your shortcuts, and remember that the most important part of any statistical test happens before you even look at the p-value.
The data doesn't care about your deadline or your dashboard. It just is what it is. Your job is to listen carefully, think clearly, and resist the urge to hear what you want to hear.
