Ever tested something and felt sure it worked — but the numbers said "not so fast"? That gap between what we feel and what the data shows is where hypothesis testing lives. And honestly, it trips up more people than they'd like to admit.
The short version is this: knowing when do we reject a null hypothesis is the difference between spotting a real effect and fooling yourself with noise. Let's talk about how that actually works in practice, not just in a textbook.
What Is Rejecting a Null Hypothesis
Picture a court case. The defendant is assumed innocent until proven guilty. In stats, that "innocent" default is called the null hypothesis — usually written as H₀. It says nothing interesting is happening. No difference, no effect, no change. The alternative — H₁ or Hₐ — is what you're really hoping to find: the new drug works, the funnel change boosts signups, the teaching method helps That's the whole idea..
Rejecting a null hypothesis means the evidence was strong enough that we stop pretending "nothing's going on" is plausible. We don't prove the alternative is true beyond all doubt. We just say the data makes the null hard to defend.
Here's the thing — rejection isn't a verdict of "the null is false." It's a verdict of "we don't have room for it anymore given what we observed."
The Null Isn't Your Enemy
A lot of beginners talk about the null like it's a villain to defeat. It isn't. That said, the null is a yardstick. That's why it gives you a baseline so you can measure surprise. Without it, every bump in your dashboard looks like a miracle No workaround needed..
What "Reject" Really Means
In plain language, reject means: the result you got would be rare if the null were true. Also, rare enough that you'd rather bet the null is wrong than believe you just hit a 1-in-20 unlucky streak. That "rare enough" cutoff is the famous significance level.
Why It Matters
Why does this matter? Because most people skip the logic and jump to "p < 0.05, ship it." That's how teams waste months on changes that never moved the needle.
In practice, getting the rejection call wrong cuts both ways. Reject a null when you shouldn't — that's a false positive, or Type I error. You launch a "winning" variant that was actually random luck. Don't reject when you should — that's a false negative, or Type II error. You kill a real improvement because you didn't listen to the data Worth keeping that in mind. Simple as that..
Turns out, the cost of each mistake depends on the field. Day to day, a pharma company shipping a fake drug hurts people. A blogger A/B testing headline emoji mostly wastes an afternoon. But the mechanics are the same That's the part that actually makes a difference..
Real talk: understanding when to reject keeps you honest. It's the line between "I found something" and "I found something worth believing."
How It Works
So how do we actually decide? Here's the path most tests follow.
Set Your Significance Level First
Before you look at results, pick α (alpha) — the max risk of a false positive you'll accept. But you choose it up front. That means you're okay being wrong 5% of the time if the null really is true. Which means 10. Some use 0.Here's the thing — 01. The default is 0.Some fields use 0.05. Not after you peek That's the part that actually makes a difference. Turns out it matters..
Run the Test and Get a p-Value
The p-value answers one question: if the null were true, how likely is a result this extreme (or more)? A tiny p means "this would be weird under the null." A big p means "eh, happens all the time That's the part that actually makes a difference..
Look — a p of 0.03 says: assuming nothing's happening, a difference this big shows up 3% of the time by chance. That's less than your 5% line. So the null looks shaky.
Compare p to Alpha
This is the actual decision rule. If p ≤ α, reject H₀. If p > α, fail to reject H₀. Which means notice the wording — we never "accept" the null. We just don't have enough to ditch it Most people skip this — try not to..
Check the Test Statistic Too
The p-value comes from a test statistic — a t-score, z-score, F, chi-square, whatever fits. Even so, each has a cutoff from a distribution. Think about it: if your statistic passes the critical value, same conclusion: reject. The p-value is just the smooth version of that old critical-value method.
Don't Ignore Power and Sample Size
A clean rejection needs enough data. Which means small samples make everything noisy. Consider this: you can do everything right and still fail to reject because your test was underpowered. Worth knowing: a non-significant result is often "we don't know," not "nothing there Small thing, real impact..
One-Tailed vs Two-Tailed
If you only care about a change in one direction, a one-tailed test gives you more sensitivity — but you must commit before seeing data. " questions. That's why two-tailed splits the alpha on both sides and is safer for "did anything change? Most people use two-tailed by default.
This changes depending on context. Keep that in mind.
Common Mistakes
Here's what most guides get wrong. Still, they act like p < 0. Think about it: 05 is a magic gate. It isn't.
Mistake 1: Treating "fail to reject" as proof the null is true. No. It means your data didn't clear the bar. The effect might be small, your sample tiny, your test wrong.
Mistake 2: Peeking and re-running until significant. You run a test Monday, not significant. Wednesday, still no. Friday it dips under 0.05 so you stop. That's p-hacking. The real error rate is now way above your alpha.
Mistake 3: Ignoring effect size. A result can be "significant" and meaningless. Huge sample, tiny lift — statistically reject the null, practically who cares? Rejecting tells you something happened, not that it matters No workaround needed..
Mistake 4: Using 0.05 like a law of nature. Fisher never meant it as a bright line. It's a convention. Some contexts deserve stricter, some looser.
Mistake 5: Believing rejection means causation. You rejected "no difference" in your experiment. Great. But if the design was a messy observational dump, the cause is still a guess.
I know it sounds simple — but it's easy to miss these in the moment, especially with a dashboard breathing down your neck.
Practical Tips
What actually works when you're standing in front of real data?
- Pre-register your plan. Write the hypothesis, the alpha, the test, the sample size. Then run it. You'll save yourself from "oops I tested 12 segments" syndrome.
- Report the confidence interval, not just the p. If the interval for the lift is 0.2% to 4%, you rejected the null and you know the size. If it's 0.001% to 0.003%, congratulations, but maybe don't celebrate.
- Use power analysis before you start. Tools exist. Plug in the smallest effect you'd care about and see how many users you need. Then wait for that many.
- Look at the raw distribution. A single p-value can hide weird skew, bot traffic, or a broken tracking pixel. Rejecting a null on garbage data is just expensive garbage.
- Say "we rejected H₀" out loud. If that sentence feels weird given the context, pause. It should feel like a specific, modest claim — not a victory lap.
And honestly? Still, the best analysts I've read treat rejection as a starting point. "Okay, null's out — now what's the real size, and does anyone care?
FAQ
When do we reject a null hypothesis in plain terms? When the p-value is less than or equal to your pre-chosen significance level (usually 0.05), meaning the observed result would be too unlikely if the null were true Still holds up..
Can you reject the null if p is exactly 0.05? Yes. The rule is p ≤ α, so 0.05 with α = 0.05 is a rejection. But it's the thinnest edge — don't act like it's a slam dunk Surprisingly effective..
What if I reject the null but the effect is tiny? You've shown a real difference, statistically. Whether it's useful is a separate question. Always check effect size and context before deciding what to
do something about it. Statistical significance is the floor, not the ceiling.
Does a non-significant result mean the null is true?
Nope. It just means you didn't have enough evidence to reject it. Maybe there's no effect, or maybe you needed more data. A non-significant result is inconclusive — not a verdict.
What's the one thing I should always check first?
The confidence interval. It tells you the range of plausible values for the true effect. If it includes zero, you can't reject the null. If it doesn't, you can — but more importantly, it shows you how big the effect might actually be.
Final Thought
Hypothesis testing is a tool, not a oracle. It can tell you whether something looks real or is likely due to chance, but it's up to you to decide whether that something matters.
The goal isn't to get more asterisks in a results table. Consider this: it's to make better decisions with uncertain data. Sometimes that means rejecting a null hypothesis. Often it means holding off on conclusions, collecting more data, or asking a different question entirely Most people skip this — try not to. Simple as that..
So before you click "publish" on your next analysis, take a breath. Check your assumptions, question your shortcuts, and remember that the most important part of any statistical test happens before you even look at the p-value.
The data doesn't care about your deadline or your dashboard. Worth adding: it just is what it is. Your job is to listen carefully, think clearly, and resist the urge to hear what you want to hear.
