This blog post summarizes a conversation with ChatGPT about the persistence of a useless way to analyze data known as null-hypothesis significance testing (NHST) that is still taught to undergraduate students, despite its contribution to the replication crisis in psychology.

How Confidence Intervals Solve the Incentive Problem of NHST

For nearly a century, Null-Hypothesis Significance Testing (NHST) has shaped the way psychological and biomedical sciences treat data. The ritual is well known: postulate a null hypothesis of no effect, collect data, calculate a p-value, and reject the null if p < .05. This procedure is simple, but its simplicity has always come at a price. NHST fosters an asymmetric incentive system: significant results are treated as scientific “wins,” while non-significant ones are dismissed as “inconclusive” and quietly disappear. This “heads I win, tails don’t count” structure has fueled publication bias and contributed to the replication crisis.

---

The Problem with NHST

NHST promises more than it delivers. Rejecting the strict null hypothesis that an effect is exactly zero is rarely a meaningful discovery; we already know most effects are not literally zero. The real question is how large the effect is, and whether it is large enough to matter. At the same time, NHST leaves researchers empty-handed when p > .05: the result is not counted as evidence for the null, only as “inconclusive.” This allows researchers to ignore unfavorable results, try again with a new sample or outcome, and publish only the “wins.”

This asymmetry is why defenders of questionable claims could dismiss replication failures as “inconclusive,” leaning on NHST logic. Even large-sample replication studies that strongly disconfirmed exaggerated original effects were framed as yielding “no evidence,” based on the faulty logic of NHST that p > .05 implies a study was a failure.

---

The Directional Defense — and Its Limits

Some have argued that NHST is not completely useless: it can serve as a tool for directional inference. As Abelson and Tukey noted, even with noisy data, rejecting H0:d=0H_0: d=0 tells us something about the sign of the effect. If the test is significant and the observed effect is negative, we may infer the population effect is negative. This saves NHST from complete vacuity, but it is still unsatisfying. Direction is only one part of the story; science progresses by narrowing the range of plausible effect sizes, not merely by declaring them positive or negative.

---

The Bayesian Proposal — and Its Limitations

In response to NHST’s asymmetry, Bayes Factors (promoted especially by Wagenmakers and colleagues) were offered as a more balanced alternative. Unlike NHST, Bayes Factors can provide relative evidence for both H0​ and H1​. A replication failure, for example, can yield a Bayes Factor that favors H0​, allowing researchers to claim support for the null — something NHST never permits.

Yet this approach introduces its own problems. Evidence for H0 is always relative to the specification of H1​. Most Bayes Factor applications use a diffuse prior for H1 (e.g., the JZS Cauchy prior centered on zero). If the observed effect is smaller than expected under this prior, the Bayes Factor tilts toward H0​. But this is really evidence against that specific diffuse H1, not absolute evidence for the null. A different prior for H1​ might shift the conclusion.

The danger is rhetorical: Wagenmakers often frames results as “evidence for the null,” when technically they are better described as rejections of a particular H1​. This is safer but less rhetorically powerful. The Bayesian solution avoids NHST’s asymmetry but inherits a prior-dependence problem: conclusions about the null depend on how generously or narrowly you defined the alternative.

---

Why Confidence Intervals Are Better

Confidence intervals (CIs) directly address the problems of NHST. Instead of asking whether we can reject a single point value (usually zero), CIs present the range of values consistent with the data at a chosen level of confidence. This has three advantages:

1. Losses Count. With CIs, both small and large hypothesized effects can be rejected. If a replication study finds CI = [–0.05, 0.10], it not only fails to support an original claim of d=.60, it actively rules it out. Unlike NHST, the result is not “inconclusive.”
2. Focus on Magnitude and Uncertainty. A CI makes the sampling error explicit. We can immediately see whether the data are too noisy to be informative, or whether they constrain the effect size to a narrow and meaningful range.
3. Flexibility Across Hypotheses. CIs allow us to test any hypothesis — not just zero. If CI = [0.05, 0.12], we can reject both “no effect” and “large effect” claims. This prevents the common hand-waving of “the effect is small but important” when the entire interval suggests trivial magnitudes.

---

The Replication Crisis as Evidence

Large replication projects demonstrate the value of CIs. Original small-N studies often produced point estimates suggesting moderate-to-large effects. Replications with much larger samples yielded narrow CIs close to zero. Even when they did not include zero, they ruled out that the original effect sizes were plausible. To describe such results as “inconclusive” (because p > .05) is deeply misleading. The relevant information is not the p-value but the fact that the replication rules out the very effects the original studies seemed to establish.

---

Moving Forward

The way to fix the incentive structure exposed by the replication crisis is not to rebrand NHST, but to adopt estimation thinking as the default. Teaching should start not from the ritual of null rejection, but from the two pillars of statistics:

• estimation of effect sizes, and
• quantification of their uncertainty.

Confidence intervals embody both. NHST can still be taught, but only as a derivative, limited tool — a special case where we ask whether zero is inside the interval. By contrast, the primary focus should be on interpreting the entire interval and deciding whether the plausible range of effects is scientifically meaningful.

---

Conclusion

NHST thrives because it creates asymmetry: positive results are rewarded, negatives are hidden. Confidence intervals solve this incentive problem. They make losses count by ruling out implausible hypotheses, whether those are strict nulls or inflated claims. They shift attention from the ritual of significance to the substance of science: what do the data tell us about the size of effects, and how certain can we be?

---