Another blog post with my virtual co-author. Web searches suggested that z-curve has been criticized for ignoring the sign of z-tests. One possible argument might be that z-curve estimates are useless because they estimate the false discovery risk under the assumption the null-hypothesis is sometimes true. Some researchers have suggested that this is never the case. There is always a non-zero effect no matter how small. This blog post counters this argument and points out that false discovery risk also controls the sign error risk. So, next time somebody goes on and on and on about NHST and sign errors (you know who), just remember this blog post.
Understanding False Discovery Risk and Sign Error Risk in Z-Curve Analysis
Researchers often interpret statistically significant results as evidence that an effect is not only real but also in the predicted direction. But how trustworthy is this inference — and what are the risks of being wrong?
In meta-science, two key concepts help answer this question:
- False Discovery Risk (FDR) – the probability that a significant result reflects no real effect.
- Sign Error Risk (SER) – the probability that a significant result reflects an effect in the wrong direction.
Let’s unpack both — and explain how tools like z-curve help us evaluate them, even without using the direction (sign) of test statistics.
🔍 False Discovery Risk (FDR): Are We Chasing Ghosts?
The false discovery risk refers to the maximum proportion of significant results in a given set of studies that could be false positives — results that appeared significant despite the true effect being zero or negligibly small.
Z-curve estimates FDR by modeling the distribution of all test statistics, typically their absolute z-values, and inferring how much of the distribution is likely due to random noise rather than true signals. When publication bias is present (as it usually is), z-curve can still provide robust estimates of the expected discovery rate (EDR) — the proportion of all conducted tests that would be significant if there were no selection bias — and from that, the maximum FDR (using a derivation from Sorić’s formula).
A higher FDR means that a large portion of the literature may not be reliable, even if individual studies report p-values below .05.
↕️ What About Sign Error Risk (SER)?
Now imagine you’re not just asking whether there’s some effect — you’re betting on the direction of the effect. This is especially common in psychology, medicine, and economics, where directional claims are common: “X increases Y,” not just “X affects Y.”
This introduces another kind of risk: sign error risk — the probability that a significant result has the wrong sign.
Importantly, z-curve doesn’t use the sign of test statistics — it analyzes absolute z-values. Critics sometimes argue this makes z-curve blind to the risk of sign errors.
But here’s the key insight:
If we assume there are no true zero effects — just small effects in both directions — then false positives are really sign errors. And under symmetry, half of the false discoveries are expected to have the wrong sign.
This leads to a powerful relationship: Maximum SER=FDR2\text{Maximum SER} = \frac{\text{FDR}}{2}
So if z-curve estimates that the false discovery risk is 14%, we can infer that the maximum sign error risk is 7% — assuming a symmetric distribution of near-zero effects.
This makes z-curve informative about direction, even though it doesn’t use signs explicitly.
🧮 When Does Sign Matter?
There are research contexts where the sign is central. This includes:
- Meta-analysis: combining effect sizes across similar studies to estimate an average effect.
- Hypothesis testing with strong directional predictions (e.g., preregistered one-sided tests).
In such cases, the sign must be preserved — and tools that ignore it may be insufficient.
But z-curve is not designed to pool effect sizes across similar studies. Its goal is different: to evaluate the credibility of a literature by modeling the distribution of significant and non-significant results.
For that purpose, absolute z-values are sufficient — and focusing on the magnitude of evidence, not the direction, is a strength, not a flaw.
✅ Practical Takeaway
- Z-curve estimates false discovery risk, even in the presence of publication bias.
- If you care about the direction of effects, you can safely interpret: Maximum Sign Error Risk≈False Discovery Risk2\text{Maximum Sign Error Risk} \approx \frac{\text{False Discovery Risk}}{2}
- This holds without needing to model the sign of test statistics — as long as near-zero effects are symmetrically distributed.
- In most research contexts, this makes z-curve a powerful tool for assessing the reliability of statistical evidence, even if it’s not a tool for meta-analytic synthesis.
🧠 Final Thought
In an era where researchers seek transparency and reproducibility, tools like z-curve help us answer crucial questions: How likely is it that our significant results are true — and in the right direction? Even without using sign, z-curve gives us surprisingly good answers.