P-curve is a statistical tool that was designed to evaluate the statistical credibility of significant results. When only significant results are published, it is unclear how much selection for significance contributed to the results. In the worst case scenario, all published results are false positives. P-curve uses a variety of approaches to test this worst case scenario. If the null-hypothesis can be rejected, the data are said to have evidential value; that is, at least some of the studies rejected a false null-hypothesis.
P-curve was published without extensive validation research. Critical examination of the method has focussed on the estimate of average power (Brunner, 2018; Brunner & Schimmack, 2020). Average power can quantify the strength of evidence against the null-hypothesis rather than simply rejecting the null-hypothesis of no evidence. For example, a set of studies could have 18% average power, suggesting that some significant results were true positives, but also showing that this literature has many studies with low power.
The problem with p-curve is that, contrary to claims by its developer, it produces inflated estimates of power when studies vary in power. For example, it predicts that 91% of replications should have been successful in the reproducibility project (Open Science Collaboration, 2015), when only 36% of the actual replications were successful. This bias is expected given the large heterogeneity in power across these studies (Schimmack & Soto, 2026). A solution to this problem is to use z-curve (Bartos & Schimmack, 2022; Brunner & Schimmack, 2020). Z-curve is explicitly designed for heterogenous data and performs well with low and high heterogeneity (Schimmack & Soto, 2026).
Morey and Davis-Stober (2025) raised further concerns about the statistical properties of p-curve. Given the similar aims of p-curve and z-curve, it is reasonable to wonder whether z-curve suffers from some of the same problems as p-curve, despite its ability to handle heterogeneity well. I asked Claude AI to examine this question and it concluded that z-curve is built on a fundamentally different approach than p-curve that avoids many of p-curve’s pitfalls. Here is a summary of the evaluation.
Full table
| Criticism | Heterogeneity-dependent? | Affects power estimation? | Generalizes to z-curve? |
|---|---|---|---|
| EV* inadmissibility (probit/concave acceptance region) | No | Yes (same transform used) | No |
| Nonmonotonicity (compound half p-curve) | No | No | No |
| Boundary sensitivity (probit maps boundary to ∞) | No | Yes | No (EM is smooth) |
| LEV/LEV* large-value blindness | No | Indirectly | No |
| Power estimation inconsistency | Yes (core mechanism) | Yes (the main finding) | No |
| Conceptual: not tests of skew | No | Partly | No (z-curve doesn’t claim this) |
| Conceptual: noncentrality ≠ effect size | No | Partly (p-curve conflates them in its framing) | Not applicable — z-curve targets power, not effect size |
P-curve’s problems go beyond heterogeneity
The most fundamental problem is inadmissibility of the core test of evidential value (EV). The core test — the version currently in the p-curve app — uses a probit transformation that produces a concave acceptance region in the test statistic space. By results from Birnbaum (1954) and Marden (1982), this makes the test inadmissible: its power is dominated by other tests for every possible alternative, including the homogeneous case. The 2015 switch from the log to the probit transformation was motivated by wanting robustness to extreme values, but admissibility requires exactly the property that was engineered out — sensitivity to large individual test statistics.
The compound half p-curve rule introduces nonmonotonicity: increasing the evidence in a single study can flip the procedure from rejection to acceptance and back, multiple times, along a monotonically increasing path. This is a purely structural consequence of the hard boundary at αpc/2 combined with the probit transform, and has nothing to do with whether effect sizes are heterogeneous.
Test LEV, which is supposed to detect “lack of evidential value,” has an additional pathology: arbitrarily large test statistics contribute zero weight to the sum, because they map to log(1) = 0. A single study with a p value just below 0.05 can dominate the test and force rejection regardless of how large every other test statistic is. Six studies with Z = ∞ plus one study at Z = 1.97 yields the same test statistic as six studies at Z = 1.97.
None of these problems affect z-curve. Z-curve uses EM estimation on a mixture of truncated normal distributions, fitting the full shape of the observed z-score distribution above the significance threshold. Large z-scores contribute information proportional to their posterior weight on high-NCP components. The EM likelihood surface is smooth and does not blow up near the truncation boundary. There is no compound decision rule. And because z-curve’s target quantities are replicability (ERR) and discovery rate (EDR) — both functions of noncentrality parameters — there is no conflation of power with effect size.
The Morey and Davis-Stober paper does not mention z-curve. It does not need to. Their formal results simply confirm, from a different direction and with different tools, what simulation studies have shown for years: p-curve’s statistical machinery is not up to the job it advertises. Z-curve was designed from the start to avoid exactly these pitfalls.
In short, z-curve is not just another p-curve. While the aims are similar, the statistical approach and the ability to handle realistic amounts of heterogeneity are very different. Morey and Davis-Stober’s critique is limited to p-curve and does not generalize to z-curve.