A Better P-Curve: The Aggregated Cauchy Association Test with PP-Values

Uncategorized

Main Points

  1. Purpose of p-curve.
    The p-curve method was introduced as a test of the null hypothesis that all significant results are false positives (that is, that (H_0) is true for all tests).
  2. Use in psychology.
    P-curve became popular in psychology as a way to demonstrate that a body of research has evidential value—in other words, that not all significant results are false positives.
  3. Criticisms.
    The method has been criticized on several grounds, including unsound statistical assumptions and inadmissible decision rules (Morey & Davis-Stober, 2025).
  4. Earlier related work.
    It has gone largely unrecognized that similar approaches for combining truncated p-values were developed much earlier in genomics (e.g., the Truncated Product Method, TPM; Zaykin et al., 2002).
  5. A modern alternative.
    A newer and more widely used approach is the Aggregated Cauchy Association Test (ACAT). Although ACAT does not assume truncation, truncated p-values can be analyzed by selecting p < α and dividing them by α to obtain pp-values that follow a uniform(0, 1) null distribution.
  6. Advantages over p-curve.
    This pp + ACAT approach addresses many of the statistical problems identified by Morey and Davis-Stober (2025), including inadmissibility, discontinuity, and sensitivity to values near α, while retaining the same logical test of the global null.
  7. Remaining limitations.
    Like p-curve, ACAT tests the hypothesis that all significant results are false positives, but it does not quantify the strength of evidence (e.g., average power) or capture heterogeneity among studies. For this reason, z-curve remains the preferred method for evaluating evidential value in a set of significant results.

References

Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4, Article MP.2018.874. https://doi.org/10.15626/MP.2018.874 open.lnu.se+2CRAN+2

Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6, Article MP.2021.2720. https://doi.org/10.15626/MP.2021.2720

Morey, R. D., & Davis-Stober, C. P. (2025). On the poor statistical properties of the p-curve. American Statistician, in press. (see also Title: Review of “On the Poor Statistical Properties of the P-Curve Meta-Analytic Procedure” Published: August 8, 2025 by Ulrich Schimmack on the Replication Index blog. replicationindex.com
URL: https://replicationindex.com/2025/08/08/review-of-on-the-poor-statistical-properties-of-the-p-curve-meta-analytic-procedure

Zaykin, D. V., Zhivotovsky, L. A., Westfall, P. H., & Weir, B. S. (2002). Truncated product method for combining p-values. Genetic Epidemiology, 22(2), 170–185. https://doi.org/10.1002/gepi.0042

Liu, Y., Chen, S., Li, Z., Morrison, A. C., Boerwinkle, E., & Lin, X. (2019). ACAT: A fast and powerful p-value combination method for rare-variant analysis in sequencing studies. American Journal of Human Genetics, 104(3), 410–421. https://doi.org/10.1016/j.ajhg.2019.01.002

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242

Leave a Reply