under development

Can’t find what you are looking for?
1. Ask an AI to search replicationindex.com to find answers that are not here)
2. Send me an email and I will answer your question and add it to the FAQ list.

Link to the tutorial:
🔗 Z-Curve 3.0 Tutorial (Introduction and links to all chapters) — https://replicationindex.com/2025/07/08/z-curve-tutorial-introduction/ (replicationindex.com)

1. Has z-curve been tested with simulation studies?

Yes. The original z-curve method was evaluated in large-scale simulation studies by Jerry Brunner. The extended z-curve 2.0 was tested by Frantisek Bartoš. The results of these simulation studies were part of the manuscripts submitted for publication. They have also been reproduced by independent researchers. The results and the code to reproduce or run new analyses are shared on the Open Science Framework.

After concerns were raised about performance with small sets of studies, new simulation studies showed good performance with just 50 significant results (see Concerns About Z-Curve: Evidence From New Simulations With Few Studies).

It is important to distinguish between estimation of the Expected Replication Rate (ERR) and the Expected Discovery Rate (EDR). The ERR estimates the average true power of the significant results. It is estimated with high accuracy even with fewer than 50 studies. The EDR predicts the distribution of non-significant results that were not observed. This prediction is more difficult and requires more information from the significant results. With small samples, confidence intervals are naturally wide, but their width is data-dependent and informative in itself.

In short, ERR estimates can be obtained even with small sets of significant results. They are also preferable to p-curve’s estimates of average power, which degrade when studies vary in power. EDR estimates are trustworthy with more than 50 significant results.

Does z-curve fail when power is homogeneous and falls between grid points?

The discrete mixture model uses fixed components at noncentrality parameters 0 through 6. When all studies share a single noncentrality that falls between two grid points — for example, NCP = 1.5 — the model must approximate a point mass using weights on flanking components. Van Zwet (2026) showed that this maximizes approximation error and can bias the EDR estimate.

Z-curve 3.0 addresses this directly. When low heterogeneity is detected, the algorithm fits a single-component model with a free noncentrality parameter and adds the estimated NCP as an additional component. The EM algorithm then places weight on this data-driven component rather than splitting weight awkwardly across the fixed grid. In simulations with NCP = 1.5, the added component recovers the correct location and the bias disappears.

When power varies across studies — as in any real meta-analysis of conceptual replications — the discrete model performs well without this correction. The mixture weights approximate the underlying distribution of power much like a histogram approximates a smooth density. Simulation studies confirm that the discrete model outperforms a parametric normal model under high heterogeneity, because it makes no assumption about the shape of the power distribution. In this sense, the discrete model is better understood as a distribution-free (nonparametric) estimator, and its flexibility is an advantage rather than a limitation.

3. Does z-curve offer options for small sample (small-N) literatures like animal research?

Short answer:
Yes — z-curve 3.0 adds new transformation methods and a t-curve option that make the method more appropriate for analyses involving small samples (e.g., N < 30). These options are designed to mitigate biases that arise when you convert small-sample test statistics to z-scores using standard normal approximations. Z-curve.3.0 also allows researchers to use t-distributions (t-curve) with a fixed df that is more similar to the distributions of test statistics from small samples than the standard normal distribution.

Details:

• The z-curve 3.0 tutorial (Chapter 8) explains that instead of only converting p-values to z-scores, you can now:
  • Try alternative transformations of t-values that better reflect their sampling distribution, and
  • Use a direct t-curve model that fits t-distributions with specified degrees of freedom instead of forcing a normal approximation. This “t-curve” option is recommended when studies have similar and genuinely small degrees of freedom (like many animal experiments). (replicationindex.com)
• These improvements help reduce bias introduced by naïve normal transformations, though they don’t completely eliminate all small-sample challenges, and performance can still be unstable when degrees of freedom vary widely or are extremely small. (replicationindex.com)

---