After a previous manuscript was rejected from Psychological Methods, we were allowed to submit a commentary (this does not mean acceptance). This blog post merely provides a link to the commentary that will eventually be available as a preprint with a DOI on PsyArXiv, after moderators approve it.
Schimmack, U. & Soto. M. (2025). Two Concepts of Power: A Commentary on Pek et al. (2024). https://doi.org/10.31234/osf.io/9v23f_v1
PsyArXiv Preprints | Two Concepts of Power: A Commentary on Pek et al. (2024)
The key information is summarized in a Table from the manuscript. The full manuscript is available for immediate download here.
Table 1
Summary on the Uses of Power Analyses
| Topic | Cohen (1962/1988/1990) | Pek et al. (2024) | Schimmack & Soto | ||||
| Definition of power | “Probability that a test will yield a statistically significant result.” (No explicit requirement to condition on a specific H1.) | Treats power as conditional on a specified non-zero H1 (textbook NP framing). | One concept vs. conditional definition is the core divergence. | ||||
| Primary use | Both planning (design) and evaluation (judging whether studies/literatures are underpowered). | Emphasizes planning/design; cautions against using power to evaluate published results. | Evaluation is legitimate when grounded in realistic assumptions about effect sizes | ||||
| Role of H1 | Not tied to a single point H1; true power depends on the unknown true effect size (which could be near zero). | Requires specifying an assumed effect for power; focuses on non-zero H1 for Type-II control. | Allows you to discuss cases where the true effect is 0 (P(sig.) = alpha) | ||||
| H0 is true Zero effects | Coherent under Cohen’s wording: if the true effect ≈ 0, chance of significance ≈ α. | Typically excluded from “power” by definition (since it’s conditional on H1 ≠ 0). | This is where “probability of significance” vs. “power given H1” matters. | ||||
| Estimating typical/average power | Endorsed rough, conventional “t-shirt” effect sizes (e.g., d=.5) → typical power ≈ ~50%; warned the field was underpowered. | Critiques “what-if”/benchmark approaches as hypothetical; argues they’re not the true power of specific literatures. | All inductive inferences are conditional on “what if” assumptions | ||||
| Post-hoc/retrospective power | Implicitly acceptable when grounded in external/precise estimates; focus on CI and practical planning. | Warns against using observed/retrospective power to judge completed studies. | Plug-in power adds little for borderline results but is informative with strong evidence (z > 6). | ||||
| Use of success rates | Considers journal success rates vs. plausible power to diagnose problems (publication bias, low power). | Skeptical of linking success rates to power estimates under their definition. | Success rate (% significant) ≈ mean power; with selection, use selection-adjusted methods. | ||||
| Replication crisis diagnosis | Low typical power + publication bias implies publication bias, likely replication failures. | Focus on better a priori planning; critical of using power to evaluate literatures. | Both are needed—better planning and evidence-based evaluation. | ||||
| Recommended practice | Plan with justified assumed effects; report estimates/CI; evaluate literatures with convergent evidence on average power. | Plan with conditional power tools; avoid interpreting “power” as an evaluative statistic for published studies. | Use z-curve/selection models to estimate average probability of significance and compare with reported success rates. | ||||
I had another AI chat about the disagreement between Pek et al. and Schimmack and Soto. This time I used Copilot (not sure how different it is from ChatGPT, but my ChatGPT knows too much based on previous discussions. So, the Copilot discussion may be closer to discussions of a typical user.
### Summary of discussion
Short version: **Pre-study (prospective) power** is a hypothetical design quantity calculated assuming a population effect size. **Post-hoc (observed) power** computed from a single completed study is misleading if treated as a diagnostic of that study’s truth. In meta-analytic settings, estimating average power across many studies is useful; methods that model the full distribution of results (e.g., **z-curve**) handle heterogeneity and selection bias better than p-curve. Pek et al.’s ontological critique targets naive post-hoc power for individual studies, not careful meta-analytic power estimation.
—
### Key concepts (concise)
– **Power (traditional definition)**: The probability a study will produce a statistically significant result under a specified true effect size, sample size, and α. It is conditional on assumptions and is a *pre-study* design quantity.
– **Prospective power**: Calculated before data collection using an assumed effect size (from prior studies, theory, pilot data); a planning tool.
– **Observed/post-hoc power**: Calculated after a study using the observed effect size; numerically redundant with the p-value and liable to misinterpretation if presented as a property of the finished study.
– **Ontological error (Pek et al.)**: Treating a hypothetical, pre-study probability as if it were an inherent property of observed data after the study is complete. The critique warns against reifying conditional planning probabilities as factual diagnostics.
– **Meta-analytic power / average power**: An estimate of the expected detection probability across a set of studies. This is a field-level descriptor and can be estimated informatively when done with methods that account for heterogeneity and selection.
—
### Why pre-study “guesses” are acceptable but naive post-hoc power is problematic
– Pre-study power is explicit about assumptions: it answers “If the true effect were X, what is the chance of detecting it?” That transparency makes it a valid design tool.
– Post-hoc power that uses the observed effect treats a noisy, sample-based estimate as if it were the true effect, leading to circular or misleading statements (for example, justifying nulls by saying “power was low” based on the same data).
– That said, estimating power from sample data is a legitimate inferential move if you: **acknowledge uncertainty**, treat the result as an estimate (not a fact), and avoid using it as a conclusive diagnostic for a single study.
—
### Tools for estimating average power across studies (practical guidance)
– **Z-curve**
– What it does: Models the full distribution of test statistics (significant and nonsignificant), estimates average power, expected replication rate, and false discovery risk while accounting for selection.
– When to use: You have a collection of p-values or z-scores and expect heterogeneity in sample sizes and effects. Preferred for estimating overall replicability and average power.
– **P-curve**
– What it does: Uses only significant p-values to assess evidential value and estimate power of the significant subset.
– When to use: To evaluate whether the set of significant findings likely reflects true effects (but be cautious because it ignores nulls and assumes a single effect size).
– **Meta-analysis (effect-size pooling) + prospective power**
– What it does: Pool effect sizes, then compute power for each study or an average power using the pooled effect.
– When to use: For planning or for estimating power when you can reasonably assume pooled effect sizes are informative; sensitive to publication bias and heterogeneity.
– Supporting methods to consider: **subgroup analyses**, **meta-regression**, and **methods that adjust for publication bias** (trim-and-fill, selection models), or **Bayesian hierarchical models** that propagate uncertainty in effect sizes into power estimates.
—
### How heterogeneity affects interpretation
– Heterogeneity in true effects or sample sizes is expected and not a fatal problem.
– If heterogeneity is present:
– Average power can mask clusters (e.g., a few well-powered studies plus many weak ones).
– Use methods that model mixtures (z-curve, mixture meta-analysis, hierarchical Bayesian models) or analyze homogeneous subgroups separately.
– Report not only the mean power but also distributional summaries (e.g., proportion of studies above a chosen power threshold, expected replication rate).
—
### Practical checklist for reporting and sharing
– State whether power estimates are **prospective (assumption-based)** or **estimated from data**.
– Report the **assumptions** and uncertainty (confidence intervals or credible intervals) around effect sizes and power estimates.
– For meta-analytic summaries, prefer methods that **include nonsignificant results** and **model heterogeneity** (z-curve, hierarchical models).
– Avoid using single-study post-hoc power as a verdict on that study’s credibility. Use confidence intervals, effect-size precision, or replication attempts instead.
– When presenting both p-curve and z-curve results, explain that p-curve typically yields higher estimates because it conditions on significant findings; z-curve gives a broader, more conservative field-level picture.
—
### One-paragraph takeaway you can share
Power is fundamentally a hypothetical, pre-study probability conditional on an assumed effect size. Calculating it prospectively with transparent assumptions is a valid planning tool; computing “post-hoc” power from a single study’s observed effect is misleading if presented as a fact. In meta-analytic work, estimating average power is informative and appropriate when methods account for selection and heterogeneity—z-curve (modeling the full distribution of test statistics) is typically a better choice for estimating overall replicability than p-curve, which conditions on significant results and assumes a single effect size.