van Zwet, E., Gelman, A., Greenland, S., Imbens, G., Schwab, S., & Goodman, S. N. (2024). A New Look at P Values for Randomized Clinical Trials. NEJM evidence, 3(1), EVIDoa2300003. https://doi.org/10.1056/EVIDoa2300003

Introduction

Randomized controlled trials occupy a privileged position in evidence-based medicine because randomization protects against many sources of confounding that complicate observational research. Yet influential critics have questioned whether even randomized evidence can be trusted at face value. Ioannidis’s Why Most Published Research Findings Are False argued that low power, bias, and low prior probabilities can make statistically significant findings more likely to be false than true, and his argument explicitly included clinical trials (Ioannidis, 2005). Andrew Gelman has expressed related skepticism, arguing that evidence-based medicine encounters serious problems when the evidence is weak and that even “clean randomized clinical trials can fail to replicate.”

However, Ioannidis and Gelman differ fundamentally about what makes a significant finding false. Ioannidis’s framework assumes that some tested effects are exactly zero. A significant result is then a false discovery when it rejects one of these true nil hypotheses. Gelman rejects this zero-versus-nonzero framing as a useful description of scientific research. In his discussion with O’Rourke, he argued that researchers are usually not studying effects that are exactly zero and questioned attempts to infer scientific truth by separating published results into exact-null and non-null components (Gelman & O’Rourke, 2014).

This disagreement became explicit in the debate over Jager and Leek’s attempt to estimate the false discovery rate in medical research. Using 5,322 reported $p$-values from five major medical journals, they estimated that approximately 14% of significant findings were false discoveries (Jager & Leek, 2014). Ioannidis strongly criticized their analysis and argued that the estimate was too reassuring (Ioannidis, 2014). Gelman and O’Rourke (2014) objected for a different reason: they questioned whether a science-wide false discovery rate based on dividing effects into exact zeros and nonzeros was a scientifically meaningful quantity at all.

Gelman and Carlin (2014) proposed focusing instead on Type S and Type M errors. Type M errors concern exaggeration of effect magnitude. Type S errors are more fundamental: a statistically significant result has the wrong sign—for example, a trial concludes that a treatment is beneficial when its true effect is harmful (Gelman & Carlin, 2014). A related quantity, the false-sign rate, has subsequently been developed in the multiple-testing literature (Stephens, 2017).

Jager and Leek’s estimated null component need not be interpreted literally as a population of effects that are exactly zero. Very small nonzero effects generate nearly uniform $p$-value distributions and are difficult to distinguish from exact nulls in a two-component mixture. If their estimated 14% false-discovery component instead represented effects very close to zero, approximately half of these significant findings would be expected to have the wrong sign. This suggests a false-sign rate of roughly 7% from this component, with only a comparatively small additional contribution expected from the stronger-effect component.

Thus, Jager and Leek’s results do not suggest that most significant medical findings either reject a true nil or point in the wrong direction. Nevertheless, their estimate was challenged on several grounds and had little influence on subsequent debates about the credibility of medical research.

Schimmack and Bartoš (2023) approached the problem differently. Rather than estimating the actual false discovery rate, we estimated an upper bound, which we called the false discovery risk. Sorić (1989) showed that the maximum false discovery rate is determined by the discovery rate—the proportion of all tests that are significant—without requiring an estimate of how many true effects are exactly zero. Using a new sample of clinical trials reported in medical journals, we estimated a false discovery risk of 13%, with a 95% confidence interval from approximately 8% to 21%.

This distinction is important because false-discovery methods do not require investigators to identify which hypotheses are truly null. The broader logic has long been used in large-scale multiple testing, particularly in genomics, where controlling the false discovery rate became an alternative to controlling the probability of any false positive (Benjamini & Hochberg, 1995; Storey, 2003). Z-curve extends this logic to literatures affected by publication selection: it estimates the discovery rate that would be expected without selection and uses this rate to obtain an upper bound on the FDR (Bartoš & Schimmack, 2022). Importantly, this approach does not require the assumption that exact nil effects actually exist. The purpose of estimating the false discovery risk is to examine how credible rejections of the nil hypothesis are.

A year after our study, van Zwet, Gelman, and colleagues analyzed 23,551 randomized clinical trials from Cochrane reviews (van Zwet et al., 2024). Their main concern was effect-size exaggeration, but they also estimated the probability that statistically significant trials had the wrong sign. Their model implied a false-sign rate of only about 2%. This remarkably low rate sits uneasily beside broad claims that statistically significant results from low-powered clinical trials are generally untrustworthy. It suggests that the principal problem identified by their model is not that significant clinical trials usually reach the wrong directional conclusion, but that their estimates of effect magnitude are noisy and selected upward.

Thus, three analyses of medical research appear to produce somewhat different pictures: Jager and Leek estimated an actual FDR of approximately 14%, Schimmack and Bartoš estimated a maximum FDR of approximately 13%, and van Zwet and colleagues estimated a false-sign rate of only about 2%. These quantities are not identical, but they address closely related questions about the credibility of statistically significant clinical-trial results. One important reason for their differences is the assumed distribution of true effects.

The present analysis examines this issue directly. Using the same Cochrane data, I fit several substantially different mixture models and examine which conclusions are robust to the choice of mixture and which depend on interpreting the fitted components as real populations of true effects.

The Credibility of Z-Curve

In a series of posts on Andrew Gelman’s blog, van Zwet criticized z-curve, the statistical method that we used to estimate the false discovery risk. His concerns included bootstrap confidence intervals in some settings and sensitivity of the expected discovery rate to misspecification of z-curve’s default discrete mixture. In particular, he showed examples in which a true noncentrality fell between z-curve’s fixed component locations and the expected discovery rate was biased.

These are legitimate concerns about model uncertainty. I responded by further developing z-curve and releasing zcurve3. One important extension is that zcurve3 no longer requires the traditional discrete mixture. Users can fit mixtures of normal distributions and vary the locations and variances of the components, making it possible to examine directly whether substantive conclusions depend on the particular representation of the latent distribution.

This extension is particularly useful here because van Zwet, Gelman, and colleagues modeled the Cochrane data with a mixture of normal distributions centered at zero. Zcurve3 makes it possible to fit the same Cochrane data with their zero-centered normal mixture, with a more flexible normal mixture in which both means and standard deviations are estimated, and with the traditional discrete z-curve model.

This provides a direct robustness test. If the principal z-curve estimands change substantially across these models, concerns about the discrete-component approximation are justified. If they remain stable despite substantial differences in the estimated mixture components, the estimands are more robust than the latent mixture itself. The same comparison can determine whether estimates of the false-sign rate, which depend directly on the inferred distribution of true effects, show the same robustness.

Reproducibility Code:
https://github.com/UlrichSchimmack/zcurve3_development_functions/blob/main/SecondLook.Cochrane.R

Zero-centered normal mixture

The first model specified three normal components with their means fixed at zero and their standard deviations freely estimated, closely reproducing the model used by van Zwet and colleagues. Zcurve3 fits absolute $z$-values and therefore represents these components as normal distributions truncated at zero. For zero-centered normal distributions, this is simply the folded representation of the same symmetric model and has essentially no substantive impact on the fit.

There are two additional differences. Zcurve3 can explicitly model selection for statistical significance, and confidence intervals were obtained with cluster bootstrap resampling to account for the nesting of individual study results within Cochrane reviews.

The z-curve plot shows that the mixture closely traces the distribution of significant results. It also predicts the nonsignificant distribution well. The similarity between the observed and expected discovery rates indicates little evidence of selection for significance in the Cochrane data. This finding is informative in its own right. Nonsignificant trials are not missing from these meta-analyses; they are considerably more common than significant trials. This does not rule out other sources of effect-size inflation, but strong publication selection against nonsignificant trials does not appear to characterize this dataset.

The estimated false discovery risk is approximately 20%, with the upper end of the 95% confidence interval at about 25%. Thus, even when z-curve is fitted with a latent distribution closely resembling the one preferred by van Zwet and colleagues, no more than approximately one quarter of significant findings could be exact-null false discoveries. This result is broadly consistent with our earlier analysis of $p$-values reported in medical-journal abstracts, despite the very different dataset and mixture specification.

Normal mixture with free means

The second model relaxed the assumption that all component means are fixed at zero. Fixing the means at zero served other purposes in van Zwet and colleagues’ application, including producing a symmetric reference distribution and symmetric shrinkage toward zero. However, the restriction could matter if the latent effect distribution were centered or concentrated away from zero.

In the Cochrane data, relaxing the restriction made little difference. The means of the two dominant components were estimated to be close to zero, and the principal z-curve estimates were virtually unchanged. Thus, the zero-mean restriction happens to be fairly benign for these data.

Default discrete z-curve

The third model fitted the default z-curve specification with seven discrete components. Once again, the principal results changed only slightly.

This illustrates an important property of z-curve. Its principal estimands are functions of the fitted distribution of test statistics rather than interpretations of individual mixture components. Different mixture models can assign very different weights and parameters to their latent components while producing nearly identical fitted densities. If they reproduce the relevant distribution of $z$-values equally well, they can therefore produce very similar estimates of the expected discovery rate, expected replication rate, and false discovery risk.

This result directly addresses one aspect of van Zwet’s criticism. A fixed discrete approximation can be biased in some data-generating scenarios, and this possibility should not be ignored. But zcurve3 makes the concern empirically testable. Researchers can fit alternative mixture specifications as a sensitivity analysis. In the Cochrane data, replacing the traditional discrete mixture with normal mixtures does not materially alter the principal z-curve conclusions.

Robust estimands, unstable components

The picture changes when the individual mixture components themselves are interpreted as latent populations.

In the two continuous normal-mixture models, the probability of an effect being exactly zero is zero by construction. The actual exact-null FDR is therefore zero under these models. This should not be confused with the false discovery risk, which remains around 20%. The latter asks how high the FDR could be without assuming that the chosen continuous latent model is literally true.

The zero-centered and free-mean normal mixtures imply false-sign rates of approximately 3.2%, reasonably close to van Zwet et al.’s reported estimate of about 2%.

The default discrete z-curve gives a different latent interpretation. About 4.5% of significant results are assigned to the component with a noncentrality parameter of zero. If this component is interpreted literally, the estimated actual FDR is therefore about 4.5%. If, instead, the zero component is regarded as a discrete approximation to a continuous collection of very small positive and negative effects, exact-zero FDR disappears and approximately half of the significant results in this component become sign errors. Under this interpretation, the false-sign rate is approximately 2.7%.

Thus, these relatively flexible models all imply a false-sign rate below about 5%. However, this apparent agreement should not be mistaken for identification of the latent distribution.

To illustrate the problem, I fitted another discrete mixture with components at noncentralities $0,2,4,$ and $6$. Removing the component at $z=1$ prevents the model from representing weak positive effects explicitly. Many of the low-power studies must therefore be assigned to the zero component instead.

The principal z-curve estimands changed only slightly. The latent interpretation changed dramatically. After accounting for significant observations above $z=6$, the zero component implies an actual FDR of approximately 23%, close to the maximum FDR permitted by the discovery rate. If the zero component is instead interpreted as a symmetric collection of effects extremely close to zero, approximately half of these findings have the wrong sign, producing an estimated false-sign rate of about 11.5%.

Nothing about the observed Cochrane data changed. Only the latent mixture specification changed.

This example illustrates why fitted mixture components should not automatically be interpreted as literal data-generating populations. Their main statistical advantage is precisely their flexibility: different mixtures can approximate the same observed density. Quantities that depend mainly on the fitted density can therefore be robust even when the decomposition into latent components is not. In contrast, quantities that require a literal interpretation of the components—including estimates of the actual proportion of exact zeros, false-sign rates, and some shrinkage quantities—can be substantially more model dependent.

This is not unique to z-curve. Stephens’s (2017) empirical-Bayes approach to false-sign rates, for example, obtains greater stability by imposing a substantive shape constraint: the latent effect distribution is assumed to be unimodal with its mode at zero. Such assumptions may be reasonable, but estimates obtained from them are conditional on those assumptions rather than determined by the observed data alone.

Conclusion

Mixture models remain relatively uncommon in meta-analysis and meta-science, and their use can invite a basic misunderstanding. The most serious mistake is to interpret the fitted components as if they were empirically identified populations of studies or true effects. Component locations, variances, and weights can be highly sensitive to model specification. Different mixtures can fit essentially the same observed distribution while implying substantially different latent decompositions. Good model fit alone therefore cannot establish that one particular decomposition is the true data-generating process.

Z-curve was designed to avoid relying on this interpretation. Its principal estimands—the expected discovery rate and expected replication rate—are global properties of the fitted distribution. The false discovery risk is subsequently obtained from the estimated discovery rate using Sorić’s bound. Consequently, substantially different mixture specifications can produce similar answers as long as they reproduce the relevant features of the observed distribution.

This distinction also clarifies the role of the nil hypothesis. It is not necessary to assume that some fixed proportion of scientific effects are literally zero in order to use false-discovery risk as a credibility criterion. If exact-zero effects do not exist, the actual exact-null FDR is zero. The Sorić bound remains useful because it asks the more conservative question: given the discovery rate, how large could the false discovery rate be? This is consistent with the broader logic of false-discovery methods already established in large-scale multiple testing.

The same point helps clarify discussions of “low power.” Van Zwet and colleagues define the signal-to-noise ratio as the true effect divided by its standard error and translate it into conventional power to reject $H_0:\theta=0$. This is mathematically legitimate, but if exact-zero effects are assumed never to occur, low power against zero cannot itself imply that significant findings are false. It primarily signals that effects are small relative to their sampling error, which creates imprecise estimates, magnitude exaggeration after selection, and some risk of sign errors. How much of this makes a study scientifically untrustworthy requires an explicit criterion.

The present Cochrane analysis provides one such criterion. Across substantially different mixture specifications, the estimated discovery and replication rates and the maximum false discovery rate are remarkably stable. In contrast, actual FDR and false-sign estimates can change substantially when the latent components are interpreted literally.

The lesson is therefore not that one mixture model is correct and another is wrong. Conclusions should be trusted to the extent that they survive reasonable changes in mixture specification. In the Cochrane data, the principal z-curve estimands pass this test. Literal interpretations of the latent mixture components do not.

References

Bartoš F, Schimmack U. Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology. 2022;6:2021.2720. doi:10.15626/MP.2021.2720.

Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Series B. 1995;57:289–300. doi:10.1111/j.2517-6161.1995.tb02031.x.

Gelman A, Carlin JB. Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspect Psychol Sci. 2014;9:641–651. doi:10.1177/1745691614551642.

Gelman A, O’Rourke K. Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics. 2014;15:18–23. doi:10.1093/biostatistics/kxt034.

Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124. doi:10.1371/journal.pmed.0020124.

Ioannidis JPA. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics. 2014;15:28–36. doi:10.1093/biostatistics/kxt036.

Jager LR, Leek JT. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics. 2014;15:1–12. doi:10.1093/biostatistics/kxt007.

Schimmack U, Bartoš F. Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values. PLoS ONE. 2023;18:e0290084. doi:10.1371/journal.pone.0290084.

Sorić B. Statistical “discoveries” and effect-size estimation. J Am Stat Assoc. 1989;84:608–610. doi:10.2307/2289950.

Stephens M. False discovery rates: A new deal. Biostatistics. 2017;18:275–294. doi:10.1093/biostatistics/kxw041.

Storey JD. The positive false discovery rate: A Bayesian interpretation and the q-value. Ann Stat. 2003;31:2013–2035. doi:10.1214/aos/1074290335.

Faridani, S. (2026). Testing for underpowered literatures. Journal of Econometrics, 257, 106312. https://doi.org/10.1016/j.jeconom.2026.106312

Abstract

Faridani (2026) introduces a nonparametric deconvolution method for estimating how much the proportion of statistically significant results would increase if studies used larger samples. The method is innovative because it avoids specifying a parametric distribution of true effects and can accommodate publication bias and point masses at zero. However, the estimand is an unconditional increase in significance rates, not statistical power conditional on a true alternative. As a result, a small estimated gain can reflect high power, many true or approximately true null effects, very small nonzero effects, or some combination of these. For research planning and cost-benefit decisions, methods that estimate the latent mixture of null and non-null effects and the power distribution among true effects may therefore provide more directly relevant information.

Review

Faridani develops an innovative method for addressing an important problem in meta-research: determining how much larger samples would have changed the statistical conclusions of a heterogeneous collection of studies. Rather than attempting to estimate the power of individual studies, the paper estimates the counterfactual increase in the proportion of statistically significant results if all sample sizes were multiplied by a common factor. The proposed estimator uses nonparametric deconvolution, accommodates publication bias, and allows substantial heterogeneity in true effects, including point masses at zero.

A major strength of the paper is its recognition that estimating the full distribution of true effects is a much harder inverse problem than estimating a particular functional of that distribution. Faridani therefore focuses on $\Delta_c$, the expected increase in the proportion of significant results when sample size is increased. This makes the problem statistically more tractable. The simulations suggest that the estimator performs reasonably well across a range of data-generating processes, and the empirical applications to economics and Many Labs provide useful demonstrations.

The interpretation of $\Delta_c$, however, deserves some qualification. Faridani motivates it as a measure of whether a literature is “undersized” and as information relevant to decisions about whether research funds should be spent on larger samples. This is a coherent estimand, but it is not the same as conventional statistical power, which is defined conditional on a false null hypothesis.

For a literature containing both true null and true alternative hypotheses,$$P(\text{sig}) = P(H_0)\alpha + P(H_1)E[\text{Power}\mid H_1].$$

Increasing sample size leaves the first component unchanged. Therefore,$$\Delta_c = P(H_1) \left\{ E[\text{Power}_{c}\mid H_1] – E[\text{Power}_{1}\mid H_1] \right\}.$$

A small value of $\Delta_c$ can therefore have several very different interpretations. Studies testing real effects may already have high power, in which case larger samples are indeed unnecessary. But $\Delta_c$ can also be small because many hypotheses are true or approximately true nulls, or because many true effects are too small for a moderate increase in sample size to help much.

A simple example illustrates the problem. Consider one literature in which every study has a moderate true effect but poor precision. Such a literature may have uniformly low conditional power and would clearly benefit from larger samples. Now compare it with a literature in which half the hypotheses are exactly null and the other half have large effects that are already detected reliably. The second literature may show an equally small or smaller increase in the number of significant results, but for a completely different reason. In the first case, low power is the problem. In the second, many non-significant results are correct indications of little or no effect.

This distinction matters for the funding interpretation. The scientific objective is not simply to maximize the number of $p<.05$ results. Ideally, a test should have a high probability of rejecting $H_0$ when a meaningful alternative is true while retaining the nominal Type I error rate when $H_0$ is true. A small increase in the unconditional rejection rate therefore does not by itself show that studies testing real effects have adequate power.

The main uncertainty lies among the non-significant results. If significant results already have high expected replicability, increasing their sample sizes has little practical value. The important question is whether non-significant results are mostly true or approximately true nulls, or false negatives arising from real effects studied with insufficient precision. The answer depends on the latent mixture of $H_0$ and $H_1$ and on the effect-size distribution within the $H_1$ component.

Faridani’s method deliberately avoids recovering this structure, which is part of what makes the estimand tractable. But this also limits its usefulness for planning future sample sizes. A deconvolution estimate of the average increase in significant results does not separately estimate the prevalence of null and non-null effects or the conditional power of studies testing real effects.

Mixture approaches such as z-curve aim to recover more of this latent structure. In particular, they can estimate the expected replicability of significant results and features of the latent power distribution. This permits a more informative separation between the prevalence of true effects and the power of studies conditional on those effects being present. Such information is more directly useful for cost-benefit analyses of future sample sizes than an average increase in significance rates that combines true or approximately true nulls with true alternatives.

A second, more technical point concerns the paper’s characterization of existing mixture methods. Faridani contrasts his nonparametric approach with methods such as Brunner and Schimmack (2020), which he describes as requiring a “specific shape” for the latent distribution. That description is somewhat too strong. A finite mixture on a prespecified grid does impose an approximation, but its purpose is precisely to approximate a wide range of latent distributions by estimating the mixture weights. The more informative distinction is therefore between different forms of regularization of the same inverse problem: finite-mixture approximation versus regularized nonparametric deconvolution.

These qualifications do not detract from the methodological contribution. Faridani provides a clever way to estimate a difficult counterfactual quantity without recovering the full latent distribution. But the resulting estimand answers a narrower question than the one most directly relevant to statistical power and research planning. It tells us, “How many more significant results would larger samples produce?” A more informative planning question is, “Which studies are underpowered because they are testing real effects, and how much would larger samples improve their ability to detect those effects?” Answering that question requires distinguishing the $H_0$ and $H_1$ components rather than averaging across them.

Robust Bayesian Meta-Analysis, or RoBMA, was developed in E. J. Wagenmakers’ research group as a method for evaluating meta-analytic evidence while allowing for the possibility that published effects are exaggerated by publication bias. An important feature of RoBMA is that it does not merely estimate an effect size; it also assigns evidence to models in which the average effect is exactly zero.

This approach builds on a form of Bayesian hypothesis testing that Wagenmakers and colleagues have long promoted as more balanced than conventional null-hypothesis significance testing. In standard significance testing, researchers can reject the nil hypothesis or fail to reject it, but a nonsignificant result does not itself constitute evidence that the effect is zero. Bayes factors, by contrast, can quantify relative evidence for either the nil or an alternative hypothesis.

This form of Bayesian hypothesis testing dates back to Harold Jeffreys’ Theory of Probability (1939), but it remained far outside the mainstream of psychological research for decades. It became highly visible in psychology after Bem (2011) reported a series of statistically significant experiments purporting to demonstrate extrasensory perception. Wagenmakers et al. (2011) reanalyzed Bem’s (2011) results with default Bayes factors and argued that the evidence for psi was weak or nonexistent despite the significant $p$-values. However, the Bayes Factors also failed to produce clear evidence that ESP does not exist.

Robust Bayesian Meta-Analysis, RoBMA for short, was developed to conduct Bayesian meta-analyses that take publication bias into account and can provide evidence for the nil hypothesis that the average true effect is zero. A Bayes-Factor that does not favor the broad alternative hypothesis of an effect is sometimes interpreted as “No evidence for an effect” (Maier et al., 2022). However, this conclusion is not warranted if the model also estimates heterogeneity around this average. Heterogeneity implies that some estimates reflect true effects. Moreover, the average effect size of positive results is implied to be greater than zero (Schimmack, 2026).

The main problem with RoBMA is that it s built on top of existing bias-correction models with strong assumptions, specifically step-function selection models (Vevea & Hedges, 1995; Hedges & Vevea, 1996). The most critical assumption of this model is that population effect sizes are normally distributed. When this assumption is violated because the dataset contains few negative results, the model assumes that selection bias removed negative results and produces a lower estimate closer to zero. In extreme cases, it can estimate that the mean is zero and find evidence for a false nil hypothesis about the mean.

When RoBMA was used with data from a meta-analysis of nudging interventions, the model adjusted the average effect of approximately $d=.43$, to d = .07. However, another bias correction methods that do not assume a normal distribution, produced an estimate of d= .44 (Schimmack, 2026b).

RoBMA Analysis of Government Nudging Studies

DellaVigna and Linos (2022) used results from large government nudge trials and compared them with results from broadly comparable academic studies. The difference was striking. In the academic literature, nudges changed behavior by about 8.7 percentage points on average. Thus, if 20% of people performed a behavior in the control condition, a typical effect of this magnitude would increase the rate to about 29%. In the government trials, however, the average effect was only about 1.4 percentage points. In the large government studies, this finding was still statistically significant, but the same effect size might be too weak in other studies to reject the nil hypothesis. Thus, the results of this study seem to be consistent with RoBMA’s bias-corrected estimate. However, an alternative explanation is that there government studies have smaller effect sizes because they are different from the nudging manipulations in academic studies. To explore these competing hypothesis, I analyzed the data with RoBMA and zcurve3, a model that does not assume a single normal distribution of population effect sizes.

The authors of RoBMA already analyzed these data, but they did not use RoBMA (Maier et al., 2024), although a reviewer suggested it. They argued that it would be inappropriate to use RoBMA with these data because the data do not fit a single normal distribution. This is an important caveat because it makes the distribution assumption salient and because the authors applied RoBMA to the academic meta-analysis without testing the distribution assumption.

Here I applied RoBMA to data that are known to violate the distribution assumption. The result was similar to the one for the academic meta-analysis: evidence for an overall mean effect was inconclusive and slightly favored the null model (inclusion BF = 0.46). The model-averaged estimate of the mean effect was $\mu = 0.016$, 95% credible interval [-0.413, 0.502]. In other words, “no evidence for an average effect in government nudging studies.” Again, this finding does not mean that nudging interventions do not work. Given large heterogeneity around the average, The analysis provided overwhelming evidence for heterogeneity in true effects (inclusion BF > 14,999), with an estimated heterogeneity parameter of $\tau = 3.34$, 95% credible interval [3.07, 3.63]. However, the results imply that nudging studies also often backfire, but that these negative results are missing, publication bias (inclusion BF > 14,999).

Figure 1 from Maier et al. (2024) shows the problem for the weight-function selection model underlying RoBMA. The distribution has a long positive tail that does not fit a single normal distribution (blue). A model that only fits the right side and attributes the low frequency of negative results to selection bias (red line). However, even this model does not fit the data well, but RoBMA has no other way to represent the data.

However, we can help RoBMA by removing extreme values from the data on both sides. However, because there are many more positive extreme values than negative extreme values, the exclusion of extreme values lowers the average of the observed data from 1.38 percentage points (pp) to 0.74 pp. In contrast, the median did not change much (0.50 vs. 0.42 pp).

Applying RoBMA to these data changed the estimate of the average effect size dramatically. The Bayes Factor now clearly rejected the nil hypothesis, BF > 14,999 (the maximum value in RoBMA). The estimated average was 0.58, only slightly lower than the simple average of 0.74 pp. This estimate also does not take the extreme positive results into account. The main point, however, is that RoBMA underestimated the average effect size when a long tail of positive results was included because it imagined a long tail of matching negative results based on the symmetry of the normal distribution. As Maier et al. (2024) already stated, this means RoBMA should not be used when data are likely to have a different distribution.

Maier et al. (2022) computed their RoBMA result ignoring the clustering of standard errors — as stated in their own figure caption. This is not a minor detail. The 448 effect sizes come from 212 studies, and treating dependent estimates as independent inflates the apparent precision that the bias correction then acts on. Without clustering, RoBMA returns a mean of essentially zero and evidence against an effect under every bias specification I tried — the full model-averaged ensemble, and a selection-model-only version with PET-PEESE removed (μ = 0.02, BF favoring the null). Once the clustering is modeled — using the multilevel extension of RoBMA (Bartoš, Maier & Wagenmakers, 2026) — the evidence reverses decisively (inclusion BF > 14,999) to a clearly positive mean, between d = 0.22 and d = 0.34 depending on the bias model. The published conclusion of “no evidence for nudging” is an artifact of ignoring the dependence structure of the data.

How Robust is RoBMA?

In statistics, robustness traditionally refers to methods that remain reliable when assumptions are violated or when data contain outliers or other atypical observations (e.g., Box, 1953; Huber, 1964; Tukey, 1960). Calling RoBMA “robust” may therefore give users the impression that it has these properties. However, robust in RoBMA refers primarily to averaging across a predefined set of meta-analytic models, not to robustness against misspecification of assumptions those models share. Bayesian model averaging can hedge against uncertainty about which model in the set is correct; it cannot protect against an assumption that every model in the set makes.

That distinction is important because RoBMA, like every statistical model, can produce misleading estimates when its assumptions are poorly suited to the data. In the present analyses, results depend strongly on assumptions about the distribution of population effect sizes and the treatment of extreme or highly heterogeneous effects. Other plausible models can produce substantially different conclusions.

The practical lesson is therefore that its results should not automatically be treated as the most robust or most credible estimate. When different models give different answers, the scientifically important task is to examine why they disagree and which assumptions are plausible for the data at hand.

Hu, B., Z. Xia, Q. Guo, C. Lu, S. Constantino, and X. Ju. 2025. “ Assessing Nudge Impact: A Comprehensive Second-Order Meta-Analysis.” Journal of Behavioral Decision Making 38, no. 5: e70053. https://doi.org/10.1002/bdm.70053.

Post-Publication Review

Hu et al. present a second-order meta-analysis of 14 meta-analyses covering 1,638 studies and nearly 30 million participants. They report an unadjusted mean effect of $d=.27$, while PET-PEESE reduces the estimate to approximately zero.

The main problem is that adding more and more studies does not solve the basic problem of extreme heterogeneity. The authors report $I^2=99.89\%$, indicating that the included meta-analyses differ enormously in their estimated effects. Under these conditions, the question “What is the effect of nudging?” is not well defined. Defaults, reminders, food placement, information labels, and many other interventions are not repeated estimates of one common treatment effect.

The publication-bias results do not resolve this problem. In fact, they disagree. The conventional random-effects estimate is $d=.27$. RoBMA gives $d=.29$, 95% CI [.15, .43], with $\tau=.26$. Trim-and-fill still gives $d=.27$, and using adjusted estimates from component meta-analyses gives $d=.28$. Only PET-PEESE produces the near-zero estimate of $d=.003$–.004.

It is therefore misleading to foreground the PET-PEESE result as if different methods converge on zero. RoBMA already averages over PET-PEESE, selection, and no-bias models, yet its model-averaged estimate remains positive. Moreover, PET-PEESE relies on the relation between effect size and standard error, which can be confounded by genuine heterogeneity. In a literature where small and large studies differ in intervention type, setting, population, and outcome, that relation need not reflect publication bias alone. A separate weight-function estimate would have been informative, but none is reported.

More fundamentally, even a perfectly estimated grand mean would tell us little. The authors themselves show that effect sizes differ substantially across categories and domains; for example, decision-structure nudges have a descriptive estimate of $d=.40$, environmental nudges $d=.45$, and food nudges $d=.33$. Their moderator tests are underpowered and do not explain the heterogeneity, but failure to explain heterogeneity does not make the grand mean meaningful.

The article therefore does not tell us much that is new about nudging. We already knew that published effects are heterogeneous and that publication bias is a concern. Pooling ever larger numbers of heterogeneous studies does not answer the scientifically important questions: Which nudges work? How large are their effects? Under what conditions do they work?

The conclusion that the “true effect of nudging” is zero has the appearance of precision, but it is closer to answering an ill-defined question. The problem is not whether the grand mean is $d=.27$ or $d=.004$. The problem is that there is no single effect of nudging to estimate.

Mertens, S., Herberz, M., Hahnel, U. J. J., & Brosch, T. (2022). The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proceedings of the National Academy of Sciences, 119(1), e2107346118. https://doi.org/10.1073/pnas.2107346118

Introduction

Hundreds of studies examined the effectiveness of nudging manipulations, and most of them reported a positive effect, often also a statistically significant one; that is, a result unlikely to be a statistical fluke. A 2022 meta-analysis included 447 effect sizes from 212 publications. The pooled meta-analytic estimate was—not 42, but 43; that is d = .43 or 43% of a standard deviation.

This answer is close to the famous answer to the question about the meaning of life, the universe, and everything else in The Hitchhiker’s Guide to the Galaxy, which was 42. The novel poked fun at the attempt to answer a complex question with a single number, but meta-analyses often do exactly this. The hundreds of studies in the nudging literature used different manipulations, different outcomes, and different populations. What is the probability that they all have the same effect size? Nil. So, the headline figure in a meta-analysis is at best the center of a heterogeneous distribution of effect sizes; it does not tell us the effect of any specific intervention.

This did not stop two research teams from reanalyzing the data with statistical methods designed to correct for publication bias and to produce another single number. This number was dramatically different. Szászi et al. (2022) used a step-function selection model and obtained an adjusted estimate of the average effect size of $d=-.01$, with a very small standard error, $SE=.02$. In other words, after bias correction, the average effect of nudging was now approximately zero.

A second commentary reported the results of another analysis using a new statistical method, Robust Bayesian Meta-Analysis (RoBMA), which averages across several models, including one similar to the model used by Szászi et al. (2022). The title of the article—which is the only thing many readers ever see—made the conclusion clear: “No evidence for nudging after adjusting for publication bias” (Maier et al., 2022). Their estimated average effect size, $d=.04$, was not exactly zero, but close enough to invite the interpretation that nudging effects are negligible or nonexistent.

There are several problems with both reanalyses that presented the average estimate as the key finding. The first one is that the average alone is meaningless, when the same model also estimates considerable heterogeneity around the average. Neither article reported the estimate of heterogeneity that the models produce, but a reanalysis of the data with the models shows considerable heterogeneity (weight-function model, sd = .5; RoBMA sd = .3). This implies that many positive effect sizes estimates were obtained with positive population effect sizes that can be replicated. Thus, the results do not imply that nudging never has an effect. At least sometimes, nudging had effects. This simple fact is easily conveyed by a figure that plots the predicted distributions of population effect sizes along with the distribution of observed effect sizes (Figure 1).

The histogram shows that 91% of the observed effect-size estimates are positive. In contrast, the fitted distributions of population effect sizes are centered close to zero and therefore imply nearly equal numbers of positive and negative true effects. How can models that imply so many negative effects fit data in which negative estimates are rare?

The answer lies in two assumptions built into the models. First, they assume that population effect sizes follow a normal distribution. The positive observed effects are concentrated near zero with a long right tail and therefore resemble the positive half of a normal distribution centered close to zero. Once the model places the center near zero, however, symmetry requires a corresponding negative half. Because this negative half is largely missing from the observed data, the discrepancy is attributed to selection: negative results are assumed to be observed, but remained unpublished.

But this is not the only explanation of the observed pattern. An alternative is that there really are relatively few negative population effects—that is, nudges usually have effects in the theoretically predicted direction—but that the distribution of positive effects is strongly non-normal. Most nudges may have effects close to zero, while a smaller number have moderate or large positive effects. Such a right-skewed distribution would also produce exactly what we see in the histogram: many small positive effects, relatively few negative effects, and a long positive tail.

The crucial point is that the models cannot distinguish these two explanations from the observed data alone. A symmetric distribution centered near zero combined with strong selection against negative results can resemble an asymmetric population distribution containing mostly positive effects and little or no selection against negative results. The near-zero mean therefore does not follow directly from the data. It follows from the assumption that the underlying distribution of population effects is approximately normal and symmetric.

Separating Positive and Negative Results

Fortunately, we do not have to decide which explanation is correct to learn something useful from these models. If all variation in observed effect sizes were merely sampling error around a single population effect, positive and negative estimates would have to be combined to estimate that common effect. But both models estimate substantial heterogeneity. In their own models, population effect sizes genuinely differ from one study to another.

We can therefore ask a different question. Rather than averaging positive and negative population effects into a single number, we can examine the two parts of the fitted distribution separately. In particular, we can ask: Among population effects that are in the theoretically predicted direction, what is their average magnitude? If desired, we could ask the corresponding question about effects in the opposite direction.

This is a different estimand from the grand mean. It is also important to distinguish it from simply averaging the 91% of observed estimates that are positive. Observed estimates contain sampling error, and some positive estimates may arise from population effects close to zero or even negative. Instead, we can use the models’ estimated normal distribution of population effect sizes to calculate the mean of the positive part of the normal distribution.

RoBMA’s estimate is d = .26, a small effect, but the step-function model produces an estimate of d = .44, very similar to the estimate of d = .43 in the original article that was based on all estimates. The mean of only positive results is d = .51, but it is reduced to d = .43 by the inclusion of the negative results.

In conclusion, the two critical reanalyses that reduced the estimated grand mean from about $d=.43$ to approximately zero did not show that the positive effects themselves were reduced to zero after correcting for publication bias. In the step-function model, the estimated mean of positive population effects remains $d=.44$. Even RoBMA implies an average positive population effect of $d=.26$. The dramatic change in the grand mean arises largely because the models infer a substantial population of unobserved negative effects that counterbalances the positive effects.

Those missing negative effects may exist. Researchers may have obtained effects in the theoretically wrong direction and failed to publish them. But this does not imply that the positive effects are zero, nor does it show that their magnitudes were dramatically inflated. In particular, the step-function model provides little evidence that selection for statistical significance—the usual mechanism invoked to explain inflated published effect sizes—accounts for the observed pattern. Its strongest selection effect is for the direction of the effect: negative results are much less likely to appear.

This makes the presentation of the reanalyses misleading. The headline result was the grand mean close to zero, while the substantial heterogeneity around that mean was not reported in a way that made clear that the model implied a broad distribution of positive and negative effects. Nor was the weak evidence for selection at the conventional significance threshold emphasized. A very different headline could therefore have been: “Little evidence that selection for statistical significance inflated positive nudging effects.”

A Zcurve3 Analysis

Figure 1 also shows that neither model fits the distribution of positive results particularly well. RoBMA captures the concentration of weak effects near zero, but fails to reproduce the long tail of stronger positive effects. The weight-function selection model captures the positive tail better, but underestimates the large concentration of weak effects. Thus, estimates of the mean of the positive population effects remain sensitive to misspecification of the assumed normal distribution.

Zcurve3 provides an alternative approach because it does not assume that population effect sizes follow a normal distribution. Zcurve3 builds on zcurve 2.0 (Bartoš & Schimmack, 2022), but adds several features that are useful for the present analysis. First, it can model directional results while preserving the sign of an effect. By fitting the model only to positive effects, it can therefore examine the credibility and magnitude of effects in the theoretically predicted direction without making assumptions about a corresponding distribution of negative effects.

The plot confirms what the previous models suggested: there is no strong selection for statistical significance among the positive results. The observed discovery rate (ODR) is 66%, whereas zcurve3 estimates an expected discovery rate (EDR) of 46%. Thus, the point estimate suggests some selection for significance, but the confidence interval around the EDR is wide, ranging from 17% to 73%, and includes the observed rate of 66%. Consequently, the data do not provide clear evidence that non-significant results are missing. At the same time, the wide confidence interval means that substantial selection cannot be ruled out.

Zcurve3 also estimates a false discovery rate of only 6%, although the confidence interval is again wide, ranging from 2% to 25%. Thus, the results are inconsistent with an interpretation in which the large number of significant positive findings consists mostly of false positives. This is important because a grand mean close to zero can easily be misinterpreted as evidence that the significant nudging results have disappeared after correction for publication bias. That is not what these results show. They suggest that most significant results were obtained with a true effect.

Finally, zcurve3 estimates an expected replication rate (ERR) of 72%, 95% CI [60%, 79%]. This means that, under exact replication conditions, approximately seven out of ten significant positive results are expected to produce another significant result. Actual replication rates may be lower because replication studies are rarely exact and because effect sizes can vary across populations and contexts.

Replicability also increases sharply with the strength of the original evidence. The local-power estimates shown below the x-axis illustrate that studies with large z-values have a high probability of producing another significant result. Thus, even if the grand mean across all nudging interventions is close to zero, the literature clearly contains individual findings with substantial statistical evidence and high predicted replicability.

Zcurve3 also provides selection-adjusted effect-size estimates. For all positive results, the estimated mean effect is d = .37, 95% CI [.14, .60]. The confidence interval is wide because there is considerable uncertainty about how many nonsignificant positive results are missing and how small their effects are likely to be. Nevertheless, even the lower bound of the confidence interval is clearly above zero.

A statistically more precise estimate can be obtained for the significant positive results that are actually used to fit the z-curve model. Their estimated mean effect is $d=.60$, 95% CI [.47, .72]. This estimate answers a narrower question: what is the average effect size of the positive significant findings on which most published claims about successful nudges are based? For evaluating the credibility and magnitude of these claims, this conditional estimate is more informative than a grand mean that averages significant, nonsignificant, positive, and negative effects from fundamentally different interventions.

However, $d=.60$ is still only an average. Considerable heterogeneity remains among these effects, $SD=.25$, 95% CI [.13, .39]. Thus, I am not claiming that “the effect of nudging is .60.” There is no single effect of nudging. Zcurve3 provides a more meaningful average for a more narrowly defined question, but the remaining heterogeneity still needs to be examined. The next step is therefore not to replace 43 or 0 with 60 as the new answer to everything, but to identify which specific nudges produce credible effects and how large those effects are.

Zcurve3 provides a forest plot that makes it easy to identify particularly promising findings. The plot shows selection-adjusted effect-size estimates in blue, together with confidence intervals that incorporate both sampling error and uncertainty introduced by selection bias. Studies are ranked by the lower bound of their confidence interval rather than simply by their z-value or point estimate. Thus, studies rise to the top only when the data support an effect that is both reasonably large and estimated with sufficient precision. This avoids giving priority merely to very large studies that can produce impressive z-values for substantively trivial effects.

At the top of the forest plot is a study by Diliberti et al. (2004). Its z-value is outside the range used by Zcurve3 for effect-size adjustment, so the plot shows the reported effect size without correction. The estimate is $d=3.08$, an enormous standardized effect. Even the lower bound of the confidence interval is $d=2.65$. A result of this magnitude stands on its own and does not require a meta-analytic estimate. It deserves careful examination. However, an extreme meta-analytic effect-size estimate should never be trusted at face value. It could reflect a computational error in the original study, a coding error in the meta-analysis, or an unusual feature of the outcome measure. Closer inspection of the original article reveals why this estimate is so large.

Diliberti et al. manipulated portion size on different days in a cafeteria. On days when larger portions were served, customers ate more. The extraordinarily large standardized effect arose from the measure of entrée consumption. Customers who finished their entire entrée all received the same maximum value—the amount of food that had been served. This pile-up at the upper limit compressed the variance in consumption. Because a standardized effect size divides the mean difference by the standard deviation, an unusually small standard deviation can produce an extraordinarily large value of $d$.

The article also reported total caloric intake for the entire meal, an outcome with substantially more variability. The absolute difference between conditions remained large, but the standardized effect was a much more plausible $d\approx .80$. Thus, the substantive finding is robust: larger portions increased food consumption. However, the $d=3.08$ estimate exaggerates the practical magnitude of the effect because of the unusually small variance in the outcome measure. It therefore contributes artificial heterogeneity to the meta-analysis—heterogeneity caused by the construction of the effect-size measure rather than by a substantively stronger effect of the intervention. No continuous theoretical moderator of nudging should be expected to explain such an extreme value.

Once the source of an extreme effect size has been identified, it is reasonable either to replace it with a more appropriate effect-size estimate or to exclude it in a sensitivity analysis when estimating the remaining unexplained heterogeneity. This is different from deleting an outlier simply because it is extreme. The reason for treating this value differently is known: $d=3.08$ is inflated by restricted variance in the particular outcome that was selected to represent the study.

Another question is whether a portion-size manipulation should be included in a meta-analysis of nudging in the first place. Maier et al. (2022) tried to address heterogeneity by conducting separate analyses for six broad domains. For the food domain, the Bayes factor favored a nonzero average effect, although the evidence was weak. In the other five domains, the Bayes factors pointed toward a mean of zero to varying degrees ($BF_{01}>1$), although only three reached the conventional 3:1 threshold for moderate evidence. Nevertheless, these results have been summarized as showing that “using more precise estimates, the results revealed evidence against the efficacy of nudges in most domains” (PsyPost). This statement illustrates how easily results from RoBMA can be misinterpreted. A Bayes factor concerns the average effect within a domain. Even a Bayes factor of 1000:1 in favor of a mean of zero would not show that nudges have no effects when the same model estimates substantial heterogeneity. Thus, all reported means should be reported with the estimate of heterogeneity to avoid this misinterpretation.

Maybe the old criticism that heterogeneous meta-analyses compare apples and oranges was right after all (Sharpe, 1997). If apple studies and orange studies have systematically different effect sizes, their average is informative about neither apples nor oranges. Combining them merely creates heterogeneity. When a moderator analysis eventually discovers the apple-orange distinction, researchers report separate estimates for apples and oranges. But they could have done that from the beginning, without first mixing them together and reporting an average effect of “fruit.”

Another question is whether a portion-size manipulation should be included in a meta-analysis of nudging in the first place. Maier et al. (2022) also examined six different domains and found support for an average effect in the food domain. However, their Bayesian statistic pointed towards evidence for the null hypothesis to various degrees (BF01 > 1) in five other domains. This finding has been reported as “using more precise estimates, the results revealed evidence against the efficacy of nudges in most domains” (psypost.org). This claims illustrates how confusing RoBMA results can be and how they are easily misunderstood. Most importantly, even a Bayes-Factor of 1000:1 in favor of a mean of zero in these analysis does not justify the claim that there is no effect when studies also show heterogeneity.

Maybe the old criticism that heterogeneous meta-analyses are like comparisons of apples and oranges was right after all (Sharpe, 1997). If apple studies and orange studies have systematically different effect sizes, their average is informative about neither apples nor oranges. Combining them merely creates heterogeneity. When a moderator analysis eventually discovers the apple-orange distinction, researchers report separate estimates for apples and oranges. But they could have done that from the beginning, without first mixing them together and reporting an average effect of “fruit.”Maybe the old criticism that heterogeneous meta-analyses are like comparisons of apples and oranges was right after all (Sharpe, 1997). If apple studies and orange studies have systematically different effect sizes, their average is informative about neither apples nor oranges. Combining them merely creates heterogeneity. When a moderator analysis eventually discovers the apple-orange distinction, researchers report separate estimates for apples and oranges. But they could have done that from the beginning, without first mixing them together and reporting an average effect of “fruit.”

This points to an even more fundamental problem with averaging positive and negative population effects. Suppose eating an apple a day keeps the doctor away, whereas eating an orange a day makes people sick. Both are real effects. They receive positive or negative signs only because of a convention about which direction is desirable or consistent with a theoretical prediction. If the two effects are equal in magnitude but opposite in sign, their average is zero. But this does not imply that there is no effect. It implies that two real effects cancel mathematically when they are averaged. The scientifically useful conclusion would be to eat apples and avoid oranges—not that “fruit has no effect.”

Meta-analyses in medicine often take a much narrower approach. Rather than combining fundamentally different treatments and outcomes under a broad label, they typically focus on closely related interventions, comparable control conditions, similar populations, and clearly defined outcomes. As a result, many clinically informative meta-analyses contain fewer than 20 studies. That may sound less impressive than a meta-analysis based on more than 200 publications, but in meta-analysis, more is not necessarily better. Often, less is more—except when it comes to the sample sizes of the original studies. A meta-analysis gains credibility from the comparability of the studies, not from the sheer number of papers thrown into it.

The broader lesson is that methods designed to evaluate the credibility of published research must themselves be evaluated critically. Meta-scientists are subject to the same incentives as other scientists, and sophisticated statistical methods do not eliminate the need to examine assumptions, alternative explanations, and the full implications of a fitted model. In this case, focusing on a grand mean close to zero obscured substantial heterogeneity, little evidence of selection for statistical significance, and considerable evidence for positive and potentially replicable effects. A credibility revolution should make scientific claims more carefully calibrated, not merely replace one headline number with another.

Selected References

Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6, Article e0000130. https://doi.org/10.15626/MP.2022.2981

Maier, M., Bartoš, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L., & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences, 119(31), e2200300119. https://doi.org/10.1073/pnas.2200300119

Sharpe D. (1997). Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away. Clinical psychology review, 17(8), 881–901. https://doi.org/10.1016/s0272-7358(97)00056-1

Szászi, B., Higney, A., Charlton, A., Gelman, A., Ziano, I., Aczel, B., Goldstein, D. G., Yeager, D. S., & Tipton, E. (2022). No reason to expect large and consistent effects of nudge interventions. Proceedings of the National Academy of Sciences, 119(31), e2200732119. https://doi.org/10.1073/pnas.2200732119

Conclusion: A meta-analysis of clinical trials in Western nations that compare psychotherapy to treatment as usual shows an average effect size of half a standard deviation with 95% of effects ranging from a quarter to three-quarters of a standard deviation. Thus, psychotherapy is an important component of treatment for depression.

Introduction

Psychotherapy works, $p < .05$.

However, a statistically significant result alone does not really help practitioners and patients assess the benefits of psychotherapy. The important question is not “Is the effect greater than zero?” but “How much does psychotherapy help?”

A single study cannot answer this question precisely because psychotherapy studies tend to have relatively small samples and therefore substantial sampling error. Meta-analysis was developed to address this problem. A simple meta-analysis combines the effect-size estimates from individual studies while taking their sampling error (standard errors) into account to obtain a more precise estimate of the average effect. As the number of studies increases, sampling error in the average estimate can become practically negligible.

The headline estimate of psychotherapy effectiveness is about 70% of a standard deviation on a measure of clinical depression such as the Beck Depression Inventory ($g=.72$), with very little sampling error ($SE=.03$; Plessen et al., 2023). This suggests that psychotherapy has a substantial positive effect. However, although the average effect can be estimated very precisely, two other sources of uncertainty undermine the usefulness of this estimate: (a) publication bias and (b) variation in true effect sizes across populations, treatments, control conditions, and other study characteristics.

Publication Bias

One problem is publication bias. Studies that show that psychotherapy is effective may be more likely to be published than studies that fail to show an effect. If so, the published literature will exaggerate effectiveness. Plessen et al. (2023) therefore included PET–PEESE, a statistical method designed to estimate the effect after accounting for a possible relationship between effect size and sampling error.

The method exploits the fact that small studies need larger estimated effects to achieve statistical significance, whereas large studies can achieve significance with smaller estimated effects. If statistically significant findings are preferentially published, this creates a relationship between sampling error and observed effect sizes. PET–PEESE uses this relationship to estimate what the effect would be as sampling error approaches zero. Across the PET–PEESE analyses in Plessen et al.’s multiverse, the estimated effect averaged only $g=.18$.

The problem is that publication bias is not the only reason why effect sizes might be related to study size. Small and large studies may differ systematically in other ways. For example, smaller studies might provide more intensive and costly treatments, whereas larger trials might use briefer or online interventions. Control groups, patient populations, and other study characteristics may also differ with study size. If these characteristics genuinely influence treatment effects, PET–PEESE can mistake real differences in treatment effectiveness for publication bias and adjust the effect downward too strongly.

We are therefore left with remarkably different answers to a seemingly simple question. A conventional meta-analysis suggests an effect of $g=.72$, whereas PET–PEESE suggests an effect closer to $g=.18$. Is psychotherapy highly effective, only modestly effective, or somewhere in between?

Fortunately, other methods can help distinguish publication bias from genuine variation in treatment effects. The first aim of this blog post is to use these methods to obtain a more credible estimate of the effectiveness of psychotherapy.

Heterogeneity

The second problem is heterogeneity. The effectiveness of psychotherapy may vary across populations, types of treatment, control conditions, and other study characteristics. In other words, there may be no single effect size that describes the effectiveness of psychotherapy under all conditions. An average can still be calculated, but it may not provide a useful prediction for any particular treatment setting.

In meta-analysis, variation in the true effect sizes across studies is called heterogeneity. In Plessen et al.’s full three-level meta-analysis, the average effect was g = .72, but the between-study variance was tau² = .364, corresponding to tau = .60. Thus, although the average effect was estimated very precisely, the true effects varied substantially from study to study.

Assuming a normal distribution of true effects, approximately 95% of study-level effects would be expected to fall between g = -.46 and g = 1.90. Thus, the same meta-analysis that estimates the average effect very precisely also allows for true effects ranging from moderately favoring the control condition to extremely large benefits of psychotherapy.

This wide range shows why a precisely estimated average does not necessarily answer the question, “How effective is psychotherapy?” An average of g = .72 tells us that psychotherapy is beneficial on average, but it provides little guidance about the effect we should expect in a particular population, treatment, or comparison condition.

To obtain more informative estimates, we need to understand why treatment effects vary across studies. The second aim of this blog post is therefore to identify one or more groups of studies with reasonably similar true effect sizes and to estimate how effective psychotherapy is under more specific conditions.

This goal may seem counterintuitive because a general rule in statistics is that larger samples are more informative. This is clearly true within an individual study: larger samples reduce random sampling error and produce more precise estimates. In meta-analysis, however, simply adding more studies does not necessarily make the answer more informative. Adding studies reduces sampling error in the average effect, but it can also increase heterogeneity if the added studies examine different populations, treatments, countries, or control conditions.

In this sense, less can be more. A meta-analysis of a smaller but more comparable set of studies may provide a more useful estimate than a much larger meta-analysis that averages over systematically different conditions. For example, an estimate of the effectiveness of cognitive behavioral therapy for adults within a particular healthcare setting and relative to a particular control condition may be more informative than a single average that combines different countries, therapies, populations, and comparison groups.

The goal is therefore not to make the meta-analysis as large as possible, but to define groups of studies for which an average effect has a clear and useful interpretation.

Munder et al.’s Meta-Analysis

The Study

I used Munder et al.’s (2022) meta-analysis because it examined a particularly plausible moderator: the treatment received by patients in the control group. In psychotherapy trials, the control condition plays a role similar to the comparison condition in a drug trial. Some patients are assigned to a waitlist and may receive little or no treatment during the study. In other trials, patients in the control group receive treatment as usual (TAU), and psychotherapy is added only for the treatment group. For example, both groups may receive antidepressant medication, while only the treatment group also receives psychotherapy.

Treatment as usual can also vary substantially in intensity and effectiveness. The more effective the treatment received by the control group, the smaller the additional benefit of psychotherapy is likely to be. This means that an average effect size that combines waitlist controls with different forms of treatment as usual may obscure meaningful differences in effectiveness.

Another plausible moderator is the country in which the study was conducted. Even without a specific hypothesis about cultural differences in the treatment of depression, psychological effects often vary across countries and cultures. Country can therefore serve as a useful proxy for differences in culture, healthcare systems, recruitment practices, and other contextual factors that may influence treatment effects.

A third possible moderator is the type of psychotherapy. Although meta-analyses suggest that many forms of psychotherapy are effective, it remains possible that some treatments produce larger effects than others.

The goal is not only to determine whether these moderators predict effect sizes. Even more important is to determine how much heterogeneity remains after accounting for them. If control condition, country, and treatment type explain a substantial portion of the variation across studies, we can obtain more informative estimates for specific sets of conditions—for example, the expected effect of CBT in U.S. studies compared with a waitlist control group.

Random Effects Meta-Analysis

I first analyzed the data with a standard random-effects meta-analysis using the R package metafor. This model estimates the average effect size and the amount of heterogeneity while assuming that the observed studies are not distorted by publication bias. The analysis produced an average effect of g = .72, SE = .08, together with substantial heterogeneity, tau = .67. These results closely replicate the findings of Plessen et al. (2023): psychotherapy appears highly effective on average, but treatment effects vary greatly across studies.

PET

I next applied PET, one of the regression-based methods used by Plessen et al. (2023) to address publication bias. PET tests whether effect-size estimates are related to their standard errors and extrapolates this relationship to a hypothetical study with no sampling error.

The analysis showed a strong relationship between effect sizes and sampling error, b = 3.29, SE = .57. The estimated intercept at zero sampling error was slightly negative, g = -.07, SE = .15, and not significantly different from zero. Taken literally, PET would therefore suggest that there is no convincing evidence for an average psychotherapy effect after correcting for publication bias.

However, this interpretation depends critically on the assumption that the relationship between effect size and sampling error is caused by publication bias. Moreover, substantial heterogeneity remained even after fitting PET, tau = .57. Thus, the model still allows for large positive effects in some studies while simultaneously estimating an average effect close to zero.

zcurve3

To examine publication bias with fewer assumptions about the relationship between effect size and study size, I also analyzed the data with zcurve3. Z-curve converts each effect size and its standard error into a z-value. A two-sided z-value greater than 1.96 is statistically significant. Z-curve uses the distribution of statistical evidence, particularly the significant z-values, to estimate the underlying distribution of evidential strength and to predict how many significant and non-significant results should be observed.

Figure 1 shows that the fitted distribution predicts somewhat more non-significant results than were actually observed. The expected discovery rate—the proportion of statistically significant findings predicted by the model—is only 39%, whereas the observed discovery rate is higher. This pattern is consistent with some selection for statistical significance. However, unlike PET, z-curve is very uncertain about the magnitude of this bias. The 95% confidence interval for the expected discovery rate extends as high as 83%, so the data are also compatible with little or no excess of significant findings.

Z-curve also reveals substantial variation in the strength of evidence across studies. Studies with non-significant z-values have low estimated power, whereas many of the statistically significant studies have moderate to high power, ranging from approximately 62% to 98%. The average estimated power of the significant studies—the Expected Replication Rate—is 74%. Z-curve also estimates the maximum proportion of statistically significant findings that could be false positives. Although the upper bound of the 95% confidence interval reaches 42%, even this conservative bound implies that the majority of statistically significant findings are unlikely to be false positives.

Zcurve3 can also provide estimates on the effect-size scale. The estimated overall mean effect is g = .33, with a wide 95% confidence interval ranging from g = .12 to g = .90. This interval contains both the small PET estimate and substantially larger conventional random-effects estimates. Rather than forcing the data toward one of these answers, zcurve3 makes the uncertainty about publication bias explicit, rather than assuming that bias is large (PET) or that there is no bias (RMA).

Making Sense of Heterogeneity

The z-curve plot shows heterogeneity in the strength of evidence below the x-axis. These values are estimates of local statistical power for studies with z-values in the corresponding ranges. Studies with small z-values have low estimated power and therefore provide little information about the magnitude of the underlying true effect. In the present data, local power is below 50% throughout the non-significant range. Around z = 2, however, estimated local power rises above 50%, reaching 62% for results just above the conventional significance thresho ld and increasing further for larger z-values.

This provides a principled way to distinguish relatively informative from highly uncertain effect-size estimates. Importantly, the criterion is not statistical significance itself. The criterion is estimated local power. In these data, the point at which local power exceeds 50% happens to coincide approximately with the conventional significance threshold. Thus, focusing on the statistically significant results in this particular dataset amounts to focusing on the subset for which zcurve3 estimates that there is more signal than noise.

zcurve3 also estimates the maximum false-positive rate. The upper bound of the 95% confidence interval is 39%. Thus, even under a conservative interpretation, the majority of results in this more informative subset are estimated to reflect a genuine positive treatment effect.

The main advantage of zcurve3 is that it is designed to examine heterogeneity. There are two types of heterogeneity to consider. Heterogeneity in the strength of evidence and heterogeneity in effect sizes.

The z-curve plot shows heterogeneity in the strength of evidence below the x-axis. These values are local power estimates for the corresponding ranges of z-values. Average power is low for non-significant results. These are mostly studies with small samples and large sampling error, and they provide little information about the magnitude of the underlying true effects. In contrast, studies with z-values greater than about 2 have considerably greater evidential strength. For results just above the conventional significance threshold, estimated local power is already 62% and increases further for larger z-values.

zcurve3 also estimates the maximum false-positive rate. The upper limit of the 95% confidence interval is 39%. Thus, even under this conservative estimate, the majority of the significant results are expected to reflect a genuine positive treatment effect. It is therefore possible to identify a subset of studies that provides substantially stronger evidence about treatment effectiveness.

zcurve3 also provides effect-size estimates for subsets of studies defined by their observed z-values. For the statistically significant results, the estimated mean effect size is g = 1.41, but uncertainty is substantial, 95% CI [.56, 2.21]. Heterogeneity among these effects is also very large, tau = .98, with a wide 95% CI [.21, 1.67]. Thus, focusing on studies with stronger evidence does not solve the heterogeneity problem. We still need to ask why some studies produce much larger effects than others. This requires examining potential moderators—that is, study characteristics that explain variation in effect sizes—with the goal of identifying one or more groups of studies for which an average effect size has a meaningful interpretation.

It therefore makes sense to examine studies with strong evidence more closely. A complication is that unusually large observed effects may partly reflect sampling error or selection. zcurve3 addresses this problem by using empirical-Bayes shrinkage to produce adjusted effect-size estimates that pull unusually large observed effects toward more plausible underlying values. Figure 2 shows these adjusted estimates.

The first five effects, representing four studies, still appear unusually large even after this adjustment. These studies may be scientifically interesting and deserve careful examination and replication, but they are not necessarily informative about the average effect in the broader group of studies. Instead, if their unusually large effects arise from study characteristics that are not shared by the remaining studies, combining them with the rest simply increases unexplained heterogeneity.

Chiang et al. provides a useful example. It is the only study from Taiwan in this meta-analysis. Consequently, its exceptionally large effect is perfectly confounded with the individual study: with only one Taiwanese study, we cannot determine whether the effect reflects Taiwan, some other feature of the study, or sampling variation. The study is informative about that particular Taiwanese sample, but it cannot establish a general Taiwanese treatment effect and it contributes little to estimating the effect for a population of Western studies. For that purpose, including it mainly adds heterogeneity that cannot be explained or generalized.

To produce a more homogeneous and interpretable set of studies, I applied several additional restrictions. First, I removed five studies with unique characteristics or questionable reporting that made their unusually large effects difficult to interpret or generalize. Second, I removed studies with large sampling error (SE > .30). This is actually a relatively modest restriction compared with Stanley, Jarrell, and Doucouliagos (2010), who proposed estimating meta-analytic effects from only the most precise 10% of published estimates when publication selection is a concern. Their argument is that highly imprecise studies can contribute more noise than useful information.

I also removed the small number of studies with negative effect-size estimates because there were too few to estimate directional selection bias separately. Finally, I excluded samples from non-Western countries. Previous research has shown that psychological effects often vary across countries and cultures, and the present data also suggested substantial country differences. Combining isolated studies from very different populations into a single average would therefore increase unexplained heterogeneity without producing an estimate that clearly applies to either population.

The following analyses therefore focus on a smaller and more homogeneous set of studies and ask a more specific question: How effective is psychotherapy for depression under reasonably comparable conditions in Western countries?

Refined Sample

Random Effects Meta-Analysis

The average effect-size estimate decreased from g = .72 to g = .50. More importantly, heterogeneity decreased dramatically, from tau = .67 to tau = .22. The corresponding 95% prediction interval ranges from approximately g = .06 to g = .94. Thus, although treatment effects still vary substantially, nearly the entire predicted distribution of true effects is now positive.

I next examined study country, waitlist versus treatment-as-usual (TAU) control groups, and type of treatment as moderators. Individual differences associated with country and treatment type were generally modest. The clearest moderator was the control condition: studies using a waitlist produced effects approximately g = .23 larger (SE = .06) than studies using TAU controls.

More important than any individual moderator coefficient, however, was their combined ability to explain heterogeneity. After accounting for country, control condition, and treatment type, residual heterogeneity decreased further to tau = .13. Centered around an overall effect of approximately g = .50, this corresponds to a range of true effects of roughly g = .24 to g = .76.

This is a much more informative answer to the question of psychotherapy effectiveness. Instead of an average surrounded by effects ranging from negative to extremely large, the refined analysis suggests that under reasonably comparable conditions psychotherapy produces effects ranging from about one-quarter to three-quarters of a standard deviation, with an average of about half a standard deviation.

PET

The PET regression again showed a strong relationship between effect-size estimates and their standard errors, b = 2.23, SE = .52. In PET, this relationship is typically interpreted as evidence that effect-size estimates are increasingly inflated as sampling error increases.

However, when country, control condition, and treatment type were added to the regression, the coefficient for sampling error was reduced by about half, from b = 2.23 to b = 1.10, and was no longer statistically significant (SE = .57).

This result illustrates a fundamental limitation of regression-based tests of publication bias. A correlation between effect-size estimates and standard errors does not reveal why the correlation exists. PET attributes this pattern to publication bias, but the same pattern can arise when small and large studies differ systematically in ways that genuinely affect treatment outcomes.

In these data, much of what initially looked like publication bias could be explained by study characteristics. In other words, the apparent bias signal was partly heterogeneity in disguise.

zcurve3

zcurve3 does not yet allow moderators to be specified directly in the model. However, there are two ways to combine zcurve3 with moderator analyses. One approach is to first fit a random-effects meta-regression, remove the variation explained by the moderators, and then analyze the resulting moderator-adjusted effect sizes with zcurve3. To preserve the overall treatment effect, I added the overall mean effect back to the residuals. Thus, the adjusted effects retain the average psychotherapy effect while removing systematic variation associated with country, control condition, and treatment type.

A second approach is to use the bias-corrected study-level effect-size estimates produced by zcurve3 and analyze these estimates in a conventional meta-regression. Here, I used the first approach.

The z-curve plot of the moderator-adjusted effects shows no evidence of excess significance. The observed discovery rate was 79%, almost identical to the expected discovery rate of 78%. If publication selection had produced a large excess of significant findings, the observed discovery rate should have been substantially higher than the rate predicted from the evidential strength of the studies. Instead, the two estimates differ by only one percentage point.

Visual inspection leads to the same conclusion. The fitted z-curve closely reproduces the observed distribution, including the non-significant results to the left of the significance threshold. Thus, after removing systematic variation associated with the moderators, the strong publication-bias signal suggested by PET is no longer apparent.

The overall mean effect is estimated to be g = .53, 95%CI [.09, .59], but the confidence interval remains relatively wide because zcurve3 continues to allow for the possibility of publication selection. In this model, some non-significant results are inferred rather than directly observed, which creates additional uncertainty about the overall mean. If the model were respecified under the assumption that there is no publication bias and all observed results were analyzed directly, the confidence interval would become much narrower—but the analysis would then essentially reduce to another conventional random-effects meta-analysis.

More informative is the convergence of the point estimates across models with very different assumptions. The conventional random-effects model and the selection model both estimate an average effect of approximately half a standard deviation. The mean among the significant results is similarly estimated at g = .52, and the median is g = .54. Heterogeneity is also small, tau = .09, 95% CI [.06, .21] compared to the heterogeneity for all studies (.98, 95%CI [.21, 1.66]. Thus, the smaller dataset does produce a more informative average estimate.

The agreement between the RMA results and these results with a selection model provides an important robustness check. A model that assumes the observed literature is unbiased and a model that explicitly allows for missing non-significant results arrive at essentially the same estimate of psychotherapy effectiveness.

Step-Function Selection Model

To further examine the robustness of the results, I analyzed the data with a step-function selection model (Hedges & Vevea, 1996; Vevea & Hedges, 1995). Unlike PET, this model does not infer publication bias from a correlation between effect sizes and standard errors. Instead, it allows the probability that a result is observed to differ across ranges of p-values.

I specified selection steps at p = .025, corresponding to statistical significance at p = .05 for a two-sided test, and at p = .50, which separates positive from negative effect-size estimates. Because negative effect-size estimates had already been excluded from this analysis, the estimated weight for this final interval is necessarily close to zero and is not substantively informative.

The selection model estimated an average effect of g = .45, SE = .03, with little remaining heterogeneity, tau = .10, 95% CI [.00, .17]. The model did find some evidence of preferential selection of statistically significant results. The relative weight for non-significant positive results was .54, 95% CI [.27, .99]. In other words, the model estimates that non-significant positive results may have been only about half as likely to be observed as statistically significant results, although there is considerable uncertainty about the magnitude of this selection.

Most importantly, allowing for this degree of publication selection had little effect on the estimated treatment effect. Because most results in this refined sample were statistically significant, the estimated mean was largely determined by studies in the region receiving full selection weight.

Thus, a second selection model, based on very different assumptions from zcurve3, reaches essentially the same conclusion: some publication selection may be present, but correcting for it does not materially change the estimated effect of psychotherapy or the amount of remaining heterogeneity.

Robust Bayesian Model Averaging (RoBMA)

A final robustness analysis used Robust Bayesian Model Averaging (RoBMA). RoBMA is useful here because it does not require us to decide in advance whether publication bias should be modeled with PET–PEESE, a step-function selection model, or not at all. Instead, it simultaneously fits a large set of models—including conventional random-effects models, PET–PEESE models, and several step-function selection models—and gives greater weight to models that are better supported by the data.

In the present data, the PET-type models received little support, whereas the selection models received considerably more weight. Thus, when RoBMA was allowed to choose among competing explanations of publication bias, the data favored the step-function approach rather than the PET interpretation.

The posterior probability that the average treatment effect is different from zero was essentially 1.00. There was also strong evidence for remaining heterogeneity, with a posterior inclusion probability of .94, and some evidence for publication bias, with a posterior inclusion probability of .72. These probabilities tell us whether these components are likely to be present, but not how large they are.

RoBMA reports both unconditional estimates, which average across models that include and exclude a particular component, and conditional estimates based only on models in which that component is present. In this analysis, the estimates were essentially identical because the evidence for a treatment effect was overwhelming. The model-averaged effect size was g = .46, SE = .04, and residual heterogeneity was small, tau = .10, 95% interval [.05, .17].

Most importantly, RoBMA independently reproduced the result obtained with the step-function model: some publication selection may be present, but accounting for it leaves the estimated psychotherapy effect at roughly half a standard deviation, with little remaining heterogeneity.

Discussion

What Have We Learned About Meta-Analysis?

This reanalysis illustrates two problems that can make meta-analytic estimates misleading even when sampling error is very small.

First, evidence of a relationship between effect sizes and standard errors should not automatically be interpreted as publication bias. Regression-based methods such as PET can mistake systematic differences between studies for selective publication. In the present data, PET initially suggested that the psychotherapy effect was close to zero. However, the relationship between effect sizes and standard errors was substantially reduced after country, control condition, and treatment type were included as moderators. What initially looked like publication bias was therefore, at least in part, heterogeneity in disguise.

Selection models that do not rely on this correlation produced a very different conclusion. zcurve3 found no clear evidence of excess significance, while the step-function selection model and RoBMA suggested that some publication selection may nevertheless be present. Importantly, correcting for this possible selection had little effect on the estimated treatment effect. Across these different models, estimates converged at approximately half a standard deviation.

The standard recommendation is therefore to use multiple methods and to include selection models. More importantly, discrepancies need to be examined in terms of the different assumptions that models make. Here this analysis showed that one model with different assumptions led to different results because the assumption was false.

Second, meta-analysis should not simply maximize the number of studies and then report heterogeneity as an unfortunate side effect. A precise average of studies that estimate systematically different effects may have little practical meaning. The goal should be to identify moderators that explain these differences and, where necessary, define more homogeneous groups of studies for which an average effect has a meaningful interpretation.

This sometimes means that less is more (Cohen, 1990). A small number of reasonably comparable studies can provide a more useful estimate than a much larger collection of studies that differ substantially in populations, treatments, control conditions, and settings. Unique studies remain scientifically valuable, but a study representing a population or treatment found nowhere else in the meta-analysis cannot tell us whether its unusual effect generalizes beyond that individual study.

The goal is therefore not to eliminate heterogeneity for its own sake. It is to explain heterogeneity well enough that the resulting average describes a meaningful population of studies.

What Have We Learned About Psychotherapy for Depression?

The substantive conclusion is considerably clearer than the original range of meta-analytic estimates from approximately g = .18 to g = .72 suggested.

For reasonably comparable studies conducted in Western countries, psychotherapy produces an average improvement in depression of approximately half a standard deviation compared with treatment as usual. After accounting for country, control condition, and treatment type, remaining heterogeneity was relatively small. The distribution of true study-level effects was approximately g = .2 to g = .8, suggesting that psychotherapy generally produces effects ranging from small to large rather than effects ranging from harmful to extremely large.

The clearest moderator was the control condition. Effects in studies using a waitlist were approximately .2 standard deviations larger than effects in studies comparing psychotherapy with treatment as usual. Country and treatment type also explained some variation, although their individual differences were generally smaller.

Thus, the best answer to the question “How effective is psychotherapy for depression?” is not a single universal number. For the types of studies examined here, a reasonable estimate is about g = .5 compared with treatment as usual, with somewhat larger effects against waitlist controls and modest variation across countries and treatment types.

These findings also point to the limits of further small psychotherapy trials. Small studies were sufficient to establish that psychotherapy works. They are much less useful for determining whether one therapy works slightly better than another or which patients benefit most. Detecting these smaller differences requires much larger samples in which other study characteristics are held reasonably constant.

The next step therefore is not simply to accumulate more small studies. It is to conduct large, coordinated, multi-site studies that can estimate treatment effects precisely enough to determine which treatments work best, under which conditions, and for which patients.

Conclusion

Psychotherapy works, p<.05, is not a scientific conclusion, even when it is based on a meta-analysis of hundreds of studies. Here I showed that the existing evidence allows for a more informative answer, at least for the conditions represented by studies conducted in Western countries. Compared with treatment as usual, psychotherapy reduces depression symptoms by about half a standard deviation on average. This is a clinically meaningful effect, but it is still an average. Across reasonably comparable studies, typical treatment effects appear to range from roughly one-quarter to three-quarters of a standard deviation. Variation across individual patients is likely to be considerably larger.

Thus, how much psychotherapy will help a particular patient remains uncertain. What the evidence does show is that, under the conditions examined here, true study-level effects that favor the control condition appear to be uncommon. Given the substantial average benefit of psychotherapy for patients with depression, psychotherapy should be offered as a core treatment option.

“Priming exercise” and psychological priming share a label, not necessarily a mechanism. Deliberate preparation for a known future competition provides no obvious need for unconscious goal activation, and the article presents no evidence that such activation actually occurs. The shared terminology leaves the proposed explanation primed for equivocation.

Holmberg and Kelly (2026) provide a useful critique of physiological explanations for “priming exercise,” but their attempt to connect this literature to psychological priming introduces a much less plausible mechanism. The connection appears to arise largely because the two literatures happen to use the same word. That shared terminology leaves the argument primed for equivocation.

In sports physiology, “priming exercise” refers to a deliberately performed bout of exercise intended to improve performance later that day. In cognitive and social psychology, priming refers to prior exposure to a stimulus that subsequently alters processing or behavior, sometimes without awareness or conscious intention. These are fundamentally different uses of the term. The fact that both involve something occurring before something else does not imply that they share a psychological mechanism.

This distinction becomes especially important when the authors invoke nonconscious goal priming. They suggest that exercise might activate performance goals and related behavioral representations that persist until later testing. They cite classic social-psychological priming research, including Bargh and colleagues, to support the possibility that goals can be activated without awareness and subsequently influence behavior.

But the proposed mechanism is poorly matched to the phenomenon being explained. An athlete does not ordinarily encounter a “priming exercise” incidentally. The athlete performs it because a competition or performance test is coming later. The later performance goal is therefore likely to have been activated before the exercise begins:

competition later → intention to prepare → priming exercise.

The goal is not plausibly dormant until the exercise somehow activates it unconsciously. Indeed, the goal is probably one of the reasons the athlete performs the exercise in the first place. During the exercise the athlete may also consciously think about the competition, technique, pacing, readiness, or expected benefits. Under those circumstances, invoking nonconscious goal activation is not merely unnecessary; the proposed causal sequence is almost backwards.

The authors’ own examples illustrate the problem. They discuss athletes rehearsing particular pacing strategies, using metronomes, receiving verbal cues, believing that squatting will improve subsequent jumping, and developing confidence or expectations about later performance. These are readily understood as deliberate preparation, task practice, expectancy, motivation, or attentional effects. None requires a nonconscious priming mechanism.

The scientific evidence offered for the nonconscious account is also weak. The article provides no exercise experiment demonstrating that a priming-exercise bout activates a previously inactive goal outside awareness and that this activation subsequently causes improved performance hours later. In fact, the authors repeatedly acknowledge that these possibilities “may” occur, are “hypothesized,” or “have yet to be directly examined.” Thus, the proposed mechanism is not an empirical finding from the exercise literature.

Instead, support is imported from a different literature on behavioral and goal priming. That literature is itself scientifically controversial, and citing classic demonstrations does not establish that the same mechanism operates in an entirely different situation involving intentional athletic preparation. The conceptual inference appears to be:

psychology calls something “priming”

• exercise science calls something “priming”
  → psychological priming may explain exercise priming.

But identical terminology is not evidence of mechanistic equivalence.

The distinction matters because the paper already identifies much more plausible explanations. Task-specific practice, motor learning, expectancy, researcher effects, motivation, and explicit performance preparation could all produce later performance changes. These mechanisms fit the actual structure of the situation: athletes know that performance is coming and intentionally prepare for it. The nonconscious goal-priming hypothesis adds an unnecessary and poorly supported causal layer.

The problem can therefore be summarized simply: an implausible mechanism is invoked to explain effects that have not been shown to require that mechanism, and the empirical justification comes largely from a separate literature that happens to use the same word.

Mir, I. A. (2026). Influencer’s physical attractiveness and content aesthetics: Conscious and preconscious determinants of fashion-branded content engagement on Instagram. Journal of Creative Communications, 21(2), 201–219. https://doi.org/10.1177/09732586241288672

“The authors invoke an unproven perception–behavior mechanism to explain causation that was never observed. A direct path in a cross-sectional SEM establishes neither causation nor preconscious processing.”

Mir (2026) examines whether fashion influencers’ physical attractiveness and the aesthetics of their branded content predict followers’ engagement on Instagram. The study uses survey responses from 300 followers of 15 fashion influencers in Pakistan and analyzes the proposed relationships with structural equation modeling and mediation analyses. The main empirical finding is straightforward: followers who rate influencers and their content more positively also report more favorable attitudes and greater engagement. The methodological problem is that the article draws causal and psychological-process conclusions that the design cannot support.

The most serious problem is that all variables were measured in a single cross-sectional self-report survey. Participants simultaneously reported how attractive they considered the influencer, how aesthetically pleasing they considered the content, their attitude toward that content, and how often they viewed, liked, commented on, and shared it. Nothing was manipulated, and there was no temporal ordering of the variables. Nevertheless, the article repeatedly describes attractiveness and aesthetics as factors that “cause,” “trigger,” “stimulate,” or “activate” engagement. Those causal statements do not follow from the design.

For example, the proposed model assumes

attractiveness → attitude → engagement.

But the same covariance pattern is compatible with numerous alternatives. Followers who engage frequently with an influencer may develop more positive attitudes and subsequently rate that influencer as more attractive. A general liking or identification with the influencer could simultaneously increase attractiveness ratings, content-aesthetic ratings, attitudes, and engagement. The structural equation model cannot distinguish among these explanations.

The sampling procedure makes this problem particularly important. Participants had to have followed one of the selected influencers for more than six months. Thus, the sample is already conditioned on sustained interest in the influencer. People who disliked the influencer, found the content unattractive, or disengaged from it are systematically less likely to appear in the sample. The resulting correlations describe differences among an already selected group of followers; they provide weak evidence for the article’s practical recommendation that firms should hire physically attractive influencers because attractiveness causes engagement.

The article’s central methodological error is even more fundamental. It claims to distinguish a “conscious” route from a “preconscious” route. The mediated path

attractiveness/aesthetics → attitude → engagement

is interpreted as conscious influence, whereas a remaining direct path from attractiveness or aesthetics to engagement is interpreted as evidence of a preconscious perception–behavior process.

A direct regression coefficient is not a measure of unconscious processing.

If attractiveness predicts engagement after statistical adjustment for an explicit attitude measure, this merely shows residual covariance between those variables. That residual association could reflect measurement error in attitude, omitted mediators, stable preferences, common response tendencies, reverse causation, or numerous other processes. Nothing in the study measures awareness, intention, automaticity, processing speed, or participants’ ability to report the causes of their behavior. Consequently, the data provide no evidence that engagement was “preconscious” or unintentional.

For the same reason, the indirect path through an explicit attitude measure does not establish a conscious causal mechanism. Participants consciously completed the attitude questionnaire, but that does not mean the psychological process producing their engagement operated consciously. Statistical mediation and conscious psychological mediation are different concepts.

This problem is especially consequential because the “preconscious” interpretation is one of the article’s principal theoretical contributions. The authors explicitly invoke the perception–behavior literature, including Bargh et al. (1996), to justify the claim that a significant direct path demonstrates automatic behavior. But the current research contains none of the experimental procedures that would be required to test an automatic perception–behavior effect. The SEM therefore cannot adjudicate between conscious and unconscious processes.

The measures themselves also create substantial interpretive problems. On page 209, physical attractiveness is measured with “stylish,” “good looking,” “sexy,” and “elegant.” Content aesthetics is measured with “striking,” “wonderful,” “fascinating,” and “lovely,” while attitude is measured with “pleasant,” “good,” “likeable,” and “my favourite.” These constructs are conceptually and evaluatively intertwined. “Wonderful” and “lovely,” for example, are not narrowly aesthetic judgments, while “stylish” and “elegant” are not purely measures of physical attractiveness. Much of the model may therefore reflect a broad positive-evaluation factor rather than distinct psychological constructs connected by causal pathways.

The observed correlations are consistent with this concern. Physical attractiveness correlates .64 with content aesthetics and .64 with attitude; content aesthetics correlates .66 with attitude and .68 with reported content consumption. Demonstrating discriminant validity with the Fornell–Larcker criterion does not eliminate the possibility that halo effects or general liking strongly influence all of these ratings.

The attempt to dismiss common-method bias is also inadequate. All predictors, mediator variables, and outcomes were obtained from the same respondent at the same time using similar rating formats. The authors test whether several sets of items can be represented by a single latent factor and conclude that poor single-factor fit shows that common-method bias is not important. That conclusion does not follow. Common-method variance does not require every item to load on one factor. Several distinguishable constructs can coexist while correlations among them are inflated by shared method, evaluative consistency, acquiescence, or halo effects.

The proposed “snowball effect” suffers from the same causal problem. The authors find that self-reported consumption behaviors—viewing, reading comments, and liking—predict contribution behaviors such as commenting and sharing, and conclude that consumption gradually causes users to progress toward more active participation. Yet consumption and contribution were measured simultaneously. Someone who frequently comments and shares influencer content almost necessarily also views and consumes that content. A positive cross-sectional association therefore does not demonstrate a temporal progression from passive to active engagement. Testing a snowball process would require longitudinal evidence showing that earlier consumption predicts subsequent increases in contribution.

There may also be an unmodeled dependence problem. The 300 respondents followed one of only 15 macro-influencers. Followers of the same influencer are not necessarily independent observations. Influencers may differ systematically in appearance, production quality, follower demographics, posting frequency, and baseline engagement. Those influencer-level characteristics could generate correlations among respondent ratings. The reported SEM appears to treat all 300 followers as independent rather than accounting for clustering by influencer.

Another reporting issue concerns the engagement scale. The response categories are described as 1 = “very often,” 2 = “often,” 3 = “sometimes,” and 4 = “never.” Thus, larger numerical values indicate less engagement. Yet positive path coefficients are consistently interpreted as greater attractiveness and aesthetics producing greater engagement. The article does not clearly state in the reported method that these scores were reverse-coded. If they were reversed before analysis, that transformation should have been explicitly documented. If they were not, the substantive interpretation of the coefficients would be reversed.

Finally, the statistical success of the model should not be confused with strong evidence for the hypotheses. All five proposed hypotheses are supported, including the weakest direct attractiveness effect, β = .12, t = 2.19. The study was not preregistered, and the sample-size justification consists largely of the statement that N = 300 is sufficient for purposive sampling and structural equation modeling rather than an a priori power analysis tied to the focal effects. Good model-fit indices demonstrate that a specified covariance model can reproduce the observed covariance matrix; they do not establish that the arrows in Figure 2 represent the true causal processes.

The study therefore supports a much narrower conclusion than the article claims. Among existing long-term followers of fashion influencers, positive ratings of influencer attractiveness and content aesthetics are associated with positive attitudes and greater self-reported engagement. That descriptive association is plausible and potentially useful.

The study does not establish that physical attractiveness or content aesthetics cause engagement, that attitude mediates these effects causally, that any residual direct relationship reflects a preconscious perception–behavior mechanism, or that passive engagement develops over time into active contribution.

A suitable experimental design would manipulate influencer attractiveness and content aesthetics independently, randomly assign participants to conditions, measure actual engagement behavior, and include measures capable of testing awareness or automaticity. A longitudinal design would be required to test the proposed consumption-to-contribution “snowball” process. Without such evidence, the article’s strongest psychological claims are interpretations imposed on cross-sectional correlations rather than findings produced by the research design.

Costa, T. (2026). The Bayesian audit: Evaluating the proportionality of scientific claims to evidence—a case study on social priming and walking speed. Frontiers in Psychology, 17, 1799078. DOI: 10.3389/fpsyg.2026.1799078


This article applies Bayes’ theorem to one conveniently selected t value and calls the result a ‘Bayesian audit.’ Most readers may simply ignore it because it appeared in Frontiers in Psychology. Those who want a more substantive reason can point to this review.

Costa (2026) introduces a “Bayesian audit,” a six-step framework intended to evaluate whether the strength of scientific claims is proportional to the evidence supporting them. The idea is sensible. Statistical significance does not tell us how strongly we should believe a scientific claim, and surprising claims based on weak evidence deserve particularly careful scrutiny. Costa illustrates the proposed method with one of social psychology’s most famous findings: Bargh, Chen, and Burrows’s (1996) claim that priming college students with words related to old age caused them to walk more slowly afterward.

Unfortunately, the audit itself is problematic. It misrepresents important features of the original study, considers only a fraction of the available evidence, and reduces a question about the magnitude and robustness of an effect to a comparison between a null and an inadequately specified alternative hypothesis.

The first problem is surprisingly basic. Costa describes the original finding as based on a study with approximately t(28) = 2.0 and p ≈ .05. But Bargh et al. actually reported two elderly-priming experiments. In Experiment 2a, the comparison was t(28) = 2.86, p < .01. They then conducted Experiment 2b as a replication and again reported slower walking, t(28) = 2.16, p < .05. Costa appears to approximate the weaker second result while failing to mention the stronger first result or even that the original article contained two studies.

That is an odd starting point for an audit. If the purpose is to reconstruct how much evidence supported the claim in 1996, both original studies should be included.

Costa also incorrectly describes participants as being “subliminally exposed to words related to old age.” They were not. Participants consciously read words while completing a scrambled-sentence task. The claimed unconscious component was that participants supposedly did not realize that the elderly-related words subsequently affected their walking. Bargh et al. themselves explicitly distinguished this procedure from subliminal priming; Experiment 3 of their paper used genuinely subliminal presentation of faces.

This distinction matters because Costa uses the apparent implausibility of unconscious effects on motor behavior to motivate skeptical prior probabilities. One should at least characterize the causal claim correctly before assigning a prior to it.

The treatment of replication evidence is even more problematic.

Costa cites Doyen et al. (2012) and Harris et al. (2013) as subsequent replication attempts. Doyen et al. did replicate the elderly-walking paradigm. In a substantially larger study using automated measurement, they found essentially no priming effect. Their second experiment further suggested that experimenter expectations could influence the result.

Harris et al. (2013), however, did not replicate elderly priming at all. They attempted to replicate Bargh et al.’s 2001 high-performance goal-priming experiments, in which achievement words were supposed to improve performance on a cognitive task. Calling Harris et al. a replication of the elderly-walking finding is simply an error.

More importantly, why is a Bayesian audit conducted in 2026 based primarily on one t statistic from 1996?

There is now a substantial literature on behavioral priming. Dai et al. (2023), for example, meta-analyzed 351 studies and 862 effect sizes and concluded that behavioral priming effects could be detected across a large literature. Conversely, Mac Giolla et al. (2024) examined 70 close replication attempts of 49 social-priming findings. Ninety-four percent produced smaller effects than the originals, only 17% were significant in the predicted direction, and among 52 replications conducted without an original author, none was significant in the original direction; the pooled effect for those independent replications was essentially zero.

These sources do not necessarily settle the question. Meta-analyses themselves can be distorted by publication bias and other forms of selection. But that is precisely why an audit should examine them critically. An audit of a 30-year-old scientific claim should evaluate the accumulated evidence, not simply convert one selected original result into a Bayes factor.

There is an even more fundamental problem with the statistical question Costa asks.

Costa assigns prior probabilities of .05, .10, and .20 to the alternative hypothesis and combines these with an estimated Bayes factor of approximately 3. This yields posterior probabilities of .14, .25, and .43, respectively. The arithmetic is straightforward. The interpretation is not.

Why should the prior probability that the effect exists be .05 or .10?

Costa acknowledges that these values are illustrative rather than derived from an elicitation procedure. But these priors largely determine the conclusion that posterior belief remains low. Starting with a 5% probability and multiplying the prior odds by a Bayes factor of 3 inevitably produces a low posterior probability.

More importantly, what exactly is the hypothesis whose prior probability is 5%?

There is a major difference between these propositions:

elderly-related words have exactly zero effect on walking speed;

elderly-related words have some nonzero effect;

elderly-related words have a psychologically meaningful effect;

elderly priming produces effects of the magnitude originally reported;

automatic stereotype activation reliably produces consequential behavioral changes.

These are not the same hypothesis.

The scientifically interesting issue today is probably not whether the population effect is exactly zero. The effect could be d = .05 or d = .10. Such an effect would make the point null hypothesis technically false while providing little support for the dramatic theoretical interpretation of the original experiments.

This is why effect sizes matter. Bargh et al.’s original studies implied very large effects. Subsequent evidence raises the possibility that the true effect, if it exists at all, is much smaller. A useful audit therefore needs to ask how large the effect is and how precisely it has been estimated—not merely whether H0 or H1 receives the larger Bayes factor.

Costa’s procedure also conflates two different kinds of priors. One is the prior model probability: how likely H1 is relative to H0 before seeing the data. The other is the prior distribution over possible effect sizes within H1. A Bayes factor for a composite alternative necessarily depends on the latter. Yet the article emphasizes sensitivity to prior model probabilities while giving much less attention to the effect-size assumptions used to obtain BF₁₀ ≈ 3.

This creates another problem with Costa’s distinction between “evidence” and “belief.” He describes the Bayes factor as quantifying evidence supplied by the data and posterior probability as combining this evidence with prior belief. But a Bayes factor is not simply a property of the observed data. It depends on the statistical models being compared, including the distribution of effect sizes assumed under the alternative hypothesis.

There is also an internal inconsistency in the treatment of the replication evidence. Costa states that the later replication attempts yielded Bayes factors close to 1 and therefore had little evidential impact. That makes sense: a Bayes factor of 1 leaves prior odds unchanged. Yet the subsequent synthesis says that posterior belief “collapses under replication.” It cannot do both. Replications with BF ≈ 1 cannot cause posterior belief to collapse. To demonstrate such a decline, one would need Bayes factors favoring the null or another competing model and then accumulate this evidence formally.

Publication bias is another conspicuous omission. The article is motivated by the replication crisis and explicitly acknowledges that biased data limit the usefulness of evidential measures. Yet the actual Bayesian calculation treats the published Bargh result as though it were an observation selected independently of statistical significance.

That is unrealistic. A BF of 3 obtained from a randomly selected study and a BF of 3 obtained from a literature in which statistically significant and theoretically exciting findings were preferentially published do not have the same evidential implications. If selection contributed to the replication crisis, an audit of the original published evidence needs to take selection seriously.

Costa also describes the original study’s low statistical power as an additional reason for skepticism. Low power certainly matters because significant results from low-powered studies tend to exaggerate effect sizes, particularly in a selected literature. But sample size has already entered the likelihood used to compute the Bayes factor. Low power is therefore not independent evidence against the hypothesis. The additional concern arises from selection, analytic flexibility, measurement error, and effect-size inflation.

The deeper problem is that Costa reduces a scientific question to H0 versus H1 when several competing explanations exist. Doyen et al.’s work raised experimenter expectancy as one possible explanation. Other possibilities include a genuinely small priming effect, effects restricted to particular conditions or individuals, procedural artifacts, or some combination of these mechanisms. A Bayes factor contrasting an exact-zero model with a generic nonzero-effect model cannot determine which causal explanation is correct.

This is especially important because rejecting H0 would not establish Bargh’s theory. Even convincing evidence for a tiny difference in walking speed would not demonstrate that automatic stereotype activation generally controls overt behavior.

The Bayesian audit is therefore based on a reasonable principle but a poor demonstration. Scientific claims should indeed be proportional to evidence. The problem is that assessing proportionality requires accurately identifying the original evidence, considering the accumulated replication literature, evaluating publication bias, distinguishing statistical from substantive hypotheses, and estimating plausible effect sizes and their uncertainty.

Ironically, the elderly-priming case illustrates the weakness of Costa’s audit more effectively than it illustrates its strengths. A proper audit should not ask merely whether one selected t statistic changes the odds that an effect is exactly zero. It should ask what three decades of evidence tell us about the magnitude, robustness, boundary conditions, and causal interpretation of the phenomenon.

On those questions, uncertainty remains. There may be a small elderly-priming effect. The evidence does not establish that the effect is exactly zero. But neither does the accumulated evidence support taking the spectacular effects reported in 1996 at face value. The important scientific task is to estimate what effect remains after accounting for uncertainty and bias. That requires more than Bayes’ theorem applied to one conveniently chosen t value.

Przybylinski, E. (2026). Whatever you say: Changing transference-based problem behavior with if–then plans. Self and Identity. Advance online publication. https://doi.org/10.1080/15298868.2026.2613846


Przybylinski (2026) reports two experiments examining whether implementation intentions can prevent problematic behaviors triggered by transference. The theoretical logic is straightforward: subtle resemblance to a significant other is assumed to activate that person’s representation automatically, which can then influence memory, goals, and behavior. An if–then plan is proposed to prevent the activated representation from guiding subsequent behavior.

The principal concern is the credibility of the evidence on which this argument rests.

The article treats automatic behavioral priming as a well-established foundation. For example, it cites Bargh et al. (1996), Bargh et al. (2001), Chartrand and Bargh (1999), and related studies as evidence that contextual cues can automatically trigger overt behavior without awareness or intention. Yet behavioral priming is precisely one of the areas most affected by the replication crisis. The article does not discuss this change in evidential status. Thus, evidence that was considered persuasive when these studies were conducted is largely presented in 2026 as though subsequent replication failures had not occurred.

This matters because the two experiments are themselves products of that earlier research era. The author explicitly states that they were conducted as dissertation research in the late 2000s, before preregistration became standard. Study 1 included only 60 participants, or 20 participants in each of the three strategy conditions, despite testing interaction hypotheses. Study 2 included 47 participants. The manuscript provides a power justification, but because the studies were not preregistered, it is unclear whether the reported power analysis reflects a prospectively specified design decision. The reported “post hoc power” provides little additional information.

The results are remarkably successful. The focal interactions involving behavioral readiness, memory, and overt submissive behavior are all statistically significant in the predicted direction. The reported effects are also very large, frequently exceeding d = 1 and reaching d = 1.69 in Study 1. Across the major focal tests, the success rate is effectively 100%, while average observed power based on the reported effects is roughly 80%.

A perfect success rate is not impossible when power is 80%, but it is more successful than expected. Schimmack (2012) emphasized that unusually high success rates relative to estimated power can indicate that published effect sizes and success rates should not be taken at face value. Here, the outcomes within each experiment are dependent, so a simple excess-success or incredibility calculation would not be appropriate. With only two studies, there is no statistical smoking gun. Nevertheless, the combination of small samples, large effects, multiple significant focal outcomes, and absence of preregistration warrants substantial caution.

The historical timing makes this concern more important. These experiments were conducted approximately 15 years before their publication. During that interval, psychology experienced a replication crisis that directly challenged the credibility of the behavioral-priming literature on which the article relies. Yet the 2026 article does not supplement the old experiments with a contemporary, adequately powered, preregistered replication.

This is particularly striking because the central experiment is readily replicable. The author remains at the same institution where the original research was conducted, and the paradigm requires undergraduate participants rather than an unusually difficult population. A preregistered replication with a substantially larger sample could have provided highly informative evidence about whether the large effects observed in the original dissertation studies survive contemporary scrutiny.

The absence of such a replication changes how the evidence should be interpreted. The results are not invalid merely because they were collected before the replication crisis. But neither the reported effect sizes nor the perfect pattern of statistical success should be treated as reliable estimates of the underlying effects without independent replication.