Blogging about statistical power, replicability, and the credibility of statistical results in psychology journals since 2014. Home of z-curve, a method to examine the credibility of published statistical results.
Show your support for open, independent, and trustworthy examination of psychological science by getting a free subscription. Register here.
“For generalization, psychologists must finally rely, as has been done in all the older sciences, on replication” (Cohen, 1994).
DEFINITION OF REPLICABILITY: In empirical studies with sampling error, replicability refers to the probability of a study with a significant result to produce a significant result again in an exact replication study of the first study using the same sample size and significance criterion (Schimmack, 2017).
See Reference List at the end for peer-reviewed publications.
Mission Statement
The purpose of the R-Index blog is to increase the replicability of published results in psychological science and to alert consumers of psychological research about problems in published articles.
To evaluate the credibility or “incredibility” of published research, my colleagues and I developed several statistical tools such as the Incredibility Test (Schimmack, 2012); the Test of Insufficient Variance (Schimmack, 2014), and z-curve (Version 1.0; Brunner & Schimmack, 2020; Version 2.0, Bartos & Schimmack, 2021).
I have used these tools to demonstrate that several claims in psychological articles are incredible (a.k.a., untrustworthy), starting with Bem’s (2011) outlandish claims of time-reversed causal pre-cognition (Schimmack, 2012). This article triggered a crisis of confidence in the credibility of psychology as a science.
Over the past decade it has become clear that many other seemingly robust findings are also highly questionable. For example, I showed that many claims in Nobel Laureate Daniel Kahneman’s book “Thinking: Fast and Slow” are based on shaky foundations (Schimmack, 2020). An entire book on unconscious priming effects, by John Bargh, also ignores replication failures and lacks credible evidence (Schimmack, 2017). The hypothesis that willpower is fueled by blood glucose and easily depleted is also not supported by empirical evidence (Schimmack, 2016). In general, many claims in social psychology are questionable and require new evidence to be considered scientific (Schimmack, 2020).
Each year I post new information about the replicability of research in 120 Psychology Journals (Schimmack, 2021). I also started providing information about the replicability of individual researchers and provide guidelines how to evaluate their published findings (Schimmack, 2021).
Replication is essential for an empirical science, but it is not sufficient. Psychology also has a validation crisis (Schimmack, 2021). That is, measures are often used before it has been demonstrate how well they measure something. For example, psychologists have claimed that they can measure individuals’ unconscious evaluations, but there is no evidence that unconscious evaluations even exist (Schimmack, 2021a, 2021b).
If you are interested in my story how I ended up becoming a meta-critic of psychological science, you can read it here (my journey).
References
Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4, MP.2018.874, 1-22 https://doi.org/10.15626/MP.2018.874
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566 http://dx.doi.org/10.1037/a0029487
Schimmack, U. (2020). A meta-psychological perspective on the decade of replication failures in social psychology. Canadian Psychology/Psychologie canadienne, 61(4), 364–376. https://doi.org/10.1037/cap0000246
The main purpose of meta-analysis is to combine the results of quantitative studies. The simplest form averages the effect-size estimates of studies. The average is a better estimate of the population effect size because it draws on the combined sample size of the individual studies, and larger samples have less sampling error. A more sophisticated version takes each study’s sampling error into account, weighting studies with larger samples more heavily than those with smaller samples. This weighted average approximates the estimate one would obtain by pooling the raw data, under the assumption that all studies estimate the same effect.
These so-called fixed-effect meta-analyses assume that the samples are all drawn from the same population and that studies used interchangeable procedures, so that they all estimate a single population effect size. This assumption is defensible for a few phenomena grounded in basic sensory or cognitive architecture. Weber’s law, for instance — that the just-noticeable difference between two stimuli is a roughly constant proportion of their magnitude — reflects a property of sensory transduction and varies little across procedures and individuals. But such near-constants are the exception. Most effects in psychology vary with situational and personality factors, within and across cultures. This variation adds real differences in the true effect sizes across studies, over and above sampling error.
In recognition of this true variation, statisticians developed random-effects models (DerSimonian & Laird, 1986). To model the variation in population effect sizes, it is necessary to select a model that describes the unobserved variation in population effect sizes. From a statistical perspective, the easiest model assumes that population effect sizes have a normal distribution. The symmetry of this assumption implies that the estimate obtained from a set of studies is an unbiased estimate of the true average because positive and negative deviations from the average population effect size cancel each other out, just like random sampling errors cancel each other out.
From a theoretical perspective, it is unlikely that population effect sizes are normally distributed, particularly when the mean effect is small relative to the between-study heterogeneity. The reason is that effects are typically coded so that positive values are consistent with a directional theoretical prediction (e.g., conflicting colors produce slower responses in the Stroop task). While the size of the effect may vary across studies, it is often implausible that the true effect in some studies would be reversed (that conflicting colors would genuinely speed responses). A normal distribution, however, has support over the entire range of effect sizes and therefore always assigns some probability to negative true effects — an implication that is difficult to justify for a directionally predicted effect. When the mean is small relative to the heterogeneity, a normal distribution implies that a non-trivial proportion of true effects are negative, which is often theoretically implausible.
In practice, however, when heterogeneity is small, violations of the normality assumption have small and often negligible effects on meta-analytic estimates of the average effect (Hedges & Vevea, 1996). This helps explain why widely used random-effects models share the assumption that population effect sizes are normally distributed, including standard random-effects meta-analysis (DerSimonian & Laird, 1986) as implemented in commonly used software (Viechtbauer, 2010), selection-model approaches (Vevea & Hedges, 1995; Vevea & Woods, 2005), and the random-effects components of Bayesian model-averaging methods (Bartoš et al., 2023; Maier et al., 2023).
It is useful to distinguish between necessary and unnecessary assumptions. All statistical methods require some assumptions to reduce the complex information in data to an interpretable statistic. However, not all assumptions of a model are necessary. In general, models with fewer unnecessary assumptions are preferable because they are robust to violations of these unnecessary assumptions by design. For example, the Pearson correlation coefficient assumes normal distribution of the two variables, whereas rank-order correlations do not. This makes rank-order correlations robust to violations of the normal distribution assumption and it is common practice to prefer rank-order correlations when variables’ distributions are not normal.
In meta-analyses, the assumption that population effect sizes are normally distributed is an unnecessary assumption. The idea of estimating the distribution of effects without assuming a parametric form is not new. Laird (1978) and Lindsay (1983) showed that the mixing distribution in a mixture model can be estimated nonparametrically by maximum likelihood, and that the resulting estimate takes the form of a discrete distribution on a finite set of support points. This provided, in principle, a fully flexible way to model heterogeneity without assuming normality. In practice, however, fitting these models was computationally demanding at the time.
Advances in optimization and computing have since removed these hurdles. Efficient algorithms for estimating mixture models — including modern convex-optimization methods (Koenker & Mizera, 2014; Kim et al., 2020) — now make it possible to fit flexible mixtures quickly and reliably. These developments led to the widespread adoption of nonparametric and empirical-Bayes mixture models in fields that analyze large numbers of effects, most notably genomics (Efron, 2010; Stephens, 2017), where the goal is to characterize the distribution of effects across thousands of tests and to identify which are likely to be real. Similar approaches have been applied in astronomy, education, and other areas that deal with many parallel estimates.
Meta-analysis, however, did not adopt these models, and continued to rely on the normal random-effects model. One reason is that the goal of most meta-analyses has been to estimate a single average effect, for which the shape of the effect-size distribution is largely irrelevant. The additional information provided by a flexible mixture — the structure of the heterogeneity itself — was not part of the question being asked.
This has changed with growing recognition of the distinction between direct and conceptual replication (Zwaan et al., 2018). Experimental psychologists rarely repeat a procedure exactly. Instead, they typically vary paradigms across studies in the belief that this is good scientific practice — probing moderators, boundary conditions, and the generalizability of an effect. As a result, the studies combined in a meta-analysis often estimate genuinely different population effects, and meta-analyses of such conceptual replications frequently show high estimates of heterogeneity (van Erp et al., 2017). In these cases the average effect size is of relatively minor interest, because it merely centers a wide distribution of different population effect sizes — an average across paradigm variants that no single study actually instantiates. The scientifically interesting question is not the average, but how and why the effects differ.
Mixture models have another advantage over random effects models for meta-analysis. In standard meta-analytic models the population distribution of effect sizes is characterized by the mean and standard deviation of a normal distribution. Importantly, this information is removed from the observed data. It is therefore not possible to use this information to make sense of the heterogeneity in the observed effect size estimates in the data. A large effect size in the data may correspond to a small population effect size because it was inflated by sampling error and vice versa. This explains why heterogeneity often plays a minor role in the interpretation of meta-analyses. Heterogeneity exists, but it cannot be explained.
In contrast, mixture models combined with statistical tools that correct for regression to the mean make it possible to obtain estimates of the population effect size of individual studies. Often these effect size estimates for single studies have large uncertainty (wide confidence intervals), but they can be averaged to identify subgroups of studies with large effect sizes. This makes it possible to explore the heterogeneity in observed studies.
To demonstrate the capabilities and advantages of mixture models over traditional random-effects models, I conducted a simulation study. To isolate the effect of the distributional assumption, the simulation used unbiased data (no publication selection). This removes selection as a confound and allows inclusion of the adaptive-shrinkage mixture model ash (Stephens, 2017), which was developed for genomics and assumes complete, unselected data. The other two models — the weight-function selection model (Vevea & Woods, 2005) and z-curve — model publication bias but reduce to unbiased estimation when no selection is present.
Z-curve is a meta-analytic mixture model (Brunner & Schimmack, 2020; Bartoš & Schimmack, 2022). Version 1 used the noncentrality parameters (ncp) of significant results to estimate the expected replication rate for direct replication studies. Version 2 used the ncp distribution across all studies to estimate the expected discovery rate. Version 3 adds an empirical-Bayes function that estimates population effect sizes by multiplying ncp estimates by their corresponding sampling errors, recovering effects from the latent distribution of noncentrality parameters.
The simulation used fixed sample sizes across studies, which ensures that mixture models fit in the effect-size metric and in the ncp metric produce identical results. Each condition contained k = 1,000 studies; the large k yields small sampling error in the estimates, making systematic biases more visible. The design crossed 4 means and 4 standard deviations of the population effect-size distribution with 4 proportions of true null hypotheses, producing 64 conditions. The population effect sizes were drawn from a t-distribution with 5 degrees of freedom. This distribution is symmetric but has heavier tails than the normal distribution, testing whether models that assume normality are robust to heavier-tailed effect distributions.
These results are not surprising, as the simulations essentially demonstrate the consequences of assuming a distribution for the population effect sizes that does not match the data-generating distribution. The mixture models, which make no parametric assumption about the population distribution, fit the data closely (z-curve RMSE = .018; ash RMSE = .012). The models that assume a normal distribution did not perform as well (weightr RMSE = .106; RMA RMSE = .092).
The present results do not show that mixture models are always unbiased. They highlight an often-overlooked consequence of violating the assumptions of widely used meta-analytic models: violations of the normality assumption can bias effect-size estimates. This bias is avoidable by adopting modern statistical methods that do not require a distributional assumption. Indeed, the present results show that even an R package developed for other purposes — Stephens’s (2017) ashr package — performs better than the models currently in use.
The present simulation assumed unbiased data. In many literatures, however, studies with significant results are far more likely to be published; in psychology, the proportion of significant results often exceeds 90% (Sterling et al., 1995). Such publication bias inflates effect-size estimates, and correcting for it requires modeling the selection process.
This creates a problem that no existing method fully addresses. The mixture models examined here — ash and RMA’s random-effects framework — either ignore selection (ash assumes complete data) or, in the case of standard random-effects models, model neither selection nor a flexible distribution. Weight-function models correct for selection but assume a normal distribution of effects, and therefore fail under heterogeneity. Each method handles at most one of the two problems.
Z-curve occupies the remaining niche: it combines a selection model, which corrects for publication bias, with a flexible mixture model, which captures heterogeneity without a distributional assumption. This is why z-curve outperforms weight-function models when studies are both heterogeneous and affected by publication bias (Schimmack, 2026) — the condition that characterizes many real literatures, and the one in which the other methods each fail on one of the two dimensions.
References:
Bartoš, F., Maier, M., Wagenmakers, E.-J., Doucouliagos, H., & Stanley, T. D. (2023). Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods. Research Synthesis Methods, 14(1), 99–116.
Böhning, D. (2000). Computer-assisted analysis of mixtures and applications: Meta-analysis, disease mapping and others. Chapman & Hall/CRC.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177–188.
Hedges, L. V., & Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21(4), 299–332.
Laird, N. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73(364), 805–811.
Lindsay, B. G. (1983). The geometry of mixture likelihoods: A general theory. The Annals of Statistics, 11(1), 86–94.
Maier, M., Bartoš, F., & Wagenmakers, E.-J. (2023). Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychological Methods, 28(1), 107–122.
Stephens, M. (2017). False discovery rates: A new deal. Biostatistics, 18(2), 275–294.
Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60(3), 419–435.
Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10(4), 428–443.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.
Efron, B. (2010). Large-scale inference: Empirical Bayes methods for estimation, testing, and prediction. Cambridge University Press.
Kim, Y., Carbonetto, P., Stephens, M., & Koenker, R. (2020). A fast algorithm for maximum likelihood estimation of mixture proportions using sequential quadratic programming. Journal of Computational and Graphical Statistics, 29(2), 261–273.
Koenker, R., & Mizera, I. (2014). Convex optimization, shape constraints, compound decisions, and empirical Bayes rules. Journal of the American Statistical Association, 109(506), 674–685.
Laird, N. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73(364), 805–811.
Lindsay, B. G. (1983). The geometry of mixture likelihoods: A general theory. The Annals of Statistics, 11(1), 86–94.
For method folks, the picture tells the full story: z-curve can estimate the true mean of a set of heterogeneous studies better than the weight-function model because the weight function model makes unrealistic assumptions about the distribution of population effect sizes. Added bonus: z-curves estimates are related to actual studies, whereas the estimates of weightr are population estimates that are not connected to the actual studies.
The root cause of the crises in psychology is poor training in scientific thinking and scientific methods. Period! I know because I have been teaching at a top-ranked university in North America for over 25 years now. The most common criticism in student evaluations is that my courses are not psychology courses, but statistics course. The reason: I use numbers when I present research findings. But most students can get a degree in psychology without using numbers. Graduate education does not help because students learn from a mentor, who also never learned to think quantitatively. So, psychology is the worst of both worlds. It is neither qualitative research that pays attention to people’s thoughts, feelings, or actual behaviors, nor is it a quantitative science that use valid quantitative information for the same purpose. It is a pseudo-science that produces meaningless numbers that mainly serve the purpose of claiming scientific support for researchers’ personal beliefs.
The problem that quantitative results in published articles cannot be trusted is now widely recognized and has been called a crisis of confidence a credibility crisis, or the replication crisis. However, the problem also exists at the meta-level when questionable published results are combined into a meta-analysis. Don’t get me wrong. Meta-analysis, like all statistical models, are not wrong. They are only wrong when incompetent researchers use these tools without understand how they work and what assumptions these models make.
Meta-analysis is easy to understand and perform when all data are available. We simply combine summary statistics to reduce sampling error and get a more precise estimate of the population effect size. Instead of running one study with N = 1,000 participants, we combine data from 25 studies with 40 participants. The result is practically the same. However, in psychology, the 25 published study are only a fraction of studies that were conducted and produced a significant result (Sterling et al., 1995). This means the effect sizes in the studies are inflated by publication bias and the same bias leads to an inflated effect size estimate in the meta-analysis. Thus, normal meta-analysis that ignore bias are as useful as a wet tissue paper on a 40°C (104°F) day in the middle of a parking lot at noon.
The solution to this problem is to use fancy statistical models that promise to correct for these biases and reveal the truth hidden in a pile of selected and p-hacked studies. The simple truth is that this goal is as attainable as making gold from base metals. However, as readers also do not understand these models and the problem of using them with uninformative data, the results are now routinely included in meta-analytic articles, if only as a sensitivity analysis that can be dismissed if it shows inconvenient or strange results.
Before I show how silly bias-correction of biased literature is, I need to present an example to show that I am not attacking a strawman model of bad meta-analysis. The example comes from a recent meta-analysis of studies that examined the influence of mortality salience on feelings, attitudes, and behaviors (Chen et al., 2025). The meta-analysis is notable for its attempt to deal with publication bias. The title even mentions publication bias in a clever way “Managing the Terror of Publication Bias.” The authors also shared their data. So, the only problem is that they did not consult with experts to make sense of their findings.
Figure 1 shows a simple histogram of the effect size ESTIMATES – these are estimates in small samples with enormous sampling error, not the actual effect sizes without sampling error.
Figure 1.
The most important observation for this blog post is that there are hardly any effect sizes below zero. To understand why this is important, it is important to understand the meaning of the sign of an effect size. In an original study, the sign has no meaning. For example, the height differences between people with XX and XY chromosomes can be positive or negative depending on the coding of XX as 0 or 1 and XY as 1 or 0, respectively.
For a meta-analysis, however, the sign becomes meaningful. In a competently conducted meta-analysis, the sign reflects the substantive hypothesis of a study. If mortality salience is coded as 1 against a control condition coded as 0, and the theory predicted an increase in a dependent variable, a positive sign implies that the result was consistent with the prediction. If the prediction implies a decrease in the DV, a negative sign is consistent with the theory, and the sign has to be reversed. Thus, if researchers mostly make correct predictions about the direction of an effect, we would expect mostly positive signs. Sampling error can still produce negative means in studies, even if the true effect is in the predicted direction, but how often that happens depends on the strength of the effect.
Now we are in the position to make sense of Figure 1. Only 2% of the effect size estimates. There are two possible explanations for this finding. Either TMT studies mostly produce positive results because most studies have true effects or there is selection bias and results that contradict theoretical predictions are not published (a third option would be coding mistakes, where coders code all results as positive and ignore substantive hypotheses).
What happens when these data are analyzed with bias-correction models? It depends on the model. The PET/PEESE model regresses effect sizes on the sampling error under the assumption that all studies have a common effect size and that larger samples are less biased.
The reanalysis that produced Figure 2 reproduced the published estimate of -.114 standard deviations. Thus, even though there are hardly any negative results in the data, the average study is supposed to have made a prediction in the wrong direction because that is what a negative mean means. An analysis that removed the 10% largest effect sizes, produced a positive estimate of .29 standard deviations. It is interesting that removing strong results increases the average. This shows that the results depend on assumptions about the amount of bias for different effect size estimates. Here the largest effect size estimates come also from the smallest studies (N < 10).
The point estimate of .29 should not be confused with the true effect size in each study. After removing sampling error, there is still considerable variability in the effect size estimates that can be quantified with the standard deviation, assuming a normal distribution. The estimate is tau = .40. This also makes it possible to create a prediction interval – a confidence interval for the hypothetical population effect sizes . To get a 95%CI we roughly multiply tau by 2 and get a range of values around the point estimate from .29 – ,80 to .29 + .80. Thus, any particular TMT study could have an effect size anywhere from -.51 to + 1.09. In terms of Cohen’s classification of effect sizes the effect sizes range from a moderate negative effect size to a strong positive effect size. In other words, the data are not telling us anything that we did not know before we ran the analysis. Terror Management effects may sometimes emerge as predicted, sometimes with surprising opposite effects, and sometimes have no notable effects, and we do not know which manipulation produces which effect.
The problem in the published article and many other meta-analysis is that the heterogeneity in effect sizes after taking random sampling error was ignored. The point estimate is only needed to center the prediction interval. The real information is the wide range of possible population effect sizes that are consistent with the model’s assumptions and the data. Every outcome except large negative effect sizes is possible.
Regression models have many limitations and even the developer of this approach has warned against the use of this model for highly heterogenous data (Stanley, 2017). A model that is more suitable for heterogeneous data is the weight-function selection model (Vevea & Woods, 2005). However, this model requires assumptions about selection bias. Chen et al. fitted a model that assumes different selection bias for significant negative results and non-significant results. Importantly, their model assumed the same amount of selection bias for negative non-significant results and positive significant results. This specification is important because Figure 1 shows that there are few negative effect sizes. A better way to see the problem here is to convert effect sizes and sampling error into z-values (z = effect size / standard error) to distinguish between non-significant (z < 1.96) and significant ones.
Figure 3 shows clearly that there is selection against negative results. Sampling error alone cannot explain the drop in effect size estimates from just above zero to just below zero.
The published article reports an estimated average effect size of .36. The model also estimated that only 26% of non-significant results were included in the meta-analysis. In other words, 74% were missing due to to publication bias. Finally, the model estimated that the population effect sizes had high heterogeneity, tau = .71. This leads to a very wide prediction interval around the point estimate of .36 ranging from .36 – 2*.71 to .36 + 2 * .71, which is -1.01 to 1.73. In other words, the model does not even exclude strong negative effect sizes as possible outcomes.
However, this model is misspecified because it ignores that selection against negative non-significant results is stronger than selection against non-significant positive results. I therefore ran the model again with an additional step at p = .5 (one-sided) that separates positive from negative results.. Consistent with the pattern in Figure 3, the model shows stronger selection against negative results (weight = .01, selection 1-weight = 99%) than for nonsignificant positive results (weight = .30, selection bias 70%). This improved model, however, produced a negative estimated average effect size of -.34. It also further increased the estimate of heterogeneity to tau = .96.
To understand this behavior of the model (I am more of a model analyst than a psycho-analyst), we need to understand the model’s assumption about the distribution of the unobserved population effect sizes. The model assumes an unobserved normal distribution, but the data are a truncated distribution at zero with no meaningful negative values. The model therefore fits the positive range of a normal distribution to the observed positive values. If this distribution is very wide, the normal has a large standard deviation and the model extrapolates it into the negative range. This leads to the wide prediction interval ranging from -1.34 to 2.58. Importantly, the negative range is entirely based on distribution assumption of the model . Changing the distribution assumption would change the results.
So, we have to think about the distribution assumption. When studies are more or less identical, there may be some extra variation in population effect sizes aside from sampling error. This variance can be approximated with a normal distribution (Hedges & Vevea, 1996). But when the set of studies has effect sizes ranging from 0 to 2, this is no longer plausible. If the average effect size is small and heterogeneity is large, a normal distribution implies that many substantive hypotheses have the wrong sign, but that is not really plausible. Many studies may have no real effect or really small ones, but it is harder to argue and to believe that researchers often get the sign of an effect wrong, especially when there is a real effect. Reminding people of their death makes them afraid is a reasonable hypothesis, and it would be surprising if studies show the opposite result.
In short, the weight-function model is not wrong, but applying a model that assumes a normal distribution to highly heterogeneous data is wrong. The model predicts many negative results that do not match any observed results. It could be selection bias, but it could also be a false distribution assumption. What to do?
A reasonable approach to make sense of results from the selection model is to focus on the positive side of the distribution. With normal distributions it is easy to get other statistics like the mean of only positive results (or any other subset of studies). We can therefore ignore studies with false substantive hypotheses and focus on studies where researchers made correct predictions about the sign of an effect (H1 is true).
With a mean of -.34 and tau = .956, we get a conditional mean for studies in which H1 is true of .65 standard deviations (a medium to large effect size) with tau of .52. As the lower bound is zero, we only need the upper bound and get .65 + 1.96 * .52 which is 1.67. This would suggest that many studies have strong effect sizes, which seems to contradict the estimated center of the distribution at -.34. This shows how meaningless these point estimates are when heterogeneity is large.
Unfortunately for terror management researchers the truncated moments are not going to rescue their literature because they are hypothetical. The reason is that we are conditioning on an unknown parameter, namely the condition that the hypothesis was true, but for any particular study we do not know whether H1 is true or not. So, the correct way to formulate this result is “if you can identify a study design in which terror management theory makes the right prediction and you can get a fairly precise estimate of the true effect size, you can expect a moderate effect size estimate.
What the weight-function model does not provide is a bias-corrected estimate of the positive effect size estimates in the dataset. The mean of the full distribution includes negative results that were either removed or never obtained. The truncated moment estimate conditions on the unknown status of the null-hypothesis. One is likely too low and the other is likely to high, but neither is conceptually the estimate we want. The average population effect size positive studies that corrects for the selection of nonsignificant results.
To summarize, state of the art meta-analyses in psychology try to deal with the terror of publication bias, but fail to do so. The main reason is that the statistical models that are available do not match the data. They were designed for meta-analysis of close replications with small variation in true effect sizes. They were not intended to be used for meta-analyses of diverse paradigms with large heterogeneity. Other methods that were developed after the replication crisis like p-curve and p-uniform have the same limitation. They work when heterogeneity is small, but they do not work for meta-analyses of diverse studies that are only loosely related by a common hypothesis.
Z-Curve to the Rescue
The quote “Insanity is trying the same thing and expecting a different result” has been attributed to Einstein. Even if that attribution is false, the insight is right. The problem with meta-analytic models is that they try to estimate a single number. This makes sense when the goal is estimation of a single population effect size, but not when every study has a different population effect size.
When we have a heterogenous literature, we need to face heterogeneity head on, and not hide it in some test that is reported and ignored. There is also heterogeneity around the estimate, p < .05. We need to see how much heterogeneity there. But to do that, we first need a model that can deal with heterogeneity without making unrealistic assumptions about the distribution of population effect sizes.
With a fresh look at the problem, we can look to other research areas that have addressed the problem of heterogeneity in effect sizes and large uncertainty about effect sizes of a specific result. Genomics tests millions of DNA segments (SNPs) and tries to find a few segments that show promising results. The goal here is to find the needles in the hey stack rather than averaging across millions of segments that have no relationship with a phenotype. As selection for the strongest observed effects leads to inflated estimates, models are needed to correct for this inflation. However, these corrected estimates are still tight to actual observed results rather than claims about some unobserved distribution of effect sizes. That makes it possible to identify specific segments in the observed data with promising results.
The same logic can be applied to meta-analysis. The goal is no longer to make claims like “the average population effect size is zero” or “the range of plausible effect sizes ranges from -1 to 1.” the goal is now to say “these studies show convincing evidence with meaningful effect sizes.”
One statistical model that can be used to answer this question is zurve (Brunner & Schimmack, 2020; Bartos & Schimmack, 2022). With a few modifications, z-curve can be used for directional meta-analysis where the sign of an effect matters. Rather than fitting z-curve to absolute z-values that ignore the sign and using folded normal components, z-curve can use truncated z-values and truncated normal components. When the model is fitted to only significant results, the difference is minor. More importantly, z-curve estimates of power can also be used to compute bias-corrected effect sizes (Efron, 2005). The reason is that power is a function of effect size and sampling error, so we can use the inverse normal to convert power into a corrected z-value and then multiply it with the sampling error to get a bias corrected effect size. The main challenge is to estimate the sampling error for unobserved non-significant results because their sample sizes are unknown. A simple approach is to use the sampling errors of the just significant results as an approximation. A weighted average of these estimates is the estimate of the true average effect size for the population of studies with positive results before selection for significance. Negative results that are observed are discarded.
Figure 4 shows the results of a simulation study in which the true average power before selection is known. The simulation modeled a beta distribution and graded selection bias. This is important because the weight-function model does well when its assumptions are met. The problem is that the assumptions are are untestable and often questionable. For example, we can simulate a literature with a mean of zero and tau of .4, but this simulation implies that a theories predictions are no better than a coin flip. Once researchers make better predictions, the normal assumption no longer holds.
While z-curve estimates are not perfect, they are conceptually meaningful and closer to the truth than either of the weight-function model’s estimates. We can now apply the model to the TMT data, excluding the few (2%) negative estimates.
The z-curve shows clear evidence of selection bias (the red dotted line is above the light purple bars of the nonsignificant results. However, the EDR estimate of 35% suggests that studies have on average 33% power to produce a significant result. Moreover, an EDR of 33% implies that no more than 10% of the significant results can be false positive results. Even the lower limit of the EDR confidence interval, 20%, allows for only 20% false positive results. This would suggest that many studies, especially significant ones, produced evidence for a true hypotheses with an effect size in the right direction. We can now also quantify the typical effect size. The overall effect size estimate is .63, 95%CI [.46 to .68]. Moreover, we can quantify the average for different ranges of z-values. The average increases from .40 for z-values between 0 and 0.5 to effect sizes greater than 1 for z-values greater than 4.
This finding is surprising, to say the least, because typical effect sizes in psychology are around d = .4 and rarely greater than 1. Before TMT researchers start celebrating, we have to reconcile these findings with the z-curve analysis published in the TMT article.
The z-curve looks notably different in that it does not have a long tail of high z-values. As a result, the EDR estimate is much lower, .08, and the 95% confidence interval includes alpha, 5% to 17%. This implies that there is no t enough evidence to reject the null-hypothesis that all significant results were obtained without a real effect, average effect size: zero, even for z-values greater than 4. So what is it? Is the average effect size close to zero or greater than 1?
To understand the different results, it is important to know that the published z-curve used a different coding of studies than the effect size meta-analysis. I fitted z-curve to these z-values and computed effect size and sampling error estimates from the z-values and degrees of freedom, assuming between-subject designs with equal cell sizes.
The plot is scaled to show the full distribution in the range of non-significant results. The model estimates reproduce the published results. The EDR is 7%, 95%CI = [5%, 17%]. The plot also shows local power for z-values from 0 to 3 stays low. Studies with z-values greater than 4 have acceptable local power but contrary to the previous z-curve, there are hardly any studies. This published z-curve produces dramatically different average effect size estimate, .12 95%CI = .02 to .19. The results also imply much lower heterogeneity because there are hardly any studies with strong evidence (z > 4) and large effect sizes.
Applying the weight-function selection model produces roughly the same results. The average effect size estimate is d = .17, and heterogeneity is small, tau = .17. Now the PET regression result also agrees, intercept = .05, tau = .19.
In conclusion, careful examination of this meta-analysis shows several problems. First, the data were coded inconsistently and different models were given different data. As it turns out, the effect size coding was wrong because F-values were coded as t-values, which dramatically inflates effect size estimates. Second, inconsistent results focused on the point estimate of models, but the point estimate is irrelevant when data are highly heterogenous (due to coding mistakes). Properly interpreted, all models suggest high heterogeneity that allows for large effect sizes among positive results. However, when the data are properly coded, the results show weak evidence that any study produced real effects, a high false positive risk, a small average effect size and small heterogeneity. These results change the final conclusion in the article.
“Given the conflicting findings that emerged across tools and the inherent trade-offs associated with each tool, we caution researchers against drawing firm conclusions about the evidential value of literature through any single analytic tool.”
Correction: The results are consistent and show that most studies provide no evidence for an effect because most effect sizes are small and studies had low power to detect or estimate these effects.
“PET-PEESE can underestimate the effect size when there is publication bias and when p-hacking is present (Carter et al., 2019), which are two conditions likely affecting the literature.”
Here bad research practices are used as an excuse to dismiss the most negative result without mentioning the real problem There is no “effect size.: there is only an average effect size and regression models still allow estimation of heterogeneity that was large in the data the authors used. Even a negative average can be consistent with many true positive effects when heterogeneity is large.
Z-curve can be a powerful tool for inferring the overall composite z-score distribution of a heterogeneous literature. However, unlike the other analyses included in this study, z-curve has not been as thoroughly evaluated by independent researchersso the statistical properties for its power estimates remain under explored. Furthermore, its power estimates are subject to the usual theoretical objections to estimating power from a fixed sample of data (for a recent commentary, see Pek et al., 2022).
This statement ignores that z-curve has been thoroughly evaluated by extensive simulations studies that have been reproduced by the editorial team during an open peer review process. The same cannot be said about the other methods that have not been vetted as rigorously or failed to do well in some conditions (Carter et al., 2019). The reference to Pek is also misleading which has been addressed in several rebuttals to this unfounded claim (Schimmack & Soto, 2026; Soto & Schimmack,2026) with no rejoinder by Pek.
The higher conditional power estimate therefore suggests some evidential value in published studies that yielded significant findings.
The authors are referring to the ERR estimate of 22% [16% , 37%]. Suddenly Pek’s criticism of z-curve is no longer relevant. More importantly, this finding implies that an exact replication of a study with a significant result has a 22% chance of a successful replication outcome. This is abysmal and one of the lowest ever found, not a cause for optimism. Surely reminders of mortality will sometimes have an effect on something, but a research program that uses different designs with an average power of 22% will not be able to identify when a manipulation works or when it is just a chance finding. In fact, the upper limit of the DR estimate is 100%. Thus, these weak studies fail to reject the hypothesis that all studies are pure noise.
“The selection models provide evidence for a small effect consistent with the MS hypothesis… We suggest that the average effect of the literature may be within the range estimated by the selection models and WAAP-WLS (i.e., r is around .18), although this average may have resulted from a mix of effects, many of which are higher than .18, and many of which are lower than .18.
The average estimate is too high once we correct for the coding mistakes. The real effect size is half of this (d = r / 2), and heterogeneity is small. This is the most conesquences conclusoin. Rather than having evidence of a wide range of positive effect sizes, we have evidence that most effect sizes are small and too small to study with the typical sample sizes of this literature.
We encourage future preregistered replications of the MS hypothesis to use smaller es timates of effect size (i.e., r = .18).
This inflated effect size estimate will only lead to a replication failure. Given the weak evidence in this literature, it may be better to start a new credible research program about coping with awareness of one’s own mortality than to invest more resources into this failed paradigm with questionable manipulations and dependent variables.
Though on their face the liberal and conservative interpretations feel contradictory, some observations are uncontroversial. The first observation is that the TMT literature consists of highly heterogenous.
Even this conclusion turns out to be false when the proper data are analyzed. Heterogeneity was caused by coding mistakes and practically vanishes when the correct data coded by the authors were analyzed. The authors did not notice that their data were inconsistent, even though a simple comparison of the z-values would have shown the discrepancy. It is natural for humans to make errors, but errors also reveal something about the person who committed the error. In this case, it reveals a lack of understanding of the methods, their assumptions, and why they may produce inconsistent results. Here inconsistency was attributed to properties of the models when the real source were inconsistent data. In the future, meta-analysts should not just report inconsistencies, but also try to explain them. That requires understanding of the tools that they use.
For our entire universe of studies, heterogeneity is estimated at τ = .72 under the selection models, which means that for the estimate of g = 0.36 (r = .18) for the entire literature from the selection models, 95% of the effects underlying studies of MS hypothesis, assuming a normal distribution of g, fall between g = −1.05 and 1.77 (or r = −.47 and .66); an extremely wide range of possible effect sizes arising from differences in study design.
The problem with this wide range of population effect sizes is the assumption of a normal distribution. Even if no negative results are observed, the assumption leads to the conclusion that negative effects were obtained but suppressed. But researchers are flexible and it is more likely that they would change the prediction in the direction of a significant result (Kerr, 1998). Thus, it is highly likely that the predicted negative results are phantom studies that do not exist. These predicted effect sizes surely do not correspond to the positive estimates in the dataset.
With these observations in mind, we conclude that there must be some nonzero underlying effects in the studies we examined.
That sounds more reassuring than it is. We have over 800 results and some of these are not false positives. Great, now what? We do not know which of these results are true or false positives. So, we haven’t really learned anything about mortality awareness from this meta-analysis. Fortunately, the analysis of the data without the coding mistake is more conclusive. Terror management research is an example of a pathological science. Researchers conduct studies but never learn form their data because they find a way to keep their theory alive. A proper analysis shows that we can put this literature to rest. That is ok. The history of science is filled with failures. It is also filled with examples where researchers are unable to learn from their errors. However, science moves on and experimental social psychology with little priming manipulations will be a little footnote in the history books.
P.S. And z-curve works and can now also estimate effect sizes.
To make sense of empirical data, researchers need statistical models, and the choice of model can influence conclusions. It is therefore important to be aware of the assumptions a model makes and, ideally, to test whether those assumptions hold in a particular dataset. This simple truth applies even to the choice of how to describe the average of a variable. As most students learn in a first statistics course, the mean and the median are interchangeable for symmetric distributions but not for skewed ones. In that example, the assumption is easy to check: test the symmetry of the data and pick the better summary.
With more complex models, testing assumptions is harder, and some assumptions cannot be tested at all. In these contexts it is common to leave the influence of assumptions unexamined. It is also awkward to phrase every conclusion as a conditional statement — “assuming the model’s assumptions hold, …” In psychology, correlational researchers are routinely required to flag that any causal claim is conditional on assumptions, or else they are told not to draw conclusions at all. Model-based results, by contrast, are often presented without a careful discussion of the model’s assumptions.
An increasingly popular model for meta-analyses is the step-function selection model (Vevea & Woods, 2005). Like other selection models, it assumes that nonsignificant results often remain unpublished while significant results are published. Other biases may also influence which significant results appear, but most selection models make the simplifying assumption that selection for significance is the key driver of publication bias.
Where selection models differ is in their treatment of nonsignificant results. Models like p-curve and z-curve fit only the significant results and do not condition on the observed nonsignificant ones; they avoid making any assumption about how selection shaped the nonsignificant results. Step-function models instead use the nonsignificant results directly. This can yield more information, but only at the cost of an additional assumption: that the selection process takes a specific form. That extra assumption is the subject of this post — and, as with symmetry and the mean, it is one we can test.
To use non-significant results, step-function models make two assumptions that have to be true to correct for selection bias.
Non-significant and significant effect size estimates come from the same normal distribution of population effect sizes.
Selection bias within a step (a range of p-values) is independent of the p-value.
Taken together, these two assumptions make predictions about the observed p-values within a step. This prediction is easier to test when the p-values are converted into z-values. Essentially, selection bias should influence the frequency of results in a step, but not the shape that one would see without selection bias. This prediction has never been tested, but it is possible to do so using a z-curve plot and z-curve analyses of the data.
Here I use actual data from a meta-analysis of Terror Management Studies that used the step-function model (Chen et al., 2025).
A simple histogram of the z-values shows that there are hardly any negative results (Figure 1). However, Chen et al. specified a model that assumes equal selection bias for all z-values between -1.96 and +1.96. Accordingly, the data imply that there really were more non-significant results with a positive effect than a negative effect. This alone implies that the mean of the population effect sizes is positive. Indeed, the model estimated a mean standardized effect size of .347 standard deviations.
An alternative possibility is that there is a stronger selection bias against negative non-significant results than for positive non-significant results. This assumption can be tested by splitting the step at z = 0 (p = .5, one-tailed) and estimating selection bias separately for positive and negative results. This model produces a dramatically different estimate of the average effect size, suggesting that studies, on average, more often produced a negative result (contrary to predictions) than a positive result, standardized mean difference = -.336.
The change in the sign of the estimate shows how sensitive estimates are to model assumptions. I used z-curve (Brunner & Schimmack, 2020; Bartos & Schimmack, 2022) to test the assumption that the distribution of the positive non-significant results is consistent with the assumptions of the step-function model.
Z-curve fits a finite-mixture model to the distribution of the significant z-values and predicts the distribution of non-significant results from the weights of the mixture components (Figure 2). The z-curve plot shows clear evidence of selection bias. That is there are more observed significant results (66%) than the model predicts (31%). However, the important question is whether the distribution of the non-significant positive z-values matches the predicted distribution shape (the red dotted line in Figure 1).
Visual inspection suggests that this is not the case. There are more observed non-significant results close to the significance criterion than close to zero. In contrast, the predicted pattern shows a flat and then decreasing shape from 0 to 1.65 (values between 1.65 and 1.96 are inconclusive because they are often reported as marginally significant results).
To complement visual inspection with a statistical test, z-curve (version 3.85) compares the distributions while holding the overall density constant (i.e., removing selection bias). A negative difference indicates that there are more z-values close to zero than predicted. A positive difference indicates that there are more z-values close to the significance criterion. A bootstrap confidence interval is used to test for statistical significance.
The median difference was z = .15, 95%CI [ .08, .38]. The mean difference was z = .10, 95%CI = [.05 to .26]. This finding is consistent with the observation that observed z-values close to zero are relatively rare. In short, the observed distribution of the non-significant results is inconsistent with at least one of the assumptions of the step-function model.
The test does not reveal which assumption is violated, and in these data both may be. The depletion of nonsignificant results near zero suggests that selection bias varies within the step, contrary to the assumption that selection is constant within a p-value range. Separately, the near-absence of negative results is hard to reconcile with the model’s estimate of large heterogeneity (tau = .96), which produces a prediction interval for population effect sizes from d = −2.20 to 1.54. A lower bound of −2.20 implies that a substantial share of studies have true effects opposite to the prediction — that mortality primes often produce the reverse of the expected effect. That is implausible; more likely, the negative tail is an artifact of assuming a normal distribution. The shape test cannot identify the true distribution of population effect sizes, but it forces meta-analysts to confront the distribution assumption rather than leave it implicit.
To demonstrate that the shape test works, I used the model weights of the actual data to simulate data without publication bias. I then removed 70% of the non-significant results without changing the distribution of the non-significant results (Figure 2).
Visual inspection shows that the distribution of the non-significant results is more similar to the predicted distribution. The shape test results are consistent with this observation. The median difference is -.05, 95%CI [-.10, .18]. The mean difference is -.04, 95%CI [-.07, .12]. Both confidence intervals include zero.
In sum, step-function models make assumptions about the distribution of population effect sizes and the selection of non-significant results to use observed non-significant results in the estimation of the amount of bias and the average and standard deviation of the population effect sizes. The advantage is that more data reduce random sampling error. The disadvantage is that violations of assumptions can introduce systematic biases. This disadvantage has been neglected in applications of step-function selection models. The ability to test the assumptions has several benefits. First, it draws more attention to the fact that assumptions influence model estimates. Second, shape tests help readers to evaluate the plausibility of the model and to question estimates that are based on data that are inconsistent with assumptions. Third, assumption tests can resolve inconsistencies between estimates obtained with different models. Results based on models that make fewer assumptions or pass assumption tests are more credible than those based on models that do not meet assumptions.
In conclusion, meta-analysts have many tools to analyze their data that often produce inconsistent results. To make sense of these inconsistencies, it is important to understand why they produce inconsistent results. Violated assumptions are one plausible reason and undermine the plausibility of results of these models.
Meta-psychology was born from a simple observation: the way psychologists used significance testing created a distorted literature. Researchers treated p < .05 as a license to publish and p > .05 as a reason to abandon a finding. Journals rewarded significant results, reviewers demanded them, and authors learned to find them. The result was predictable: literatures stuffed with too many significant results, exaggerated effect sizes, and too few honest failures (Sterling, 1959; Sterling et al., 1995).
This critique is now familiar. Null-hypothesis significance testing, reduced to a dichotomous decision rule, encourages bad scientific behavior. It turns evidence into a yes-or-no ritual, treats p = .049 as a discovery and p = .051 as a non-event, and rewards selective reporting. Meta-psychologists have made this point repeatedly, and largely correctly.
But there is an irony. Having criticized psychologists for using a dichotomous significance test to decide which original findings count, meta-psychologists often reach for the same logic to decide whether a literature is biased.
The original sin was this: p < .05 means the effect is real.
The meta-analytic version becomes this: p < .05 means publication bias is present.
The form of the reasoning has not changed. Only the target has moved up one level.
Why this fails is clearest if we ask what a significance test can ever legitimately buy us. The most charitable defense of significance testing is that a significant result may carry information about the sign of an effect: it can tell us which direction is more plausible, even when it says little about magnitude (Jones & Tukey, 2000). That defense collapses for publication bias, because the sign is known before we collect a single study. Selective reporting favors significant results; it does not run the other way. A test whose only defensible output is a direction we already know contributes little.
What it contributes instead is a verdict that is uninformative in both directions. A significant bias test conflates magnitude with detectability: in a large literature, a trivial and harmless amount of selection can reject the null. A nonsignificant bias test conflates small bias with low power: in a small literature, severe selection can easily fail to reach significance. Either way, the binary outcome tells us little about the quantity we care about. “There is publication bias, p < .05″ is, to borrow Cohen’s (1994) famous example, about as useful as “the earth is round, p < .05.” And “There was no evidence of publication bias, p > .05” is akin to “The earth is flat, p > .05.”
The deeper irony is that meta-psychologists have relocated the mistake they diagnosed. Original researchers treated significance as a discovery machine. Bias researchers sometimes treat significance as a bias-detection machine (Siegel et al., 2021). The error is identical: a difficult inferential problem is compressed into a binary decision.
Some have pushed the argument further, claiming that tests for publication bias are useless (e.g., Simonsohn, 2014). But the folly of nil-hypothesis testing, which incidentally undermines p-curve as much as many other significance-based methods, is not a reason to ignore publication bias. We do not abandon original research because the significance ritual is empty (Cohen, 1994). We replace the ritual with something more informative.
The reform for original research was to report effect-size estimates with confidence intervals that express uncertainty. The reform for bias detection should be the same. The goal is not to decide whether bias exists, but to estimate how much is present, how uncertain that estimate is, and whether the amount of bias consistent with the data changes the substantive conclusion.
Some publication-bias methods already estimate quantities of this kind, or carry the information needed to. Yet in practice that information is discarded, and the result is reduced to whether a test was significant or a method “detected bias” (Siegel et al., 2021). And no common metric for the amount of bias has been widely adopted.
The most natural metric is the excess of significant results. If a literature reports significant findings 80% of the time but the true probability of producing significant results is between 20% and 40%, we have clear evidence of substantial bias.
This is why the amount matters more than its presence. Bias can be easy to detect yet too small to change any conclusion, or large enough to overturn a conclusion yet impossible to detect in a small set of studies, where these tests have the least power (Renkewitz & Keiner, 2019). A binary test cannot tell these cases apart; an estimate with an interval can.
In short, meta-analysis needs the same methodological reform that original research needed. It is time to abandon the nil-hypothesis ritual and replace it with estimation: estimate the amount of publication bias, quantify the uncertainty with confidence intervals, and evaluate whether conclusions remain credible after adjusting for the plausible levels of selection.
Fortunately, unlike unpublished primary studies hidden in file drawers, the data behind published meta-analyses are often available or recoverable. That makes it possible to reexamine decades of meta-analytic conclusions and ask the question that matters: not whether publication bias can be detected, but whether the amount of bias compatible with the data changes what we should believe.
The past decade has not been kind to experimental social psychology. Study after study failed to replicate and entire literatures have turned out to be built on nothing (a.k.a. statistical noise mining).
“Another day, another idol falls. This one has been teetering for years, so the collapse didn’t come as a shock. But that doesn’t make it any less painful.” (Michael Inzlicht).
It all started with a leading journal publishing an article with the crazy claim that people can foresee the future and practicing after a test can improve exam scores (Bem, 2011). This claim was quickly revealed to be false (and possibly a hoax, Gelman) after a big replication study failed to show the same results (Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P., 2012).
In a media interview Bem explained that his experiments were never meant to be taken seriously. (Daniel Engber, 2017, Slate).
“If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?’
While the past decade has not been good for experimental social psychologists, it has produced a new group of psychologists to examine the causes of the replication crisis in experimental social psychology. As they look at the practices of research psychologists, they are meta-psychologists, psychologists who study other psychologists.
One of them is Blake McShane, who did his dissertation on statistical models to analyze time-series data (McShane, 2010). Given his background in statistics, managerial science, applied economics, and marketing, it is fair to say that he entered this field without first-hand experience of research practices that produced the replication crisis. He also does not cite seminal papers that foreshadowed the crisis by Cohen (1962, 1990, 1994). Instead, his main approach to examining meta-psychological questions appears to rely on his expertise in conducting simulation studies (McShane & Böckenholt, 2014, McShane, Böckenholt, & Hansen, 2016, 2020).
The problem with these simulation studies is that they repeat the same problems that plagued experimental social psychology at the meta-level. Just like Bem’s studies are not empirical tests, but rhetorical devices, McShane’s simulations are rhetorical devices to illustrate a point that does not require simulation evidence, namely.
[models] perform reasonably well in the setting for which they were designed, …[but] they are sensitive to deviations from their model assumptions.
In the 2016 article, the simulations violated assumptions of models that assume homogeneity and they failed. However, the simulations met the assumptions of another model and (no surprise) it worked well. However, McShane did not cite an earlier study that showed the model also has problems when its assumptions are not met (Hedges & Vevea, 1996).
Later simulation studies further confirmed that McShane’s preferred model does not work so well under realistic conditions (Carter et al., 2019), a finding not cited by McShane et al. in 2020. Pressed on this point that his simulations favored his preferred model, he might reply
“If you looked at all my past simulations, they were always rhetorical devices. I created conditions to show how things work when assumptions are met. I used simulations as a point of persuasion, and I never really worried about, ‘Does this apply to real data’ ”
In conclusion, a simulation that shows a model works when its assumptions are true and does not work when its assumptions are false is merely a demonstration, not an evaluation of a model under realistic conditions.
Inferential statistics in psychology lacks a unified practice. Debates about statistical inference tend to organize around a 2 × 2 structure: one dimension distinguishes frequentist from Bayesian approaches, the other distinguishes hypothesis testing from effect-size estimation. This produces four familiar schools: frequentist hypothesis testing (t tests, ANOVAs, p < .05), Bayesian hypothesis testing (Bayes factors, as advocated by Wagenmakers and Rouder), frequentist effect-size estimation (the New Statistics, confidence intervals, meta-analysis), and Bayesian effect-size estimation (posterior distributions, ROPE). The categories are not mutually exclusive in statistical theory, but as advocacy traditions in psychology they are real and consequential.
Hypothesis testing
Effect-size estimation
Frequentist
NHST / Fisher-Neyman-Pearson hybrid
New Statistics / estimation statistics
Bayesian
Bayes factors / Bayesian model comparison
Bayesian estimation / posterior intervals / ROPE
Each school has genuine advantages. Frequentist hypothesis testing provides a transparent decision procedure. Bayes factors can quantify evidence for a null hypothesis, which became attractive when failed replications made it plausible that some effects were exactly zero. Frequentist estimation, following Cohen’s critique that the nil hypothesis is almost never literally true, redirects attention from binary decisions to effect magnitudes. Bayesian estimation adds prior information, which can improve estimates in small samples if the prior is well-calibrated.
The schools also have genuine conflicts — about the meaning of probability, the role of prior information, and whether hypothesis testing is a coherent goal at all. These conflicts have generated decades of methodological debate and a substantial literature of mutual criticism.
What the four schools share, however, is more important than what divides them. All four presuppose that the data contain enough information to support the inferences being drawn. A Bayes factor, a p-value, a confidence interval, and a posterior distribution are all answers to the question of what the data say. None of them first asks whether the data are precise enough to say anything useful. That prior question — how much sampling error is in this study? — is the topic of this article.
How Noisy Is the Study?
Inferential statistics generalizes from observations to broader conclusions. Larger samples reduce sampling error and make inductive inference more reliable. This is well known. What is less appreciated is that sampling error is typically reported in raw units, which makes it difficult to judge whether a study is precise enough to support its conclusions.
The solution is to standardize sampling error the same way Cohen standardized effect sizes. For mean differences between two groups, Cohen’s d divides the raw mean difference by the pooled standard deviation, yielding a unit-free effect size. We can apply the same logic to the standard error. The result — the standard error of the standardized mean difference — can be called the standardized standard error (SSE):
SSE = 2 / √N
This single number captures how noisy a study is, on the same scale as d itself. For hypothesis testing, the ratio d/SSE approximates a z-score; an absolute value above 2 rejects the nil hypothesis at α = .05. For confidence intervals, the approximate 95% CI is simply d ± 2·SSE, giving a total width of 4 SSEs. Both testing and estimation, in other words, are direct functions of SSE.
The practical implications are immediate. Common sample sizes in psychology produce large SSEs:
N (total)
n per group
SSE
40
20
.32
100
50
.20
200
100
.14
1000
500
.06
A study with SSE = .32 produces a 95% confidence interval roughly 1.28 d-units wide. If the true effect is small (d = .2), the interval runs from approximately −.44 to .84 — spanning small negative to large positive effects. The data cannot determine even the sign of the effect, let alone its magnitude.
No statistical method removes this limitation. Bayesian estimation can incorporate prior information, but unless that prior comes from genuinely trustworthy external evidence, it cannot substitute for data that are not there. A weakly informative prior applied to a study with SSE = .32 will produce a posterior that is dominated by the prior, not the data — which is not estimation, it is prior retrieval. The study is simply not informative about small effects.
What researchers typically do in this situation is report the point estimate without its SSE, or omit confidence intervals when results are non-significant. This conceals rather than communicates the study’s imprecision. A d = .4 reported without its SSE tells readers nothing about the true population effect size.
Fools Gold: Effect Size Estimation in Small Samples
Small samples are occasionally defensible. If an effect is expected to be large and resources are limited, a modest goal — demonstrating that the effect is positive — may be worth pursuing. But this is a directional claim, not a magnitude claim, and the distinction matters.
Consider a study with N = 40 participants (n = 20 per group), where SSE = .32, and suppose the true effect is Cohen’s large effect, d = .8. The signal-to-noise ratio is .8 / .32 = 2.5, giving approximately 70% power at α = .05 — slightly below Cohen’s recommended 80%, but perhaps acceptable under resource constraints.
Now suppose the observed estimate is d = .8. The approximate 95% confidence interval is d = .8 ± 2(.32), or d = .16 to d = 1.44. The interval excludes zero, so the study supports a positive direction. But it simultaneously includes a small effect (d = .2), a medium effect (d = .5), a large effect (d = .8), and an extremely large effect (d > 1.0). The data say almost nothing about magnitude.
The most misleading way to report this result is to write d = .8, p < .05 — as if the point estimate were a precise measurement. It is not. It is one noisy draw from a distribution with SSE = .32. A large library of statistical methods exists for correcting, shrinking, or reweighting such estimates. These methods can help when they incorporate trustworthy external information. But they cannot manufacture precision that is not in the data. A posterior distribution is not narrower than a likelihood unless the prior carries real information — and a generic weakly-informative prior does not.
A statistically significant result from a small study may support a directional claim. It does not support a magnitude claim. Reporting d = .8 as if it were a reliable effect-size estimate, rather than a noisy estimate compatible with effects ranging from small to very large, is confusing fool’s gold for real gold.
Estimation and Hypothesis Testing with Confidence Intervals
A point estimate has little standalone value if it is compatible with qualitatively different population effect sizes. An estimate of d = .5 is uninformative if the true effect could plausibly be small, medium, or large. The confidence interval makes this uncertainty explicit — but it does more than that.
Confidence intervals are widely understood as a test of the nil hypothesis: if the interval excludes zero, the effect is statistically significant. This is the least interesting thing a confidence interval does. An interval also rejects every parameter value outside it. A confidence interval ranging from d = .2 to d = .6 rejects not only d = 0, but also d = −.3, d = −.5, and d = .8. It tells us the effect is neither absent nor large. As intervals become narrower, they reject more values and carry more information about magnitude, not just direction.
This reveals that estimation and hypothesis testing are not competing approaches — they are two descriptions of the same inferential act. The apparent conflict arises because most discussions treat “hypothesis testing” as synonymous with nil-hypothesis testing. Rejecting the nil hypothesis answers only the directional question: is the effect positive or negative? Rejecting a broader range of values on both sides of a narrow interval answers the magnitude question: how big is the effect, approximately?
The practical implication is direct. Replacing p-values with confidence intervals does not by itself improve psychological science. A wide confidence interval is more honest than a p-value, but it is not more informative. It displays uncertainty rather than hiding it — which is progress — but it does not resolve the uncertainty. Resolution requires narrow intervals, and narrow intervals require small SSE. The reform agenda that simply substitutes confidence intervals for p-values has stopped one step short of the real argument: what psychologists need are studies with SSE small enough that the interval, wherever it lands, actually constrains the answer.
The same logic applies in reverse. A confidence interval that ranges from d = −.05 to d = .05 includes zero — so the traditional nil-hypothesis test is non-significant, and the result is typically filed away as uninformative. But this interval rejects d = .2, d = .5, d = −.2, and d = −.5. It provides strong evidence that the effect, whatever its sign, is too small to matter. Non-significance from a precise study is not a failure to find an answer. It is an answer: the effect is negligible. The asymmetry that treats only significant results as informative is a consequence of fixating on the nil hypothesis rather than reading the full interval.
Interpreting Standardized Sampling Error
Cohen’s lasting contribution was not just the standardized effect size but the norms that made it interpretable: .2 is small, .5 is moderate, .8 is large. These benchmarks gave psychologists a shared vocabulary for thinking about effect magnitudes. Standardized effect-size estimates are now routinely reported in psychology articles.
No equivalent norms exist for standardized sampling error. Psychologists report d and r, but rarely the SSE that determines how much those estimates can be trusted. This is the gap Cohen’s framework left open. Just as he provided benchmarks for effect sizes, we can provide benchmarks for sampling error:
SSE = .10 — small. The estimate is precise enough to support magnitude claims. SSE = .20 — moderate. Directional claims are reliable; magnitude claims require caution. SSE = .30 — large. Only directional claims about large effects are supportable.
In a two-group between-subjects design, these values correspond to total sample sizes of approximately N = 400, N = 100, and N = 44, respectively. Studies with SSE above .30 should generally be treated as exploratory. Statistically significant results from such studies are likely to produce inflated effect-size estimates — a direct consequence of the winner’s curse, where only the noisiest overestimates clear the significance threshold.
The advantage of SSE over raw sample size as a reporting standard is that sampling error depends on more than N. Repeated-measures designs, within-person contrasts, reliable outcome measures, and strong covariates all reduce SSE independently of sample size. A study with 30 participants and 50 repeated reaction-time trials per condition may have smaller SSE than a between-subjects study with N = 200. Sample-size rules of thumb cannot capture this. SSE can. By asking “how noisy is this estimate?” rather than “how many participants were there?”, researchers focus on the quantity that actually governs what inferences the data can support.
There is also an upper bound on when SSE becomes a practical concern. For very large studies — N above roughly 1,000, where SSE falls below .06 — the confidence interval remains formally correct but becomes substantively less important. The point estimate is already a close approximation of the population value, and the margin of error is small enough to be reported as a footnote rather than foregrounded in the interpretation.
Large public opinion surveys work this way: the ±2 percentage point margin appears in fine print because it rarely changes the conclusion. At this scale, studies are genuinely estimating population parameters rather than noisily gesturing at them. The SSE framework matters most in the range where psychology actually operates — studies with N between 40 and 500, where sampling error is large enough to be consequential but small enough to be routinely ignored.
Once more this suggestion prevents “sample size bragging” and false claims that a study with N = 100,000 participants is superior to a study with N = 1,000 participants. Once sample sizes are over 1,000 other characteristics of studies are much more important than sample size.
Conclusion
The debates reviewed at the outset — frequentist versus Bayesian, testing versus estimation — are genuine. But they share a prior question that none of them fully answers: how much information does this study contain? That question has a direct, interpretable answer: SSE.
SSE answers the first question every researcher should ask about their data. The confidence interval answers the second. Together, they reduce the essential inferential task to two numbers and one formula:
d ± 2 × SSE
If the interval falls entirely above zero, the data support a positive effect. If it falls entirely below zero, the data support a negative effect. If the interval is narrow enough to exclude effects too small or too large to matter, the data constrain the answer usefully. If the interval is wide, the data are too noisy — and no statistical method, frequentist or Bayesian, changes that.
The word estimate matters throughout. A d-value is not the effect size. It is an effect-size estimate, and its standalone value depends entirely on the SSE around it. A study that produces d = .8 with SSE = .32 has not demonstrated a large effect. It has demonstrated that a large effect is one of many values consistent with the data.
Real studies are often more complex, and SSE can be harder to compute in multilevel or multivariate designs. But the principle is simple and general: interpretation requires an effect-size estimate and its sampling error. Without both, researchers are not interpreting evidence.
The chapter offers interesting insights into the history of Project Implicit by two insiders who worked for Project Implicit. This blog post provides comments on this history from the perspective of an outsider.
1. Big Sample Envy
“Nosek wanted to use the IAT in his research but was only allotted fifteen participant hours through the Yale participant pool” (p. 98).
In most sciences, it is a blessing to be at a rich ivy league university with expensive equipment. Psychology is different because it relied mostly on undergraduate students as participants and classes at fancy ivy universities are small. This gave large state universities like Ohio State University or the University of Illinois at Urbana-Champaign. One might think, rich universities could just pay participants, but that did not appear to be the case. Thus, psychologists at the top universities often published studies with very small samples (Bargh et al., 1996), which led to the replication crisis in the 2010s (Doyen et al., 2012; Kahneman, 2012, 2017).
Project Implicit was born out of the desire to collect data with large samples.
“In the first version of the website, I set up the application to compute the scores within the app and just send a single line of data to the database– e.g., block means, errors. I could watch the file grow live with each person completing a test and their result being added to the database. It was truly mesmerizing. Watching a new line come in every few seconds compared to how laborious data collection had been before. It was some thing of a conversion experience to going all-in on on-line data collection.” (Brian Nosek, quoted in Ratliff & Smith, p. 98).
For an outsider, the statement is a clear admission that the primary purpose of Project Implicit was research and the use of online administration to get data from many people.
Ratliff and Smith further mention that the National Institute of Mental Health awarded a research grant ($2.5 million) to “further develop the virtual laboratory on the Internet” (p. 98).
False Feedback and Deception
The article also mentions the preconditions for research conducted with Project Implicit. ( (1) studies can be no longer than fifteen minutes (around ten minutes is the goal), (2) study text should be no higher than an eighth-grade reading level (3) studies may not include deception (4) studies must include some kind of measure about which participants receive feedback (5) an appropriate debriefing that fulfills the educational mission of the organization must be offered.
Several of these points are noteworthy from an outsider’s perspective. The short time frame makes it impossible to study causes or consequences of implicit biases experimentally. Even correlational studies that relate IAT scores to other measures may take longer. Thus, most studies are limited to the IAT scores themselves or correlations with demographic variables. This limits the usefulness of the virtual laboratory to study actual causes and consequences of implicit biases in real life. Not surprisingly, millions of people have completed an IAT, but sample sizes with actual measures of behavior are much smaller and often unable to reveal meaningful relationships (Kurdi et al., 2019).
The absence of deception and the requirement to provide feedback about IAT performance create a tension that is rarely acknowledged. One type of study in psychology deliberately gives people false feedback about a desirable trait. These studies use deception and require extensive debriefing to ensure that participants are not harmed by the false information. Project Implicit does not give blatantly false feedback, but many people will receive false feedback if a test has low validity. For example, an IQ test that correlates r = .6 with true intelligence (whatever that is) will give 20% of participants false feedback that they are below average (IQ below 100) if their true score is above average. IAT scores are much less valid than intelligence tests and even more people get false feedback. An ethical debriefing would require warning people that one possible explanation for a surprising result is measurement error, however Project Implicit has failed to provide this information. This resistance to debriefing participants properly about the low validity of IAT scores contradicts the claim that IAT research on Project Implicit should avoid deception and properly debrief participants.
The lack of proper debriefing can be explained by the insiders’ belief in implicit biases and the ability of IATs to measure them.
“When we started graduate school in 2003, few people outside of the field of social psychology were talking about implicit bias. We earnestly explained to our friends and family that people have attitudes and stereotypes that influence how they see and interpret the world around them, and they might not even know it is happening. They were skep tical. We told them about tests that help scientists uncover and quantify these biases. They were notc onvinced. We told them to read Blink (Gladwell, 2005). A “real” author wrote that; they started to get it. Now, of course, implicit bias is discussed everywhere– court rooms, police departments, offices of human resources, corporate boardrooms, elementary schools, and colleges. The idea that even “good people” may harbor unwanted attitudes and stereotypes is commonplace, ordinary, perhaps even a bit insipid. We seem to have forgotten that, just two decades ago, these ideas were quite radical.” (Ratliff & Smith, p. 97).
Research on the unconscious, however, shows how hard it is to study unconscious processes and that widespread beliefs in them do not mean that they exist. At one point in time, academic psychologists were attacked for questioning the validity of repressed memories and it is now widely accepted that some (not all!) of these memories were constructions of events that never happened.
Like some psychoanalysts who lashed out against scientific critics, Project Implicit insiders dismiss valid scientific criticism without engaging with the scientific arguments.
“we disagree with arguments that moderate correlations between IAT scores and self-report suggest that the constructs are redundant (Schimmack, 2021), and thus implicit bias is uninteresting. These and similar arguments are difficult to reconcile with many people’s surprise and even resistance when confronted with evidence of their own bias” (Ratliff & Smith, p. 112).
This response is almost comically similar to a cartoonish psychoanalyst who tells a patient that (a) “you unconsciously want to kill your father,” (b) you unconsciously want to sleep with your mother,” or (c) “you unconsciously want to have a penis.” When the patient responds that this is clearly not the case, the psychiatrists claims that they are just using defense mechanisms to deny the truth about their hidden motives.
According to Ratliff and Smith any denial of biases revealed by the IAT is a defensive response, when most of the time, it is much more likely that the IAT scores are biased. They also mischaracterize Schimmack’s evidence, which may reveal a defensive reaction of their own. Schimmack showed that a large portion of the variance in IAT scores is random and systematic measurement error. Once measurement error is statistically corrected, IAT scores and self-reports on the race IAT are highly correlated. Thus, there is no evidence that IAT scores reflect anything that could diverge from people’s self-perceptions. Moreover, their self-reported attitudes are often stronger predictors of behavior than the small amount of unique variance in IAT scores, even in studies done by IAT proponents (Axt et al., in press; Greenwald et al., 1998).
Accuracy and Ethics of Feedback
The section “Accuracy and Ethics in Providing IAT Feedback” promises to address these problems, but falls short of engaging with the low validity of IAT scores as measure of implicit biases.
“Research shows the IAT is an effective educational tool for raising awareness about implicit bias, but the IAT cannot and should not be used for diagnostic or selection purposes (e.g., hiring or qualification decisions). For example, using the IAT to choose jurors is not justifiable, but it is appropriate to use the IAT to teach jurors about implicit bias” (Ratliff & Smith, p. 115).
What this statement leaves out is the reason why IATs should not be used for diagnostic purposes. The reason is that IAT scores have woefully inadequate validity; that is most of the variance in these scores is measurement error. So, how is it ethical to give people feedback about these scores if they are often invalid? The most revealing statement in the whole article is Ratliff and Smith’s answer to this question:
“This brings up an important question on which Project Implicit’s Scientific Advisory Board reflects frequently– is it ethical to pro vide participants feedback on their IAT performance? Thus far, the team has answered this question in the affirmative (a point to which we will return at the end of this section), but the team closely follows the literature on IAT reliability and malleability to make this decision and are open to reconsidering should the evidence suggest it is prudent to do so.”
The question is whether we can trust a team of researchers who are interested in collecting data in the virtual laboratory to make this ethical decision without conflict of interest. Maybe they should consult outsiders to avoid motivated biases that could harm people who receive false feedback without proper debriefing.
Aside from conflict of interest, a bigger problem is that the Project Implicit members have no formal training in developing, evaluating, and administering psychological tests, a discipline known as psychometrics and despite the similar name, largely removed from psychology. Even undergraduate students learn at some point that reliability is insufficient to evaluate test scores, but Ratliff and Smith never discuss validity and systematic measurement error in IAT scores.
They also confuse effect sizes for group means with scores of individuals. “The reasoning for these particular cut-offs is that, given that the standard deviations of IAT D-scores are rarely greater than 0.5 (Nosek et al., 2007), these IAT D-score cutoffs correspond approximately to Cohen’s d effect sizes of 0.3 (slight preference), 0.7 (moderate preference), and 1.3 (strong preference). These are above Cohen’s conventional cutoffs (i.e., 0.2, 0.5, 0.8), because the confidence interval around the estimate of a single score is likely to be greater than that of the confidence interval based on a sample mean. In other words, the feedback is somewhat conservative” (p. 101). This claim shows lack of knowledge about the scoring of test scores and the true amount of uncertainty around an individuals’ test score. Not surprisingly, they see no problem in providing invalid feedback based on their false assumption that the scoring is conservative.
The chapter does provide some interesting information about changes to the feedback that people are given. In the beginning, feedback claimed that IAT scores reveal unconscious biases. Ratliff and Smith emphasize that talks and educational materials no longer use the term unconscious (p. 112). Instead, “for several years now Project Implicit has used the term active awareness to reflect the fact that unawareness of implicit bias might be because one has not reflected deeply about their biases rather than because one cannot” (p. 112).
However, there is no evidence for this claim. A search on the Project Implicit website did not retrieve any relevant hits that mention active awareness and evidence that IAT scores reflect biases that operate without active awareness. Instead, the website continues to claim that implicit biases exist without awareness.
Some outsiders might consider this double deception. The description of the way Project Implicit is presenting itself to the public is deceiving readers who do not fact check the claim and the claim “without awareness” deceives people who visit the website that the test can tell something about them that they do not already know.
Conclusion
In conclusion, Project Implicit was created as a research laboratory for short studies with the aim to get responses from a large number of people. Many other researches have surveys posted, but do not get millions of visitors to do their surveys. Project Implicit has benefited from an affiliation with Harvard that suggests to many Americans that it is solid science and from marketing the IAT as a “window into the unconscious” (Banaji & Greenwald, 2013). Criticism of the validity of the IAT has been brushed aside with the claim that “Project Implicit gives feedback to participants about their IAT performance because of the perceived educational value in doing so.” The question remains who perceives this value. Many outsiders do not think that it is educational to give people false feedback about their unconscious. If the IAT is no different than a Rorschach test, why does it still get support from psychological science.
Fortunately, thanks to popular articles and blog posts the general public is learning more about the problems with the IAT and the concept of implicit biases (Schimmack, 2026; Singal, 2017). This blog post provides further evidence that the organization behind the online administration of the IAT lacks the scientific qualifications to do so and has put self-interest over ethics. Despite growing scientific evidence that IATs do not measure implicit biases, visitors are not given proper information about the accuracy of their feedback. Instead, resistance to the feedback is described as defensive. Ironically, the response by the scientific advisory board to criticism is a lot more defensive and less defensible than responses by people to do not believe the IAT.
Project Implicit is a nonprofit company founded in 1998 by three social psychologists: Tony Greenwald (University of Washington) Mahzarin Banaji (Harvard University) Brian Nosek (University of Virginia)
Project Implicit is mainly known as the company that hosts a website where people receive (false) feedback about their implicit associations based on the Implicit Association Test (IAT). The website is hosted by Harvard University, which is prominently displayed in web searchers, presumably because many Americans associate Harvard with excellent science.
However, the ethical oversight for the activities of Project Implicit rests with the Institutional Review Board with the University of Virginia’s IRB for Social and Behavioral Sciences. The Harvard branding is real but largely a legacy of Banaji’s professorship there; the organization is legally independent of Harvard.”
Project Implicit is now also hosting on an independent site as About the IAT – Project Implicit. Thus, the connection with Harvard may come to an end, but the website hosted by Harvard is still operational.
People
Based on the ProPublica 990 data, the leaders of Project Implicit in the fiscal year 2025 were:
Amy Jin Johnson — Executive Director (the only compensated employee, at $111,038)
Dr. Brian Nosek — President (University of Virginia; co-founder)
Dr. Kate Ratliff — Treasurer (University of Florida)
Keith Maddox, PhD — Director
Jarvis Idowu — Director
Bayet Ross Smith — Director
The affiliation with University of Virginia and Brian Nosek’s role as president and co-founder make it clear that Brian Nosek is the main person responsible for the ethical integrity of Project Implicit’s scientific work and the administration of IATs to the general public.
Financials
The picture that emerges is of a very small operation that is burning through reserves. As a 501(c)(3), Project Implicit files Form 990s with the IRS, which are publicly accessible. The ProPublica Nonprofit Explorer has their filings going back to 2011.
For fiscal year ending September 2025: revenue of $104,552, expenses of $296,971, a net loss of $192,419, and total net assets of $365,382. The dominant revenue source was program services (82% of revenue, at $86,100), with investment income making up most of the rest. Public donations were negligible at $675 (0.6%).
The prior year (FY2024) showed revenue of $273,966 against expenses of $489,223 — another large deficit — and the year before that (FY2023) showed revenue of $436,655 against expenses of $522,546.
So revenues have dropped sharply over three years (~$437K → ~$274K → ~$105K) while expenses remain high relative to income. They are drawing down net assets at a significant rate.
The main revenue of Project Implicit are fees for program services:
Corporate/organizational DEI training and consulting — companies, government agencies, universities, and HR departments pay Project Implicit to run implicit bias workshops, license the IAT for their own use, or deliver training programs. This has been a significant revenue stream for them, especially during the DEI boom years of 2020–2022.
Licensing or access fees — organizations that want to use the IAT infrastructure for research or applied purposes may pay for that.
Speaking and educational programs — paid engagements where Project Implicit personnel deliver training.
The trajectory tells an interesting story. Program service revenue went from ~$308K (FY2023) to ~$240K (FY2024) to ~$86K (FY2025) — a collapse of roughly 72% in two years. That almost certainly tracks the broader pullback in corporate DEI spending that accelerated after 2023 and especially into 2024–2025. Thus, while the website hosts hundreds of different IATs, the race IAT is the bread and butter IAT that funds the organization. The collapse in revenues can be explained by the changing political climate under the “Make Racism Great Again” policies of the MAGA government. There is no evidence that sustained criticism of the validity of IATS in general and the race IAT specifically over the past decades has contributed to this sharp drop in revenues.
Project Implicit’s mission statement has changed considerably over time, against the backdrop of accumulating scientific criticism of the IAT and the organization’s broader institutional repositioning. The changes are visible not only in the language itself, but also in where the organization now presents itself to the public.
An older version still visible on the Harvard-hosted site describes an organization that “provides consulting, education, and training services on implicit bias, diversity and inclusion, leadership, applying science to practice, and innovation” (app-prod-03.implicit.harvard.edu, retrieved June 1, 2026). The earliest version cached by the Wayback Machine, from 2013, contains the same language. The current projectimplicit.net site describes its educational work in considerably more cautious terms, as providing “research-based educational programs that translate findings from cognitive science into clear, accessible understanding of judgment and decision-making, without prescribing behavior change or organizational intervention.”
The phrase “without prescribing behavior change or organizational intervention” marks a significant retreat. The earlier language presented Project Implicit as an organization that translated implicit-bias science into diversity, inclusion, leadership, and applied organizational practice. The current language distances the organization from prescriptive behavior change and organizational intervention. This does not mean that Project Implicit has abandoned all consulting or educational services. Rather, it means that the organization has narrowed the public rationale for those services. It no longer presents itself as directly prescribing organizational change, but as providing research-based education about judgment and decision-making.
That retreat is important, but it is incomplete. Even the current mission statement continues to claim the authority of “research-based” education and the translation of “findings from cognitive science.” Those phrases preserve the impression that Project Implicit is communicating settled scientific knowledge. But the central scientific problem remains unresolved. The issue is not whether racial disparities, prejudice, or discrimination exist. They plainly do. The issue is whether IAT scores validly measure implicit prejudice at the individual level, and whether individualized feedback about hidden racial bias is scientifically justified.
The evidence does not support that stronger interpretation. IAT scores have limited validity, weak relations with behavior, and substantial ambiguity in what they measure (Schimmack, 2021). They are influenced by task-specific processes, cultural associations, and systematic sources of measurement error. In the case of the race IAT, the color-valence confound raises the additional possibility that scores partly reflect general associations with black and white rather than racial attitudes themselves. These limitations are not minor qualifications. They go to the construct validity of the measure and to the ethical defensibility of giving people individualized feedback about hidden racial bias.
Ethics
The administration of psychological tests to assess individuals with clinical relevance is regulated by professional bodies such as the American Psychological Association. However, these strict ethical rules do not apply to test that are administered for other purposes. Anybody can host a website and give people scores on some test.
Millions of people have taken tests like astrological birth chart generators or the “What kind of pizza are you? test (Pizza Test). However, as academics, Brian Nosek and Project Implicit are required to have ethical approval for the administration of IATs, especially because they are using the data for research purposes. Currently, the IRB of the University of Virginia is responsible for the ethical oversight of Project Implicit’s activities.
The IRB protocol obtained from Brian Nosek — the only document he could find, dated 2006 — confirms that the ethical oversight of Project Implicit has not kept pace with the scientific evidence.
The 2006 protocol acknowledges that participants may be “surprised” and “concerned” by their results, and promises debriefing that contextualizes scores as having “no direct implications for individual scores.” But it makes no mention of the limited reliability of IAT scores, the color-valence confound, the absence of construct validity evidence, or the specific risks to African American participants of being told they harbor hidden pro-White bias.
A protocol written in 2006, before the major validity critiques were published, and apparently never formally updated, cannot provide adequate ethical oversight for a research enterprise that has since accumulated overwhelming evidence of the instrument’s limitations. The fact that Nosek’s response to a direct request for the current IRB protocol was to send a 20-year-old document is itself an answer.
UVA seems to treat this project like any other research project, but Project Implicit research is different because it gives people feedback about potential hidden biases. The key claim is that they measure processes that are not directly accessible to introspection. This is also used to explain why people may receive feedback that is inconsistent with their self-perceptions — the supposed reason being that the test revealed something true about them that is not accessible to conscious awareness, much like a psychoanalyst claiming to recover a forgotten or repressed memory. These claims are controversial because they are difficult to verify, and the epistemic structure is problematic: participants cannot dispute the feedback on the basis of their own experience because the whole point is that the bias is hidden from them. The danger is that discrepancies between IAT scores and self-perceptions are more likely to reflect measurement error in the IAT than truly hidden biases — a conclusion supported by published psychometric research (Schimmack, 2021). As a result, a substantial proportion of participants will receive false feedback about racial attitudes they do not hold and people are not given proper debriefing that the most likely reason for surprising results is measurement error.
Implicit Biases of Project Implicit
Given the seriousness of providing people with feedback about hidden biases on topics like prejudice, depression, and suicide, one might expect that Project Implicit has carefully evaluated the psychometric properties of IATs — that is, assessed the accuracy of IAT scores. However, this is not the case. None of the three founding members has training in psychometrics or demonstrated understanding of modern test theory, as evidenced by their failure to apply basic psychometric concepts such as discriminant validity, convergent validity with other implicit measures, or the fundamental constraint that validity cannot exceed reliability (Schimmack, 2021).
Most of the discussion of measurement error in the IAT literature has focused on random measurement error and situational influences on IAT scores. This limited focus ignores that IAT scores can also be influenced by systematic measurement error. Random error averages out across repeated administrations; systematic error does not. If IAT scores are systematically influenced by factors such as cognitive ability or task-switching rather than hidden bias, repeated testing will not produce valid feedback about hidden biases. Neglect of systematic measurement error is common in psychology, but the ethical stakes are considerably higher when such error invalidates personal feedback about sensitive topics like racial prejudice, depression, or suicidal ideation.
The finding that the average white, Asian, or non-white Hispanic American finds it easier to associate white with good and black with bad rather than the other way around does not mean that they are prejudiced against Black people. It also does not show that they are unbiased. In fact, self-reports show that a substantial number of people are aware of and willing to admit their prejudices.
Brian Nosek, the director of Project Implicit, has ignored scientific criticism of the interpretation of IAT scores made by numerous researchers using independent lines of argument. One is that IAT scores show low convergent validity with other implicit measures — meaning that a person classified as biased on the IAT may not be classified as biased on other implicit measures of the same construct. Yet visitors to the Project Implicit website are offered only the IAT, with no acknowledgment that other implicit measures exist or that they frequently disagree with IAT scores. While the name Project Implicit implies a focus on implicit constructs, the site is really just promoting the Implicit Association Test, even though it lacks validity to measure implicit biases.
Is the race IAT itself racist?
The scoring of the race IAT rests on a simple assumption. If reaction times in favor of white-good, black-bad are faster than black-good and white-bad, a person shows an implicit bias favoring whites. This scoring assumes that a value of zero corresponds to a psychological attitude that is neutral and unbiased. While this assumption is intuitively appealing, it requires scientific evidence. An alternative possibility is that scores on the race IAT are also influenced by factors that have nothing to do with prejudice.
One way to validate the assumption is to see how scores on the IAT are related to actual behaviors. If zero reflects neutrality, people with scores above zero should show prejudice in their behaviors and people with scores below zero should show the opposite pattern, a preference for Black people. However, no compelling evidence has been provided that reaction time differences map directly on amount of bias in behavior.
A critical analysis of the literature failed to provide evidence for the scoring of the race IAT that is used to provide people with feedback about their hidden biases (Blanton, Jacard, Strauts, Mitchell, & Tetlock, 2015) [ironically, Mitchell is also affiliated with UVA that oversees the ethics of Project Implicit]. There has been no response to this criticism and no research to demonstrate that the scoring of the race IAT is valid by Project Implicit since then. There is also no response by Brian Nosek or other founders of Project Implicit to more recent criticisms (Schimmack, 2021).
Moreover, there has been research that has examined why the IAT may have a bias towards white-good/black-bad associations; that is, the test itself is biased. The first problem is that American culture is filled with racial stereotypes that associated Black people with negative attributes. Mere awareness of these stereotypes may influence IAT scores, even if people hold favorable attitudes towards specific Black people or even African Americans as a group (Olson & Fazio, 2004). Even African Americans are aware of these stereotypes and their responses may be influenced by these associations. In support of this argument, responses are more neutral on other tasks that rely on specific stimuli (faces of European and African Americans) rather than abstract associations.
More challenging for the race IAT is the finding that simple color associations explain a substantial portion of the variance in scores on the race IAT (Smith-McLallen, Johnson, Dovidio, & Pearson, 2006). This means the race IAT is not a pure measure of racial biases because it is contaminated by general associations related to the colors white and black. Although this problem was reported 20 years ago, it has been largely ignored by the research community and by Project Implicit. The implication is that African Americans who like white cars and white clothing may receive feedback that they have a hidden bias against African Americans.
Durgin, Diop, Lewis-Owona, and Eaton (in press) replicated and substantially extended Smith-McLallen et al.’s findings across six experiments. They showed that the correlation between color IAT scores and race IAT scores is of similar magnitude to the test-retest correlation of the race IAT itself, suggesting that the two instruments are measuring largely the same underlying construct. Critically, the shared variance between the color and race IATs was not explained by explicit racial bias but by metaphoric alignments of black and white — the deep cultural association of darkness with evil present across racial groups. Even Black participants showed similar metaphoric color alignments to White participants, and a blue-gray color IAT showed no correlation with the race IAT, confirming the effect is specific to black-white alignments rather than a general method artifact.
These results undermine the validity of race IAT scores, especially for African Americans. This matters because the validity of test scores must be assessed within populations, not just in aggregate. However, IAT validation studies have relied exclusively on White or mixed samples, meaning the test has never been properly validated for African Americans. Durgin et al.’s findings suggest that race IAT scores are even less valid for African Americans than for European Americans, as the metaphoric color bias and in-group effects pull in opposing directions, making individual scores particularly difficult to interpret.
Good Intentions and Bad Behavior
Racists often accuse social psychologists of a left-leaning, liberal bias. However, racial equality is enshrined in the 13th, 14th, and 15th Amendments to the Constitution of the United States, passed after the Northern States won the Civil War against the Confederate States that sought to maintain slavery. Working towards Martin Luther King’s dream of actual racial equality is therefore aligned with the moral and political ideals of the United States.
Project Implicit was founded on the idea that many Americans embrace Martin Luther King’s dream but often act in violation of egalitarian principles — sometimes due to limitations in their ability to control their behavior, and sometimes because they are not even aware that their actions are influenced by race. The founding vision of Project Implicit was that a five-minute reaction time task could help people become aware of their biases, and that this awareness would be a first step toward changing their behavior.
The problem is that early on, research findings suggested that the race IAT could not deliver on this promise. However, well-known motivated biases made it impossible for Nosek, Banaji, and Greenwald to acknowledge their own biases and temper their enthusiasm about IATs as “windows into people’s unconscious” (Banaji & Greenwald, 2013). Instead, they continued to promote the test, generated substantial revenues for Project Implicit, and aggressively promoted the concept of implicit biases to a broad public audience and ignored valid criticism of IATs as measures of implicit biases.
At this point, the dream of Martin Luther King and the dream of Nosek, Banaji, and Greenwald diverged. Project Implicit promoted a research program and a task that did not increase awareness of bias and did not reduce racism. In fact, the recent surge in open, old-fashioned racism may partly reflect a backlash against DEI programs and implicit bias training. Some people did not resent feedback that they were racist — they resented the implication that racism is bad and that they need to change. These people are now fighting back against DEI programs because they wish to maintain the racial hierarchy established during slavery and perpetuated through the Jim Crow laws of former Confederate states.
Project Implicit was built on a false understanding of racism in the United States, an invalid measure of racial bias, and a failure to connect laboratory findings to actual discriminatory behavior. These problems might have been recognized sooner had Project Implicit — which derived most of its revenues from the use of the race IAT in DEI training — consulted with African American communities or scholars. There is little public evidence that their work on racial issues involved meaningful engagement with the actual targets of racial discrimination.
Giving False Feedback to African Americans
It seems that Brian Nosek trusted the validity of the race IAT even when self-reports of African Americans suggested otherwise (Jost, Banaji, & Nosek, 2004). Millions of people have taken the race IAT on the Project Implicit website and also reported their consciously accessible preferences. Many of them were African Americans and research articles show their results at the aggregate level.
A robust finding based on hundreds of thousands of scores shows a striking dissociation in African Americans’ racial attitudes: on explicit self-report measures, African Americans show strong ingroup favoritism — clearly preferring their own group — yet on the race IAT they score close to zero, showing neither consistent preference for Black nor for White (Nosek et al., 2007; Jost et al., 2004).
This dissociation has two possible interpretations. Either African Americans hold two genuinely different attitudes — one conscious and pro-Black, one unconscious and neutral or pro-White — or they hold one attitude, the explicit measure captures it accurately, and the IAT is biased for this group in ways that suppress the ingroup preference that is clearly present in self-reports. The second interpretation is strongly supported by the documented color-valence confound in the race IAT, the near-zero mean being equally consistent with cultural contamination of the measure, and the fundamental psychometric principle that validity cannot exceed reliability.
Nevertheless, Nosek, Banaji, and Greenwald — three non-African American scholars with no documented engagement with African American communities or scholars — chose the most psychologically and politically loaded interpretation available: that many African Americans harbor a hidden pro-White bias rooted in system justification, a motivated tendency to endorse the existing social order even when that order places them at the bottom of the racial hierarchy.
This is a remarkable claim. Translated out of theoretical language, it asserts that the race IAT reveals that many African Americans are unconsciously motivated to maintain a social system that affords them fewer rights, lower status, and less economic opportunity than White Americans. The claim is made on the basis of a psychometrically compromised instrument, without consulting African American communities or scholars, and in direct contradiction of the most obvious behavioral evidence available. African Americans vote overwhelmingly Democratic — approximately 80% overall and 90% among women — consistently supporting the party associated with anti-racism policies and government intervention to address racial inequality. This is not the behavior of a group that unconsciously endorses the racial status quo. More broadly, African Americans have actively resisted racial hierarchy throughout their entire history in the United States, from the abolitionist movement and Reconstruction to the civil rights movement and beyond. System justification theory, as applied to African Americans through the race IAT, mistakes the cognitive fingerprints of living under racism for psychological endorsement of it.
Although this claim was made in the most highly cited article in the journal Political Psychology (1,277 citations in Web of Science), it has received little critical attention outside the academic literature. Black activists and scholars working on racism have largely ignored this work rather than directly challenging it — not because they accept it, but because Project Implicit’s research program is so disconnected from the empirical traditions and practical concerns that dominate Black psychology and anti-racism activism. This neglect further underscores that Project Implicit operates largely in isolation from broader anti-racism efforts in the United States. African American scholars from W.E.B. Du Bois onward have had good reasons to be skeptical of psychological instruments developed by White researchers to make claims about the inner lives of Black Americans — the history of IQ testing used to pathologize Black communities is instructive. Project Implicit repeated this pattern without appearing to recognize it. The fundamental problem is that the focus of Project Implicit is the measure, not the construct of racial bias. An organization genuinely committed to understanding and reducing racism would follow the evidence wherever it leads, including away from its flagship instrument. Project Implicit has done the opposite.
It is particularly troubling that this interpretation of African Americans’ scores was made by prominent members of Project Implicit, including Nosek himself. If the system justification interpretation is wrong — and the psychometric evidence strongly suggests it is — then African Americans who receive pro-White feedback on the race IAT are being told something false and potentially harmful about their own psychology. The ethical stakes are highest precisely for this group, yet the 2006 IRB protocol makes no mention of the specific risks to African American participants, provides no tailored debriefing to address the system justification interpretation, and offers no guidance on how to contextualize a pro-White result for a Black participant who strongly identifies with their own group. This is not a minor oversight. It is the most serious ethical failure in Project Implicit’s research program.
Conclusion: So, What is Project Implicit?
In my opinion, Project Implicit is a research project built around an experimental paradigm. Participants are asked to perform two complementary reaction time tasks, and the outcome is the difference in response times between them. This task is called the Implicit Association Test. Like many experimental paradigms, the IAT gives social psychologists something to do and write articles about. This academic research is inexpensive and not directly connected to real-world problems. It is basic research by academics in the ivory tower, for researchers in other ivory towers.
However, Project Implicit took this experimental paradigm and presented it to the public as a valid measure of hidden biases and unconscious processes, and as a tool capable of assessing those processes at the level of individual people. It provided individuals with feedback about their scores on a publicly accessible website, used the research to support seminars and public speaking engagements about implicit bias, and claimed that this work could address real social problems. This marketing was extremely effective, in part due to Banaji’s affiliation with Harvard, and Project Implicit generated substantial revenues over two decades while ignoring mounting evidence that the IAT is not a valid instrument for studying racism or reducing it.
Largely unrelated to this scientific evidence, the resurgence of open racism in American politics is draining Project Implicit of revenue, and the organization appears to be running out of money. This would be a serious loss if Project Implicit had made genuine progress in the fight against racism. But it did not. Instead, it deflected attention from real problems and drained resources — financial, institutional, and intellectual — from more effective anti-racism efforts. The projected demise of Project Implicit is therefore a blessing in disguise.
Unfortunately, the real problem of racism remains. Many Americans are unwilling to abandon their racial prejudices and to treat all people as equal under the law. Martin Luther King’s dream remains elusive — not because we lacked a reaction time task to measure hidden bias, but because we lacked the collective will to confront the bias that was never hidden at all.
References
Axt, J. R., Connor, P., Hoogeveen, S., Clark, C. J., Vianello, M., Lahey, J. N., Hahn, A., To, J., Petty, R. E., Costello, T. H., Mitchell, G., Tetlock, P. E., & Uhlmann, E. L. (in press). On the relationship between indirect measures of Black vs. White racial attitudes and discriminatory outcomes: An adversarial collaboration using a sample of White Americans. Journal of Personality and Social Psychology.
Banaji, M. R., & Greenwald, A. G. (2013). Blindspot: Hidden biases of good people. New York: Delacorte Press.
Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61(1), 27–41.
Blanton, H., Jaccard, J., Strauts, E., Mitchell, G., & Tetlock, P. E. (2015). Toward a meaningful metric of implicit prejudice. Journal of Applied Psychology, 100(5), 1468–1481.
Durgin, F. H., Diop, S. M., Lewis-Owona, J., & Eaton, O. (in press). A downside of conceptual metaphor: Metaphoric alignments of black and white. Manuscript submitted for publication.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74(6), 1464–1480.
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85(2), 197–216.
Hahn, A., & Gawronski, B. (2019). Facing one’s implicit biases: From awareness to acknowledgment. Journal of Personality and Social Psychology, 116(5), 769–794.
Jost, J. T., Banaji, M. R., & Nosek, B. A. (2004). A decade of system justification theory: Accumulated evidence of conscious and unconscious bolstering of the status quo. Political Psychology, 25(6), 881–919.
Karpinski, A., & Hilton, J. L. (2001). Attitudes and the Implicit Association Test. Journal of Personality and Social Psychology, 81(5), 774–788.
Kurdi, B., Seitchik, A. E., Axt, J. R., Carroll, T. J., Karapetyan, A., Kaushik, N., … & Greenwald, A. G. (2019). Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis. American Psychologist, 74(5), 569–586.
McFarland, S. G., & Crouch, Z. (2002). A cognitive skill confound on the Implicit Association Test. Social Cognition, 20(6), 483–510.
Meier, B. P., Robinson, M. D., & Clore, G. L. (2004). Why good guys wear white: Automatic inferences about stimulus valence based on brightness. Psychological Science, 15(2), 82–87.
Meier, B. P., Fetterman, A. K., & Robinson, M. D. (2015). The brightness of your smile: The solar hypothesis of the affect-brightness link. In Handbook of embodied cognition and sport psychology. MIT Press.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration website. Group Dynamics: Theory, Research, and Practice, 6(1), 101–115.
Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., … & Banaji, M. R. (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18(1), 36–88.
Olson, M. A., & Fazio, R. H. (2004). Reducing the influence of extrapersonal associations on the Implicit Association Test: Personalizing the IAT. Journal of Personality and Social Psychology, 86(5), 653–667.
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2013). Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies. Journal of Personality and Social Psychology, 105(2), 171–192.
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2015). Using the IAT to predict ethnic and racial discrimination: Small effect sizes of unknown societal significance. Journal of Personality and Social Psychology, 108(4), 562–571.
Schimmack, U. (2021). The Implicit Association Test: A method in search of a construct. Perspectives on Psychological Science, 16(2), 396–414.
Smith-McLallen, A., Johnson, B. T., Dovidio, J. F., & Pearson, A. R. (2006). Black and White: The role of color bias in implicit race bias. Social Cognition, 24(1), 46–73.
Worden, R. E., Najdowski, C. J., McLean, S. J., Worden, K. M., Corsaro, N., Cochran, H., & Engel, R. S. (2024). Implicit bias training for police: Evaluating impacts on enforcement disparities. Law and Human Behavior, 48(5–6), 338–355.
Hoenig, J. M., & Heisey, D. M. (2001). The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897
The Influential Warning About Power Calculations
Hoenig and Heisey (2001) wrote an influential article titled “The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis.” The article warned researchers against computing statistical power from the results of a completed study. Subsequent articles have repeated this warning, and it is now widely considered a fallacy to compute “observed power” after the results are known.
One of the less convincing arguments against post-hoc power calculations is that observed power is just a transformation of the p-value. Many statistical quantities are transformations of each other. That does not make them useless. A p-value is a transformation of a test statistic, and a test statistic is often a ratio of an effect-size estimate to its standard error. The core information obtained from an empirical study is the effect-size estimate and its standard error. Confidence intervals are another way to represent that information. The fact that confidence intervals are transformations of the same information does not mean that they should not be computed.
The stronger argument is that post-hoc power calculations often provide misleading answers to the questions researchers want to ask. The concept of power is tied to hypothesis testing. A meaningful power calculation asks: How likely was my study to reject the null hypothesis if a specific alternative hypothesis was true? The problem is that most researchers do not make explicit quantitative predictions. Directional claims that an effect is positive do not translate into a single power value. The probability of obtaining a statistically significant result depends on the unknown population effect size. Thus, the true power of a study is unknown. The results of a completed study, which include sampling error, cannot be used to observe the true power of that study. This is why the term “observed power” is misleading. We cannot observe population parameters; we can only estimate them with uncertainty.
Hoenig and Heisey’s main point was that sampling error and conditioning on statistical significance lead to misleading claims about power. If a study produced a significant result, post-hoc power cannot be very low, because the observed effect-size estimate had to be large enough relative to its standard error to cross the significance threshold. For z-values, a two-sided p-value of .05 corresponds to a critical value of about 1.96. Thus, a significant result requires an estimate that is about two standard errors away from zero. The reverse is true for nonsignificant results. Studies that fail to reach significance will tend to produce low post-hoc power estimates. Researchers who obtain significant results therefore get observed-power estimates that look moderate or high, whereas researchers who obtain nonsignificant results get observed-power estimates that look low. The fallacy is to assume that these values represent the true power of the study. Significant results were not necessarily obtained with moderate or high power, and nonsignificant results cannot automatically be attributed to low power.
Selection for significance means that the estimates are biased. When results are selected because they are statistically significant, the observed effects are typically larger than the true effects. This is a form of regression to the mean, although the mean itself is unknown. If all significant results were false positives, the true probability of obtaining a significant result would simply be the Type I error rate, typically 5%. Yet the observed power of significant false positives is much higher (~ 62% on average) This shows how dangerous it is to confuse post-hoc power with true power.
The Omitted Warning About Post-Hoc Effect-Size Calculations
The previous discussion shows that the problem with post-hoc power calculations is not the calculations. The problem is that the calculation treats the observed effect-size estimate as if it were the true population effect. When a result is selected because it is statistically significant, this assumption is often misleading. Significant results from small studies tend to have inflated effect-size estimates because only estimates large enough to cross the significance threshold are likely to be noticed, reported, or published. Thus, the root problem is biased effect-size estimation, not the conversion of an effect-size estimate into a power estimate.
This creates an odd asymmetry in current statistical practice. Reporting effect-size estimates from observed data is widely recommended and often required, whereas using the same estimates to compute post-hoc power is widely dismissed as a fallacy. But if post-hoc power is misleading because it is based on a noisy and selected effect-size estimate, then the same concern applies to the selected effect-size estimate itself.
The importance of this concern varies across research areas. In some fields, effect-size estimation is the central goal. Public opinion polling is a simple example. A poll with a reasonably large sample may estimate support for a candidate with a margin of error of only a few percentage points. In that setting, the point estimate is useful because the estimate is relatively precise and is not usually selected for publication because it crossed an arbitrary significance threshold.
The situation is different in many areas of psychology. Sample sizes are often small, standard errors are large, and the incentive structure rewards statistically significant findings. Since the 1950s, reviews of the psychological literature have shown that the large majority of published articles report at least one statistically significant result used to support a substantive claim (Sterling, 1959; Sterling et al., 1995). In this context, post-hoc effect sizes calculations can be highly misleading. A study may provide evidence about the direction of an effect, but the reported effect size can be much larger than the true population effect.
I am not the first to note that selection for statistical significance inflates effect-size estimates from small studies. Button et al. (2013), for example, emphasized that low-powered studies produce exaggerated estimates and unreliable findings. However, this literature has generally stopped short of treating routine interpretation of selected effect-size estimates with the same severity as post-hoc power analysis. This is striking because both quantities are compromised by the same input: a noisy effect-size estimate selected because it crossed a significance threshold.
The problem is especially serious when the observed effect size looks impressive precisely because the study had low precision. In a small study, statistical significance often requires a large observed effect. As a result, the studies least able to estimate the true effect accurately are also the studies most likely to report exaggerated effect sizes when they are significant. The ego-depletion literature illustrates this problem. Early post-hoc effect sizes suggested a sizable average effect, whereas later large-scale tests with small confidence intervals produced small post-hoc estimates and the confidence interval still included zero, suggesting that many studies reported notable post-hoc effect sizes when the true value was practically zero..
Thus, reporting observed effect-size estimates from significance-selected studies can have the same basic problem as reporting post-hoc power. The values may look informative, but they can be driven by the same selected and inflated estimate. The result is a misleading evidential package: a significant p-value, a seemingly large effect size, and sometimes a high observed-power estimate, all derived from the same noisy result.
Hoenig and Heisey’s warning about post-hoc calculations therefore applies not only to post-hoc power calculations, but also to post-hoc effect size calculations. The issue is not that one calculation is inherently fallacious while the other is automatically good practice. The issue is that both can be misleading when researchers treat selected, imprecise estimates as if they revealed the true magnitude of an effect.
Confidence Intervals as Conservative Hypothesis Tests
The previous discussion shows that the problem is inherent in the data, not in a particular calculation based on the data. Studies with large standard errors provide imprecise quantitative information. The central task of statistical inference is to represent this uncertainty honestly.
The solution is nearly a century old and is associated with Neyman’s theory of confidence intervals. Given an acceptable error rate, such as 5%, uncertainty in an estimate can be represented by adding and subtracting approximately two standard errors from the observed estimate. In the long run, under the assumptions of the model, this interval will contain the true value 95% of the time. The remaining 5% of intervals will miss the true value because sampling error moved the estimate too far upward or downward.
Confidence intervals are familiar to most people because they are used in public opinion polling. Assuming unbiased sampling of respondents, election polls will contain the true population value within the reported margin of error approximately 95% of the time. The same logic applies in fields with larger sampling error, but the resulting intervals can be very wide. Cohen (1994) speculated that psychologists were reluctant to report confidence intervals because they could be embarrassingly wide. A study might report an impressive value of a post-hoc effect size calculation, d = .60, but the 95% confidence interval might range from d = .06 to d = 1.14. This interval covers nearly the full range of plausible effect sizes without any data collection. In such a case, the point estimate of d = .60, is misleading and fails to alert readers that the true effect size could be close to zero.
Psychology has gradually embraced the reporting of confidence intervals as a good practice, but confidence intervals are often treated as secondary qualifications attached to the value from a post-hoc effect size calculation. The fallacy is to focus on this single value and to ignore the wide range of equally plausible values. In practice, many articles focus on the single post-hoc value and ignore the wide range of other values that are compatible with the data.
The proper use of confidence intervals is to treat them as conservative hypothesis tests. A confidence interval that does not include zero can support a directional claim. If the lower bound of the interval is d = .06, the study provides evidence that the true effect is positive. However, it does not justify the claim that the effect is moderate or large simply because the post-hoc effect size value is moderate or large. The study rules out values below the lower limit of the interval. It does not rule out small positive effects. The appropriate conclusion is therefore limited: the effect is likely positive and larger than d = .06. That is the only magnitude claim the study can support.
Confidence intervals do not solve the problem of selection bias. When studies are selected because they are statistically significant, the entire interval can be shifted upward, including the lower bound. Thus, even a lower bound such as d = .06 may still be inflated; the true effect could be smaller, zero, or even negative. Nevertheless, confidence intervals reduce the abuse of post-hoc calculations because they prevent researchers from making strong claims about the size of an effect when the data do not warrant such claims. A study with an observed effect of d = .60 and a confidence interval from d = .06 to d = 1.14 may justify the limited claim that the result is statistically compatible with a positive effect, but it does not justify the stronger claim that the study demonstrated a moderate or large effect.
The same logic applies to nonsignificant results in psychology. A nonsignificant result is not automatically uninformative. It is uninformative when the confidence interval is so wide that it fails to rule out effects that would matter theoretically or practically. Conversely, a nonsignificant result can be highly informative when the confidence interval is narrow enough to rule out effects of meaningful size. This point deserves its own discussion, but for the present argument the lesson is simple: the evidential value of a result depends less on whether p is below .05 than on what values the confidence interval rules out..
To encourage the proper use of confidence intervals, we should reconsider the prominent reporting of post-hoc effect-size values in low-precision studies. For example, we could require that sampling error is less than .1 or some criterion of sufficient precision before articles report point estimates.
Of course, point estimates are needed to compute confidence intervals and reporting them is not the real problem. The problem is when they are treated as the headline result when the interval is wide. In such cases, the post-hoc effect size values are just as misleading as post-hoc power values. Moreover, the focus on post-hoc effect size values rewards researchers who selectively publish large effects from small samples and report impressive and inflated values, while disadvantaging researchers who invest resources in studies with narrow confidence intervals that provide actual quantitative information about true effect sizes.
Calling post-hoc effect size calculations a fallacy may seem radical, but it is consistent with Hoenig and Heisey’s widely accepted warning against post-hoc power calculations. “Observed power” is misleading because it is easily mistaken for true power. “Observed effect-sizes” are misleading for the same reason: they are easily mistaken for true effect sizes. The key fallacy is not a particular statistical calculation. The key fallacy is confusing uncertain estimates with the true values they are meant to estimate.
In short, it is time to treat post-hoc effect sizes with the same suspicion as post-hoc power values. This does not mean that we should avoid quantifying effects size estimates. We just need to remember that all estimates are estimates are estimates and not confuse estimates with unknown true values. Neyman provided us with a statistical tool to do so. We just have to start using it properly in psychology as it is already being used in other fields.
“You can’t teach an old dog new tricks.” (Proverb)
Dolores Albarracín and the Defense of Old Social Psychology
Dolores Albarracín is a prominent social psychologist at the University of Pennsylvania whose work focuses on attitudes, persuasion, and behavior change. She has held major editorial positions in the field, including editor-in-chief of Psychological Bulletin from 2014 to 2020 and currently editor of the Attitudes and Social Cognition section of the Journal of Personality and Social Psychology (JPSP).
But which social psychology does she represent: the old social psychology of selectively publishing studies that confirmed researcher expectations, or the new open science that reports results independent of their desirability.
The answer can be found in two meta-analyses of the contested social (implicit) priming literature that has been the posterchild of the replication crisis. Albarracin published not one, but two meta-analyses in defense of social priming in Psychological Bulletin (Weingarten et al., 2016; Dai et al., 2023); the first one while she was editor of the journal.
The second one had to deal with the fact that many replication failures by new social psychologists willing to publish replication failures showed no evidence. Albarracin and her co-authors dismiss this evidence. They suggest that replication studies are themselves biased toward null results — a “reverse publication bias” — and therefore should be discounted or at least treated with the same suspicion as the old studies that used unscientific practices and selection of significant results to claim the effects are real and important.
The support for their claim is a blog-post about political bias in social psychology. In contrast, the publication bias in the older studies is not taken seriously leading to the dubious claim that implicit priming is a real phenomenon, even though Albarracin herself has not been able to demonstrate her own findings again in pre-registered new studies.
It is telling that somebody with this track-record and open hostility to the new and open social psychology is now editor of the very same journal that published Bem’s (2011) pseud-scientific evidence of extrasensory abilities. The irony is hard to miss. The journal that published false claims about extrasensory abilities is now controlled by somebody who makes false claims about open science practices and the credibility of implicit priming studies This is not a good look for social psychology in the 2020s.
Science is self-correcting, but nobody said that this process is fast and painless. It may require another decade for social psychology to fix all the problems that gave JPSP the name Journal of Pseudo-Scientific Psychology. Sadly, Albarracin is part of the problem, not of the solution. Fortunately, time is on the side of progress and the time for old social psychology is running out.
Cookie Consent
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
