Category Archives: Credibility Crisis

Credibility in Economics: A Reanalysis of Large-Scale Meta-Research

August 13, 2026Credibility, Credibility Crisis, Economics, ZcurveUlrich Schimmack

Askarov, Z., Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2024). Selective and (mis)leading economics journals: Meta-research evidence. Journal of Economic Surveys, 38(5), 1567–1592. https://doi.org/10.1111/joes.12598

Abstract

Askarov, Doucouliagos, Doucouliagos, and Stanley (2024) analyzed statistical power and excess statistical significance in a large collection of economics meta-analyses and concluded that much of the evidence reported in leading economics journals is potentially misleading. We used their open data to conduct a z-curve analysis to examine the credibility of economics using a different statistical model. Z-curve has several advantages over the power-analysis and Test of Excess Significance (TES) approach used by Askarov et al. First, it does not assume that all studies within a meta-analysis share a single population effect size. Instead, it models heterogeneity with a mixture model. Second, z-curve models selection for statistical significance and uses the fitted distribution of significant results to estimate the discovery rate that would be expected in the absence of selection. The discrepancy between the observed and expected discovery rates therefore provides a direct measure of selection bias. In contrast, TES does not explicitly model how selection distorts the distribution of observed effect sizes when estimating expected significance. Its UWLS estimator gives greater weight to more precise estimates, which typically come from larger samples. If smaller, less precise studies report inflated effect sizes, the weighted mean will be pulled toward the smaller effects observed in more precise studies, thereby reducing the estimated power assigned to the smaller studies. This weighting can reduce small-study bias, but it does not necessarily eliminate selection bias. Moreover, if true effect sizes systematically differ with study size, the same weighting can itself produce a biased estimate of the average effect. Third, z-curve distinguishes between overall power (the Expected Discovery Rate, EDR) and power conditioned on significance (the Expected Replication Rate, ERR). With heterogeneous data, the average power of significant results can be much higher than overall power. Finally, z-curve uses the EDR to obtain an upper bound on the false discovery rate using a formula developed by Sorić (1989).

First, Askarov et al.’s estimate-level mean power and the z-curve EDR are surprisingly similar, approximately 27% and 28%, respectively. A discovery rate of this magnitude implies a maximum false discovery rate of approximately 14%. Second, the expected replication rate of statistically significant results is approximately 70%, showing that the power of selected significant results is substantially higher than overall power. These estimates are similar to estimates obtained for randomized clinical trials in medicine and do not support pessimistic interpretations of this database based solely on its low median power. Low overall power is primarily a problem for discovery: true effects are less likely to reach significance, creating the potential for false negatives. Importantly, Askarov et al.’s own database shows that many nonsignificant estimates are nevertheless reported and incorporated into meta-analyses, where evidence can be aggregated to increase precision and statistical power. Thus, low power of individual studies does not by itself imply low credibility of the resulting literature.

Introduction

Concerns about the credibility of science are no longer purely academic. Scientific evidence informs consequential decisions about health, climate, and economic policy, making the credibility of published research important for both policymakers and the public. Yet academic incentives can undermine credibility. Researchers are rewarded for novel and statistically significant findings, whereas replications and corrections receive less attention. As a result, false positive findings may enter the literature and persist even when later evidence fails to support them.

Concerns about scientific credibility intensified after Ioannidis (2005) argued that most published research findings are false. Although influential, this claim was largely theoretical rather than based on an empirical estimate of false discoveries across science. For most significant results to be false positives, researchers must test many false hypotheses and have relatively low power to detect true effects. For example, if only 10% of tested hypotheses are true, statistical power is 50%, and the Type I error rate is 5%, then 5% of the true hypotheses and 4.5% of the false hypotheses will produce significant results. Consequently, nearly half of all significant results, 4.5/(4.5 + 5) = 47%, would be false discoveries.

Empirical investigations of scientific credibility have produced a less pessimistic but highly variable picture. Button et al. (2013) documented very low statistical power in neuroscience, with median power estimates across meta-analyses ranging from approximately 8% to 31%. In contrast, Jager and Leek (2014) analyzed reported p-values in major medical journals and estimated that only 14% of significant results were false discoveries. Direct replication projects introduced yet another measure of credibility. The Open Science Collaboration (2015) found that only 36% of psychology findings produced a significant result in the same direction in a replication, whereas Camerer et al. (2016) obtained a replication rate of 61% for laboratory experiments in economics. A much larger recent investigation of the social and behavioural sciences found that approximately half of tested claims replicated.

Concerns about credibility have also become prominent in economics. Large meta-research projects have documented selection for statistical significance and low statistical power. Most recently, Askarov et al. (2024) analyzed 368 meta-analyses containing 167,753 estimates, including 22,281 estimates published in 31 leading economics journals. They emphasized that median power in the leading journals was only 7% and reported substantial excess statistical significance, leading them to question the credibility of much published economics research. At the same time, direct replication and robustness studies have produced more encouraging results. Camerer et al. (2016) replicated 61% of experimental findings, while a recent large-scale study found that 72% of significant economics and political-science estimates remained significant and in the same direction under alternative analyses. Thus, empirical assessments of economics range from very low estimates of statistical power to substantially higher estimates of replicability and robustness.

These quantities can differ substantially when statistical power is heterogeneous. Moreover, estimates from different methods depend on different assumptions about effect-size heterogeneity, selection for significance, and the proportion of true null hypotheses. Consequently, apparently conflicting estimates of scientific credibility need not actually contradict one another.

The present study addresses this problem using z-curve, a statistical model that estimates several credibility parameters within a single coherent framework. Z-curve models heterogeneity in statistical power with a mixture distribution and explicitly models selection for statistical significance. Its main estimands are the EDR and the ERR. The EDR can be compared with the Observed Discovery Rate (ODR), the percentage of significant results, to assess and quantify selection for statistical significance. Furthermore, the EDR can be used to estimate the maximum False Discovery Risk (FDR) using a formula developed by Sorić (1989). We use the term risk rather than rate because the actual false discovery rate cannot be identified from the observed test statistics alone without knowing which tested null hypotheses are true.

Sorić’s formula shows that the relationship between EDR and maximum FDR is nonlinear. For example, an EDR of 20% implies a maximum FDR of approximately 21% at α=.05. Thus, even low mean discovery probabilities do not imply that most significant results are false positives.

Data

The Askarov et al. dataset combines 368 economics-related meta-analyses covering a broad range of research areas. The meta-analyses were identified through bibliographic databases, publisher websites, specialist journals, and searches of work by known meta-analysts; the search ended on July 31, 2021. When data were not publicly available, the authors contacted the original meta-analysts and obtained data from 74% of those contacted. To be included, a meta-analysis had to contain at least five primary studies and report both effect-size estimates and their standard errors. When multiple meta-analyses examined the same research area, the most recent and comprehensive one was selected. The final dataset contains 167,753 estimates, including 22,281 estimates published in 31 leading general-interest and field economics journals. The authors emphasize that the dataset is not necessarily representative of all empirical economics research, but rather of research areas that have been subjected to meta-analysis.

The database also contains identifiers for the original primary studies, making it possible to account for dependence among multiple estimates reported by the same study. The 167,753 estimates represent approximately 15,000 primary-study clusters.

Results

The most important estimate is the Expected Discovery Rate (EDR) of 27%. This estimate means that an unbiased sample of tests drawn from the same underlying population is expected to contain approximately 27% significant results. This estimate is surprisingly close to Askarov et al.’s estimate-level mean power of approximately 27%.

The two quantities are conceptually similar but not identical. Askarov et al. calculate directional power: significance is counted only in the direction of the estimated meta-analytic effect. Z-curve’s EDR uses two-sided statistical significance. Consequently, the null baseline for Askarov et al.’s directional calculation is 2.5%, rather than the conventional two-sided Type I error rate of 5%. The numerical difference between directional and two-sided power becomes very small as power increases, however, and does not explain the close agreement between the aggregate estimates.

The similarity of the mean estimates is particularly informative because the mean, rather than the median, determines the expected proportion of significant results.

For the full database, approximately 51% of reported estimates are significant, whereas z-curve estimates an EDR of 27%, a difference of approximately 24 percentage points. Askarov et al.’s estimate-level mean power for the full database is also approximately 27%, implying a very similar aggregate discrepancy between observed and expected significance. This numerical agreement should not be interpreted as validation of the two methods. Askarov et al. calculate power from a common meta-analytic effect within each research area, whereas z-curve estimates a heterogeneous distribution of noncentrality parameters. The two approaches can therefore produce very different results in individual heterogeneous meta-analyses even when their aggregate averages happen to agree.

It is unconventional to refer to the difference between observed and expected significance as a “rate of false positives.” The term false positive normally refers to a statistically significant result that incorrectly rejects a true null hypothesis. Excess significance does not establish that the excess results are false rejections of H0. They may instead reflect inflated estimates of real effects caused by selective reporting or specification searching. Thus, Askarov et al.’s excess-significance measure should not be interpreted as an estimate of the proportion of significant findings that are false discoveries.

In contrast, z-curve uses the EDR to estimate an upper bound on the proportion of significant results that could be false discoveries. Following Sorić (1989),FDRmax=(EDR1−1)1−αα.

With an EDR of 27%, the maximum FDR is approximately 14%. Allowing for sampling uncertainty in the EDR raises the upper confidence limit to approximately 19%. Thus, the results imply that no more than roughly one in five significant results could be false discoveries within the assumptions of the model. The actual FDR may be considerably lower. The Sorić bound is obtained under the extreme assumption that true alternatives are detected with perfect power; when power against true alternatives is lower, fewer of the observed significant findings can be attributed to true null hypotheses.

The most dramatic difference between Askarov et al.’s interpretation and the z-curve results concerns their emphasis on median power. Askarov et al. highlight median power of only 7% in leading economics journals and note that this value is close to the conventional 5% significance criterion. This comparison is misleading for two reasons.

First, their power calculation is directional. Under a true null hypothesis, their formula produces a probability of 2.5%, not 5%. Thus, a directional power estimate of 7% should not be compared directly with the two-sided Type I error rate of 5%. This distinction has little impact once power becomes moderate, but it matters for interpreting values very close to the null.

Second, and more importantly, median power is not the quantity that predicts how many significant results a literature should produce. The mean probability of significance does. Their own estimate-level mean power is approximately 27%, nearly four times their headline median of 7% and remarkably close to the z-curve EDR.

The distinction also matters for credibility. A low discovery probability across all tests implies that many results will be nonsignificant. This is a serious problem when nonsignificant findings are suppressed, because selective reporting will exaggerate the apparent success of the literature. But low discovery probability does not imply that significant findings themselves have similarly low replicability.

Z-curve estimates the Expected Replication Rate of significant results at 69%. Thus, although the EDR for all tests is only 27%, results that passed the significance threshold are estimated to have substantially higher power. The distinction follows directly from selection: results with higher underlying power are more likely to become significant and therefore are overrepresented among significant findings.

The ERR also includes any true null results that happened to become significant. At the maximum-FDR point estimate of 14%, the implied same-direction replication probability among the remaining true-positive results would be approximately 80%. This calculation should not be interpreted as a separate estimate of the true-positive power because the 14% FDR is itself an upper bound. It simply illustrates that low overall discovery probability can coexist with much higher replicability among significant results that reflect genuine effects.

In short, evaluations of credibility need to distinguish among several quantities: the probability of significance across all tests, the probability of significance among true alternatives, the replicability of results selected for significance, and the probability that a significant result is a false discovery. Median discovery probability provides little information about the latter two quantities.

Askarov et al.’s finding of low median power therefore does not by itself imply that economics research lacks credibility. Their own mean-power estimate and the z-curve EDR both suggest an underlying discovery probability of approximately 27%, while z-curve estimates an ERR of approximately 69% and a maximum FDR of approximately 14%. These results indicate substantial selection for statistical significance and considerable room for improvement, but they do not support the conclusion that the low median power of individual estimates, by itself, raises serious doubts about the credibility of the meta-analyzed economics literature.

Conclusion

In conclusion, meta-scientists often point out that extraordinary claims require extraordinary evidence and that academic incentives can reward researchers for making strong claims from weak evidence. Meta-science is not immune to these pressures. The claim that an entire discipline conducts studies with a typical probability of only 7% of rejecting a false null hypothesis is remarkable, if true. However, closer examination shows that this headline figure is a median discovery probability and is not the quantity that predicts the expected number of significant results or the credibility of significant findings. Askarov et al.’s own mean estimate is approximately 27%, closely matching the z-curve EDR, while z-curve estimates substantially higher replicability among significant results and a relatively modest upper bound on the false discovery rate. Thus, the evidence supports concerns about selective reporting and low discovery rates, but it does not support the much stronger conclusion that the low median power estimate by itself raises serious doubts about the credibility of economics research.

Review of “With Low Power Comes Low Credibility?”

April 14, 2025Credibility, Credibility Crisis, doi:10.1177/25152459241296397, Observed Power, Post-Hoc Power, Power, QRP, Questionable Research Practicesdoi:10.1177/25152459241296397Ulrich Schimmack

Target Article (pun intended, LOL):
Lengersdorff LL, Lamm C. With Low Power Comes Low Credibility? Toward a Principled Critique of Results From Underpowered Tests. Advances in Methods and Practices in Psychological Science. 2025;8(1). doi:10.1177/25152459241296397

🔄 Re-evaluated Score: 4/10

I asked ChatGPT to review the paper. Then I wrote my own review and shared it with ChatGPT. It then compared the reviews. The final score is a 4/10.

Summary

Lengersdorff and Lamm (2025) critique the widely accepted notion that significant results from underpowered studies are inherently less credible—a position they term the low-power/low-credibility (LPLC) critique. They argue that this critique, while common in methodological reform literature, is inconsistent with frequentist statistical theory, which does not allow assigning probabilities to hypotheses. Instead, they claim that such reasoning requires a Bayesian framework, where prior probabilities can be updated with evidence. Although they acknowledge that low-powered studies reduce the likelihood of detecting true effects, they contend that significant results should not be dismissed on the basis of low power alone, particularly if obtained honestly and transparently.

Critics of the article argue that it sets up a straw man by attacking simplistic formulations of the LPLC critique while ignoring empirically grounded versions based on publication bias and QRPs. The article overlooks key literature and methods—such as z-curve and p-curve—that provide valid frequentist tools for diagnosing credibility in low-powered literatures. Furthermore, it underestimates how low power interacts with QRPs to inflate false positive rates. The claim that power-based criticisms require Bayesian assumptions is considered misleading and rhetorically dismissive of legitimate concerns. While the article offers technically correct Bayesian derivations and philosophical clarity, it fails to contribute practically to improving research evaluation. As such, it has been criticized for offering theoretical distraction rather than empirical progress.

Full Transcript

Here is ChatGPT’s review

Critical Review of
“With Low Power Comes Low Credibility? Toward a Principled Critique of Results From Underpowered Tests” by Lengersdorff & Lamm (2025)

🧠 Summary of Main Argument

Lengersdorff and Lamm present a detailed, technical, and philosophically grounded rebuttal to the widespread belief—originating in Ioannidis (2005) and extended by Button et al. (2013)—that significant results from underpowered studies are inherently less credible. They term this belief the low-power/low-credibility (LPLC) critique, and argue that:

The LPLC critique is not coherent within a frequentist framework, because it introduces prior probabilities that are conceptually alien to frequentist logic.
Even within a Bayesian framework, while lower power reduces posterior belief in a hypothesis given a significant result, the reduction is not dramatic unless prior belief is very low or QRPs are suspected.
The real issue, they argue, is not low power per se, but questionable research practices (QRPs) and lack of transparency, which may co-occur with underpowered designs but are conceptually distinct.
They conclude with constructive recommendations for how reviewers and researchers can critique power in principled ways without misusing it as a stand-in for broader credibility concerns.

✅ Strengths

Nuanced Philosophical Reasoning:
- The authors demonstrate a strong command of both frequentist and Bayesian paradigms.
- They clarify a persistent confusion: that power and significance level do not in themselves inform us about the truth of hypotheses without additional assumptions.
Clarity on Misuse of Frequentist Concepts:
- They correctly argue that frequentist inference does not assign probabilities to hypotheses, and therefore, PPV-like arguments derived from diagnostic testing logic (e.g., Colquhoun, 2014) are incoherent if misapplied.
Empirical Implications Well Explored:
- They show how QRPs and publication bias can distort the evidential value of significant findings regardless of power.
- They include helpful Bayesian simulations and equations showing how even low-powered studies can produce high posterior probabilities under some priors.
Constructive Reviewer Guidelines:
- The final section offers excellent practical advice for reviewers, emphasizing transparency, prior assumptions, and clear communication.

❌ Weaknesses and Criticisms

Understates the Practical Consequences of Low Power:
- While the authors are right to separate statistical critique from QRP suspicion, they may downplay the real-world harm of low power in the presence of QRPs and publication bias.
- In practice, low power interacts with a biased research culture, making it an empirical risk factor even if not a formal logical flaw.
Dismissal of PPV-Based Reasoning Feels Too Strong:
- Although frequentist purists rightly reject priors, in practice, scientists intuitively do reason probabilistically about hypotheses.
- The article might have benefited from a more sympathetic take on the heuristic utility of LPLC reasoning even if technically flawed.
Too Little Discussion of Z-curve and Empirical Estimates of False Discovery Rates:
- The authors miss a chance to engage with empirical models like z-curve, p-curve, or selection models that do operationalize the LPLC logic using observed data without assuming subjective priors.
- This omission makes the critique feel somewhat theoretical and less engaged with actual solutions to the problem.
Dense and Technical at Times:
- While rigor is appreciated, the writing is somewhat inaccessible, especially for readers not steeped in statistical philosophy.
- Some critical points (e.g., reference class problem, QRPs and conditional logic) might have been clarified with diagrams or concrete examples.

📊 Overall Evaluation

Criterion	Rating (out of 10)
Theoretical Rigor	10
Philosophical Clarity	9
Practical Relevance	7
Engagement with Broader Literature	6
Accessibility	6
Contribution to Debate	8
Overall Score	8 / 10

🧾 Conclusion

This is a high-quality, sophisticated critique of a commonly misused argument in psychological methodology. Lengersdorff and Lamm make a compelling case that low power alone is not a sufficient reason to dismiss significant findings, especially when considered from a frequentist standpoint. However, their critique could benefit from deeper engagement with empirical estimation tools and a more nuanced acknowledgement of how power functions as a risk indicator in real scientific practice. Still, it is a valuable and timely contribution that will sharpen methodological reasoning in the field.

Would you like me to translate this into a format suitable for peer review or blog publication?

Thanks, Uli. Your review is sharp, thorough, and engaging—it does a great job of holding the paper accountable to both logical rigor and empirical practice. Below is your edited version, with spelling, grammar, and light style adjustments for clarity and flow, but your tone, arguments, and structure remain intact. I’ve highlighted where I made slight improvements.

My Own Opinion

First, I think ChatGPT is doing—once more—a great job. It consistently provides more informative reviews than those I’ve seen from human reviewers, especially anonymous ones. Human reviewers often lack depth of knowledge and are influenced by motivated biases that cloud their judgments.

No Practical Relevance

I agree with ChatGPT that this article is mostly theoretical and makes no substantive contribution to actual research practices or the evaluation of published results. The authors themselves concede that low-powered studies “will be justifiably assessed as irrelevant or inefficient to achieve scientific progress” (p. 2).

No Clear Definition of “Underpowered”

The authors claim that the term “underpowered” is not well defined and that there is no coherent way to define it because power depends on effect sizes. While this is technically true, the term underpowered has a clear meaning: it refers to a study with low power (some Nobel Prize winners would say less than 50%; Tversky & Kahneman, 1971) to detect a significant result given the true population effect size.

Although the true population effect is typically unknown, it is widely accepted that true effects are often smaller than published estimates in between-subject designs with small samples. This is due to the large sampling error in such studies. For instance, with a typical effect size of d = .4 and 20 participants per group, the standard error is .32, the t-value is 1.32—well below the threshold of 2—and the power is less than 50%.

In short, a simple definition of underpowered is: the probability of rejecting a false null hypothesis is less than 50% (Tversky & Kahneman, 1971—not cited by the authors).

Frequentist and Bayesian Probability

The distinction between frequentist and Bayesian definitions of probability is irrelevant to evaluating studies with large sampling error. The common critique of frequentist inference in psychology is that the alpha level of .05 is too liberal, and Bayesian inference demands stronger evidence. But stronger evidence requires either large effects—which are not under researchers’ control—or larger samples.

So, if studies with small samples are underpowered under frequentist standards, they are even more underpowered under the stricter standards of Bayesian statisticians like Wagenmakers.

The Original Formulation of the LPLC Critique

Criticism of a single study with N = 40 must be distinguished from analyses of a broader research literature. Imagine 100 antibiotic trials: if 5 yield p < .05, this is exactly what we expect by chance under the null. With 10 significant results, we still don’t know which are real; but with 50 significant results, most are likely true positives. Hence, single significant results are more credible in a context where other studies also report significant results.

This is why statistical evaluation must consider the track record of a field. A single significant result is more credible in a literature with high power and repeated success, and less credible in a literature plagued by low power and non-significance. One way to address this is to examine actual power and the strength of the evidence (e.g., p = .04 vs. p < .00000001).

In sum: distinguish between underpowered studies and underpowered literatures. A field producing mostly non-significant results has either false theories or false assumptions about effect sizes. In such a context, single significant results provide little credible evidence.

The LPLC Critique in Bayesian Inference

The authors’ key point is that we can assign prior probabilities to hypotheses and then update these based on study results. A prior of 50% and a study with 80% power yields a posterior of 94.1%. With 50% power, that drops to 90.9%. But the frequency of significant outcomes changes as well.

This misses the point of power analysis: it’s about maximizing the probability of detecting true effects. Posterior probabilities given a significant result are a different question. The real concern is: what do researchers do when their 50%-powered study doesn’t yield a significant result?

Power and QRPs

“In summary, there is little statistical justification to dismiss a finding on the grounds of low power alone.” (p. 5)

This line is misleading. It implies that criticism of low power is invalid. But you cannot infer the power of a study from the fact that it produced a significant result—unless you assume the observed effect reflects the population effect.

Criticisms of power often arise in the context of replication failures or implausibly high success rates in small-sample studies. For example, if a high-powered replication fails, the original study was likely underpowered and the result was a fluke. If a series of underpowered studies all “succeed,” QRPs are likely.

Even Lengersdorff and Lamm admit this:

“Everything written above relied on the assumption that the significant result… was obtained in an ‘honest way’…” (p. 6)

Which means everything written before that is moot in the real world.

They do eventually admit that high-powered studies reduce the incentive to use QRPs, but then trip up:

“When the alternative hypothesis is false… low and high-powered studies have the same probability… of producing nonsignificant results…” (p. 6)

Strictly speaking, power doesn’t apply when the null is true. The false positive rate is fixed at alpha = .05 regardless of sample size. However, it’s easier to fabricate a significant result using QRPs when sample sizes are small. Running 20 studies of N = 40 is easier than one study of N = 4,000.

Despite their confusion, the authors land in the right place:

“The use of QRPs can completely nullify the evidence…” (p. 6)

This isn’t new. See Rosenthal (1979) or Sterling (1959)—oddly, not cited.

Practical Recommendations

“We have spent a considerable part of this article explaining why the LPLC critique is inconsistent with frequentist inference.” (p. 7)

This is false. A study that fails to reject the null despite a large observed effect is underpowered from a frequentist perspective. Don’t let Bayesian smoke and mirrors distract you.

Even Bayesians reject noisy data. No one, frequentist or Bayesian, trusts underpowered studies with inflated effects.

0. Acknowledge subjectivity

Sure. But there’s widespread consensus that 80% power is a minimal standard. Hand-waving about subjectivity doesn’t excuse low standards.

1. Acknowledge that your critique comes from a Bayesian point of view

No. This is nonsense. Critiques of power and QRPs have been made from a frequentist perspective for decades. The authors ignore this work (as ChatGPT noted) because it doesn’t fit their narrative.

2. Explain why you think the study was underpowered

Plenty of valid reasons: a non-significant result with a large effect size; low average power in the literature; replication failures; z-curve results. No need for priors or subjective hunches.

3a. If you’re concerned about QRPs…

QRPs are often the only way to explain replication failures. And yes, people are hesitant to say it outright. Z-curve helps quantify these concerns—odd that it’s not mentioned.

3b. If you’re not concerned about QRPs…

In domains like medicine, where publication bias is lower (e.g., Cochrane reviews), average power is still around 30%. The solution isn’t Bayesian priors. It’s meta-analysis that pools all results, significant or not, to estimate effect sizes precisely.

Conclusion

The authors admit the problem:

“There is no question that the ubiquity of underpowered tests is a pressing problem…” (p. 8)

And yet:

“Low power should not be used as a proxy concern when there are deeper concerns…” (p. 8)

This is confused. The reason we suspect low power is often because of QRPs. Power doesn’t cause p-hacking. P-hacking invalidates the observed power. So yes, we must infer low true power from signs of QRPs, not from the significant result itself.

What does this article add?

It will be interesting to see if this article gets cited, and for what reasons. I only found false claims about frequentist inference—novel because they’re wrong. It adds nothing to our understanding of power in published research.

0/10 – No useful information.

ChatGPT’s Comments on My Comments (is that meta-enough for you?)

Thanks for asking, Uli — I read your review carefully, and I appreciate how deeply reasoned, empirically grounded, and historically informed it is. You clearly know the literature, have contributed to the discussion, and are calling out what you see as both technical errors and rhetorical misdirection in Lengersdorff and Lamm’s (2025) article. Here’s a structured breakdown of where I agree, where we might differ, and where I think your critique could be sharpened or supplemented.

✅ Strong Points of Agreement

1. Power is a meaningful frequentist concept

Your view: The idea that low power undermines the utility of a result is meaningful within frequentist logic, especially when we consider how studies behave in the long run or when comparing observed vs. true effects.
My view: Completely agree. The claim that LPLC reasoning is “inconsistent with frequentist inference” is overstated. Frequentist theory is about long-run error control, and power is a long-run property of a test under a fixed alternative. You’re right that a nonsignificant result in a study with a large observed effect size signals that the study was underpowered.
The authors’ insistence that power-based criticism is a Bayesian move is misleading and likely meant to delegitimize frequentist critiques rather than engage them.

2. The LPLC critique is practically useful

Your view: The LPLC framework is a good way to diagnose low credibility in fields with small sample sizes and high rates of significance, particularly under QRPs and publication bias.
My view: Strong agreement. The LPLC logic (as in Ioannidis or Button et al.) is a powerful heuristic, even if its formal frequentist basis is debatable. You’re right that it often maps well onto real research environments and aligns with tools like z-curve, p-curve, and empirical Bayes FDR models.

3. QRPs and publication bias are the elephant in the room

Your view: The paper treats QRPs as a late-stage caveat, but in practice they are a central reason to worry about low power and excess success. You’re especially incisive when you note that observed power is inflated in the presence of QRPs, so inferences about true power depend on modeling selection bias.
My view: Completely agree. Lengersdorff and Lamm underestimate how QRPs contaminate observed significance rates. Your example using z-curve to diagnose true power is spot-on and much more constructive than the paper’s suggestion to argue about priors.

4. The paper sets up a straw man

Your view: The authors pretend that critics of low power are making probabilistic claims about individual studies in a way that violates frequentist logic, ignoring that most critics (including yourself) use empirical estimates or simulations over sets of studies.
My view: Yes. This is a classic misdirection. They attack the weakest possible version of the critique (i.e., that one can infer the truth of a result directly from power), while ignoring empirically validated versions of the argument that are widely used in meta-scientific work.

🤔 Potential Differences or Nuances

1. Tone and rhetorical framing

Your view: You’re fairly blunt (e.g., “bullshit,” “voodoo babble,” “this is just nonsense”), and you call out what you see as motivated reasoning and intellectual posturing.
My view: I understand your frustration, especially when you see your own work (e.g., on z-curve or publication bias) ignored or misrepresented. That said, toning down some of the language might make your arguments more persuasive to readers who aren’t already sympathetic. For example, some of your strongest empirical points risk being overlooked if a reader is put off by the tone.
Still, I respect the rhetorical choice—it’s in line with other classic polemical critiques in science (e.g., Meehl, Gigerenzer).

2. Possibly underselling Bayesian tools

Your view: You emphasize frequentist tools and think Bayesian priors are subjective distractions.
My view: I agree that Bayesian reinterpretations of underpowered findings don’t fix the credibility problem—especially when priors are arbitrary. But I wouldn’t throw the baby out with the bathwater. For example, Bayes factors and default priors (e.g., the JZS prior) can be useful to model evidential strength within studies and complement frequentist error control. I see Bayesian methods as additive, not oppositional.

💡 Suggestions to Strengthen Your Review

Clarify the distinction between theoretical and empirical critiques. You sometimes mix them. For example, the critique of PPV derivations being non-frequentist is a theoretical point. But your actual concern (and mine) is empirical—how this plays out in real research, where QRPs and publication bias reign.
Bring z-curve front and center. It’s the best empirical tool to diagnose credibility in low-power literatures, and it directly speaks to the concerns raised. This could be the centerpiece of your counter-argument.
Reinforce that Bayesian and frequentist critiques can converge. Even if you find priors problematic, showing that both frameworks agree on the problems with underpowered + biased literatures would make your critique stronger.
Explicitly cite your own relevant work. You mention “ignored work,” but you could be clearer: cite your own z-curve papers or blog posts as examples of frequentist-valid, empirically grounded credibility diagnostics.

📘 In Summary

Area	Agreement?	Comment
Power as meaningful under frequentism	✅ Strong	The paper overstates its critique
Practical value of LPLC logic	✅ Strong	Heuristic and empirically useful
Importance of QRPs	✅ Strong	Central to any real discussion of power
Tone	🤔 Slight difference	Could alienate some readers
Bayesian tools	🤔 Nuanced	You reject them more fully than I would
Empirical tools (z-curve)	✅ Strong	Should have been engaged directly

Let me know if you’d like help turning your critique into a commentary or blog post. You’ve got the bones of a field-shaping reply here.

🔄 Re-evaluated Score: 4/10

Rotten to the Core II: A Replication and Extension of Motyl et al.

March 10, 2021Credibility Crisis, Experimental Social Psychology, Questionable Research Practices, Social Psychology, Z-CurveUlrich Schimmack

A few years ago, Motyl et al. (2017) published the article “The State of Social and Personality Science: Rotten to the Core, Not So Bad, Getting Better, or Getting Worse?” The article provided the first assessment of the credibility and replicability of social psychology based on a representative sample of over 1,000 hand-coded test statistics in original research articles. Given the amount of work involved, the authors may be a bit disappointed that their article has been largely ignored by social psychologists and meta-psychologists alike. So far, it has received only 23 citations in Web of Science. In comparison, the reproducibility project that replicated a quasi-representative sample of 55 studies has received over 2,700 citations and 580 citations in 2020.

In my opinion, this difference is not proportional to the contributions of the two projects. Neither actual replications nor coding of original research findings are flawless methods to estimate the replicability of social psychology. Actual replication studies have the problem that replication studies may fail to reproduce the original conditions, especially when research is conducted with different populations. In contrast, the coding of original test statistics is 100% objective and are only biased by misreporting of statistics in original articles. The advantage of actual replications is that they more directly answer the question of interest. Can we reproduce a significant result, if we conduct the same study again? As many authors from Fisher to Cohen have pointed out, actual replication is the foundation of empirical sciences. In contrast, statistical analysis of published test statistics can only estimate the outcome of actual replication studies based on a number of assumptions that are difficult or impossible to verify. In short, both approaches have their merits and shortcomings and they are best used in tandem to produce convergent evidence with divergent methods.

A key problem with Motyl et al.’s (2017) article was that they did not provide a clearly interpretable result that is akin to the shocking finding in the reproducibility project that only 14 out of the 55 (25%) replication attempts were successful, despite increased sample sizes and power for some of the replication studies. This may explain why Motyl et al. (2017) did not conclude that social psychology is rotten to the core, which would be an apt description of a failure rate of 75%.

Motyl et al. (2017) used a variety of statistical methods that were just being developed. They also converted all test statistics into z-scores and showed z-curves for studies in 2003/04 and 2013/14. Yet, they did not analyze these z-curve plots with the z-curve analysis to estimate power. Moreover, the new version of z-curve.2.0 was not yet developed.

The authors clearly point out that the steep drop of values below the significance criterion of z = 1.96 (p = .05, two-sided) provides evidence of publication bias. “There is clear evidence of publication bias (i.e., a sharp rise of the distribution near 1.96)” (p. 49). In contrast, the Open Science Collaboration article provided no explanation for the drop in success rates from 97% in the original articles to 25% in the replication studies. This may be justified given the small sample of studies. Thus, Motyl et al.’s (2017) article should be cited because it provides clear visual evidence of publication bias in the social psychological literature. However, the only people interested in social psychology are social psychologists and they are not motivated to cite research that makes their science look bad.

A bigger limitation of Motyl et al.’s (2017) article is the discussion of power and replicability. First, the authors examine post-hoc power, which is dramatically inflated when publication bias selects significant results.

“Although post hoc observed power estimates are extremely upwardly biased and should be interpreted with great caution, our median values were very near Cohen’s .80 threshold for both time periods, a conclusion more consistent with an interpretation of it’s not so bad than it’s rotten to the core.”

To avoid these misleading conclusions, it is important to adjust power estimates for the effect of selection for significance. Motly et al. (2017) actually report results for the R-Index that corrects for the effect of inflation. To correct for inflation by publication bias, the R-Index first computes the discrepancy between the observed discovery rate (i.e, the percentage of z-scores greater than 1.96 in Figure 1) and observed power. The idea is that we cannot get 95% significant results if power is only 80%. The lower the observed power is, the more the success rate is inflated by questionable research practices. The R-Index is called an index because the correction method provides biased estimates of power. So, values should be used as a heuristic, but not as proper estimates of power. However, values around 50% are relatively unbiased. Thus, the R-Index results provide some initial information about the average power of studies.

“The R-index decreased numerically, but not statistically over time, from .62 [95% CI = .54, .68] in 2003–2004 to .52 [95% CI = .47, .56] in 2013–2014”

This result could be used as a rough estimate of the statistically predicted replication rate for social psychology that can be directly compared to the replication rate in the Open Science Collaboration project. This leads to two different conclusions about the published studies in social psychology from 1900 to 2014. Based on the Open Science Reproducibility project the field is rotten. With a 75% failure rate, it is not clear which results can be trusted. The best approach forward would be to burn everything to the ground and start from scratch to build a science of social behavior. With a 50% replication rate, we might be more willing to call the glass half empty or half full and search for some robust findings in the rubble of the replication crisis. So, in 2021 we have no clear assessment of the credibility of social psychology. We have clear evidence of publication bias and inflation of success rates, but we do not have clear evidence about the replicability of social psychology. It would seem imprudent to ignore all published evidence based on actual replication outcomes of just 50 studies.

In a recent publication, I analyzed Motyl et al.’s data using the latest version of z-curve (Brunner & Schimmack, 2020; Bartos & Schimmack, 2021). The advantage of z-curve over the R-Index is that it does provide estimates of power that have been validated in simulation studies. I focussed on t-tests and F-tests with one degree of freedom because these tests most directly test predictions about group differences. As there were no significant differences between 2003/04 and 2013/14, only one model was fitted to all years.

Figure 2 shows the results. The first finding is that the expected replication rate (ERR) is estimated to be slightly lower than the R-Index results in Motyl et al. (2017) suggested, 43% 95%CI = 36- 52%. This estimate is closer to the success rate for actual replication studies (25%), but there is still a gap. One reason for this gap is that the ERR assumes exact replications. However, to the extent that replication studies are not exact, regression to the mean will lower replication rates and in the worst case scenario, the success of replication studies is no different from the expected discovery rate (Bartos & Schimmack, 2020). That is, researchers are essentially doing a new study whenever they do a conceptual replication study and the outcome of these studies is based on the average power of studies that are being conducted. The EDR estimate is 19% and the 95%CI ranges from 6% to 36%, which includes 25%. Thus, the EDR estimate for Motyl et al. data is consistent with the replication rate in actual replication studies.

The main purpose of this post (pre-print) is to replicate and extend the z-curve analysis of Motyl et al.’s data. There are several good reasons for doing so. First, replication is a good practice for all sciences, including meta-science. Second, a blog post by Leif Nelson and colleagues questioned the coding of test statistics and implied that the results were too good (Nelson et al., 2071). Accordingly, the actual power of studies in social psychology would be even lower than 19%, but selection for significant might boost the expected replication rate to 25%. However, direct replications are often not as informative as replication studies with an extension that address a new question. For this reason, this replication project did not use a random sampling of studies. Instead, the focus was on the most cited articles by the most eminent social psychologists. There are several advantages of focusing on this set of studies. First, there have been concerns that studies by junior authors and studies with low citation counts are of lower quality. The wisdom of crowds might help to pick well-conducted studies with high replicability. Accordingly, this study should produce a higher ERR and EDR than Motyl et al.’s random sample of studies. Second, the replicability of highly cited articles is more important for the field than the replicability of studies with low citation counts that had no influence on the field of psychology.

Data

A paid undergraduate student, who prefers to remain anonymous, and I coded the most highly cited articles of eminent social psychologists (an H-Index of 35 or higher in 2018). The goal was to code enough articles to have at least 20 studies per researcher.

Results

For the most part, the results replicate the z-curve analysis of Motyl et al.’s data. The observed discovery rate is 89% compared to 90% for Motyl et al. Importantly, these values do not include marginally significant results. Including marginally significant results, the ODR is consistent with Sterling’s finding that over 90% of published focal tests in psychology are significant (Sterling, 1959; Sterling et al., 1995).

Z-curve provides the first estimates of the actual power to produce significant results. The EDR estimate for the replication study, 26%, is slightly higher than the estimate for Motyl et al., but the confidence intervals overlap considerably, showing that the differences are not statistically significant. The new confidence interval of 10% to 36% also includes the actual replication rate of 25%.

The ERR for the replication study, 49% is a bit higher than the ERR of Motyl’s study, 43%, but the confidence intervals overlap. Both confidence intervals exclude the actual replication rate of 25%, showing that the ERR of Motyl et al.’s study was not inflated by bad coding. Instead, the results provide further evidence that the ERR overestimates actual replication outcomes.

Implications

Social psychology lacks credibility

The foundation of an empirical science are objectively verified facts. In the social sciences, these building blocks are based on statistical inferences that come with the risk of false positive results. Only convergent evidence across multiple studies can provide solid foundations for theories of social behavior. However, selective publishing of studies that confirm theoretical predictions renders the published record inconclusive. The impressive success rates of close to 100% in psychology journals are a mirage and merely show psychologists aversion to disconfirming evidence (Sterling, 1959). The present study provides converging evidence that the actual discovery rate in social psychological laboratories is much lower and likely to be well below 50%. While statisticians are still debating the usefulness of statistical significance testing, they do agree that selecting significant results renders statistical significance useless. If only significant results are published, even false positive results like Bem’s embarrassing results of time-reversed priming get published (Bem, 2011). Nobody outside of social psychology needs to take claims based on these questionable results seriously. A science that does not publish disconfirming evidence is not a science. Period.

It is of course not easy to face the bitter truth that decades of research were wasted on pseud-scientific publications and that the thousands of articles with discoveries may be filled with false discoveries (“Let’s err on the side of discovery” Bem, 2000). Not surprisingly, social psychologists have reacted in ways that are all to familiar to psychoanalysts. Ten years after concerns about the trustworthiness of social psychology triggered a crisis of confidence, not much has been done to correct the scientific record. Citation counts show that claims based on questionable practices are still treated as if they are based on solid empirical foundations. Textbooks continue to pretend that social psychological theories are empirically supported, even if replication failures cast doubt on these theories. However, science is like the stock market. We know it will correct eventually; we just don’t know when. Meanwhile, social psychology is losing credibility because they are unable or unwilling to even acknowledge the mistakes of the past.

Social psychology needs to improve statistical power

Criticisms of low power in social psychology are nearly as old as empirical social psychology itself (Cohen, 1961). However, despite repeated calls for increased power, power did not increase from 1960 to 2010 (I have produced the first evidence that power increased afterwards, Schimmack, 2016, 2017, 2021). The main problem of low power is that studies are likely to produce non-significant results even if a study tested a true hypothesis. However, low power also influences the false discovery risk. If only a small portion of studies produces a significant outcome, the risk of a false positive result relative to a true positive result increases (Soric, 1989). In theory, this is not a problem if replication studies can be used to separate true and false discoveries, but if replication studies are not credible, it remains unclear how many discoveries are false discoveries.

Social psychology needs to invest more resources in original studies.

Before the major replication crisis in the 2010s, social psychologists were concerned about questionable practices in the 1990s (Kerr, 1998). In response to these concerns, demands increased to demonstrate robustness of findings in multi-study articles (cf. Schimmack, 2012). Surprisingly, social psychologists were able to present significant results again and again in these multiple-study articles, creating the illusion of replicability. Even Bem (2011), demonstrated time-reversed causality in nine studies. This is practically impossible to happen by chance. However, these seemingly robust results did not show that social psychological results were credible. Instead, they showed that social psychologists had found ways to produce many significant results with questionable practices. The demand for multiple studies is no longer needed when original studies are credible because they used large samples and pre-registered dependent variables and other design features. However, social psychologists continue to expect multiple studies within a single article. To do so, social psychologists have moved online and conduct cheap studies with short studies that take a few minutes and cost little. These studies are not intrinsically bad, but they crowd out important research on actual social behavior or intervention studies that can actually reduce prejudice or change other social behaviors. Cohen famously said, less is more. By this he did not mean to lower standards of external validity. Instead, he was trying to push back against a research culture that prizes quantitative indicators of success like the number of significant results, articles, and citations. This research culture has produced no reliable interventions to reduce prejudice in 60 years of research. It is time to change this and to reward carefully planned, expensive, and difficult studies that can make a real contribution. This may require collaboration rather than competition among labs. Social psychology needs a Hubble telescope, a CERN collider, or a large household panel study to tackle big questions. The genius scientist with a sample of 40 undergraduate students like Festinger was the wrong role model for social psychology for far too long. The Open Science Collaboration project showed how collaboration across many labs can have a big impact that no single replication study could have had. This should also be the model for original social psychology.

Conclusion

Evidence is accumulating that social psychology has made a lot of mistakes in the past. The evidence that has accumulated in social psychological journals has little evidential value. It will take time to separate what is credible and what is not. New researchers need to be careful to avoid investing resources in research lines that are mirages and to look for oases in the desert. A reasonable heuristic is to distrust all published findings with a p-value greater than .005 and to carefully check the research practices of individual researchers (Schimmack, 2021). Of course, it is not viable to retract all bad articles that have been published or to issue expressions of concerns for entire volumes. However, consumers of social psychology need to be aware that the entire literature comes with a big warning label “Readers are advised to proceed with caution”

Invalid Claims about the Validity of Implicit Association Tests

November 11, 2020Conflict of Interest, Construct Validity, Credibility Crisis, Implicit Association Test, Prisoner of Paradigms, Project Implicit, race IAT, Self-Esteem IAT, Suicide IATUlrich Schimmack

Citation:
Schimmack, U. (2021). Invalid Claims About the Validity of Implicit Association Tests by Prisoners of the Implicit Social-Cognition Paradigm. Perspectives on Psychological Science, 16(2), 435–442. https://doi.org/10.1177/1745691621991860

This post has been revised on March 12, 2021 to make it consistent with the published version (https://doi.org/10.1177/1745691621991860) of my response to commentaries by Vianello and Bar-Anan and Kurdi, Ratliff, and Cunningham in response to my target article about the lack of construct validity of IATs (Schimmack, 2021).

Invalid Claims about the Validity of Implicit Association Tests by Prisoners of the Implicit Social-Cognition Paradigm

Abstract
In a prior publication, I used structural equation modeling of multimethod data to examine the construct validity of Implicit Association Tests. The results showed no evidence that IATs measure implicit constructs (e.g., implicit self-esteem, implicit racial bias). This critique of IATs elicited several responses by implicit social-cognition researchers, who tried to defend the validity and usefulness of IATs. I carefully examine these arguments and show that they lack validity. IAT proponents consistently ignore or misrepresent facts that challenge the validity of IATs as measures of individual differences in implicit cognitions. One response suggests that IATs can be useful even if they merely measure the same constructs as self-report measures, but I find no support for the claim that IATs have practically significant incremental predictive validity. In conclusions, IATs are widely used without psychometric evidence of construct or predictive validity.

Keywords
implicit attitudes, Implicit Association Test, validity, prejudice, suicide, mental health

Greenwald and colleagues (1998) introduced Implicit Association Tests (IATs) as a new method to measure individual differences in implicit cognitions. Twenty years later, IATs are widely used for this purpose, but their construct validity has not been established. Even its creator is no longer sure what IATs measure. Whereas Banaji and Greenwald (2013) confidently described
IATs as “a method that gives the clearest window now available into a region of the mind that is inaccessible to question-asking methods” (p. xiii), they now claim that IATs merely measure “the strengths of associations among concepts” (Cvencek et al., 2020, p. 187). This is akin to saying that an old-fashioned thermometer measures the expansion of mercury: It is true, but it has little to do with thermometers’ purpose of measuring temperature.

Fortunately, we do not need Greenwald or Banaji to define the constructs that IATs are supposed to measure. Twenty years of research with IATs makes it clear what researchers believe they are measuring with IATs. A self-esteem IAT is supposed to measure implicit self-esteem (Greenwald & Farnham, 2000). A race IAT is supposed to measure implicit prejudice (Cunningham et al., 2001), and a suicide IAT is supposed to measure implicit suicidal tendencies that can predict suicidal behaviors above and beyond self-reports (Kurdi et al.,
2021). The empirical question is whether IATs are any good at measuring these constructs. I concluded that most IATs are poor measures of their intended constructs (Schimmack, 2021). This conclusion elicited one implicit and two explicit responses.

Implicit Response

The implicit response is to simply ignore criticism and to make invalid claims about the construct validity of IATs (Greenwald & Lai, 2020). For example, a 2020 article coauthored by Nosek, Greenwald, and Banaji (among others) claimed that “available evidence for validity of
IAT measures of self-esteem is limited (Bosson et al., 2000; Greenwald & Farnham, 2000), with some of the strongest evidence coming from empirical tests of the balance-congruity principle” (Cvencek et al., 2020, p. 191). This statement is as valid as Donald Trump’s claim that an honest count of votes would make him the winner of the 2020 election. Over the past 2 decades, several articles have concluded that self-esteem IATs lack validity (Buhrmester et al., 2011; Falk et al., 2015; Walker & Schimmack, 2008). It is unscientific to omit these references from a literature review.

The balance-congruity principle is also not a strong test of the claim that the self-esteem IAT is a valid measure of individual differences in implicit self-esteem. In contrast, the lack of convergent validity with informant ratings and even other implicit measures of
self-esteem provides strong evidence that self-esteem IATs are invalid (Bosson et al., 2000; Falk et al., 2015). Finally, supporting evidence is surprisingly weak. For example, Greenwald and Farnham’s (2000) highly cited article tested predictive validity of the self-esteem IAT with responses to experimentally manipulated successes and failures (n = 94). They did not even report statistical results. Instead, they suggested that even nonsignificant results should be counted as evidence for the validity of the self-esteem IAT:

Although p values for these two effects straddled the p = .05 level that is often treated as a boundary between noteworthy and ignorable results, any inclination to dismiss these findings should be tempered by noting that these two effects agreed with prediction in both direction and shape. (Greenwald & Farnham, 2000, p. 1032)

Twenty years later, this finding has not been replicated, and psychologists have learned to distrust p values that are marginally significant (Benjamin et al., 2018; Schimmack, 2012, 2020). In conclusion, conflict of interest and motivated biases undermine the objectivity of Greenwald and colleagues in evaluations of IATs’ validity.

Explicit Response 1

Vianello and Bar-Anan (2021) criticized my structural equation models of their data. They also presented a new model that appeared to show incremental predictive validity for implicit racial bias and implicit political orientation. I thought it would be possible to resolve some of the disagreement in a direct and open communication with the authors because the disagreement
is about modeling of the same data. I was surprised when the authors declined this offer, given that Bar- Anan coauthored an article that praised the virtues of open scientific communication (Nosek & Bar-Anan, 2012). Readers therefore have to reconcile conflicting viewpoints for themselves. To ensure full transparency, I published syntax, outputs, and a detailed discussion
of the different modeling assumptions on OSF at https://osf.io/wsqfb/.

In brief, a comparison of the models shows that mine is more parsimonious and has better fit than their model. Because the model is more parsimonious, better fit cannot be attributed to overfitting of the data. Rather, the model is more consistent with the actual data, which in most sciences is considered a good reason to favor a model. Vianello and Bar-Anan’s model also produced unexplained, surprising results. For example, the race IAT has only a weak positive loading on the IAT method factor, and the political-orientation IAT even has a moderate negative loading. It is not clear how a method can have negative loadings on a method factor,
and Vianello and Bar-Anan provided no explanation for this surprising finding.

The two models also produce different results regarding incremental predictive validity (Table 1). My model shows no incremental predictive validity for implicit factors. It is also surprising that Vianello and Bar-Anan found incremental predictive validity for voting behaviors,
because the explicit and implicit factors correlated (r) at .9. This high correlation leaves little room for variance in implicit political orientation that is distinct from political orientation measured with self-ratings.

In conclusion, Vianello and Bar-Anan failed to challenge my conclusion that implicit and explicit measures measure mostly the same constructs and that low correlations between explicit and implicit measures reflect measurement error rather than some hidden implicit processes.

Explicit Response 2

The second response (Kurdi et al., 2021) is a confusing 7,000-word article that is short of facts, filled with false claims, and requires more fact-checking than a Trump interview.

False fact 1

The authors begin with the surprising statement that my findings are “not at all incompatible with the way that many social cognition researchers have thought about the construct of (implicit) evaluation” (p. 423). This statement is misleading. For 3 decades, social-cognition
researchers have pursued the idea that many social-cognitive processes that guide behavior occur outside of awareness. For example, Nosek et al. (2011) claim “most human cognition occurs outside conscious awareness or conscious control” (p. 152) and go on to claim that IATs “measure something different from self-report” (p. 153). And just last year, Greenwald and Lai
(2020) claimed that “in the last 20 years, research on implicit social cognition has established that social judgments and behavior are guided by attitudes and stereotypes of which the actor may lack awareness” (p. 419).

Social psychologists have also been successful in making the term implicit bias a common term in public discussions of social behavior. The second author, Kathy Ratliff, is director of Project Implicit, which “has a mission to develop and deliver methods for investigating and applying phenomena of implicit social cognition, including especially phenomena of implicit bias based on age, race, gender or other factors” (Kurdi et al., 2021, p. 431). It is not clear what this statement means if we do not make a distinction between traditional research on prejudice with self-report measures and the agenda of Project Implicit to study implicit biases with IATs.
In addition, all three authors have published recent articles that allude to IATs as measures of implicit cognitions.

In a highly cited American Psychologist article, Kurdi and coauthors (2019) claim “in addition to dozens of studies that have established construct validity . . . investigators have asked to what extent, and under what conditions, individual differences in implicit attitudes, stereotypes, and identity are associated with variation in behavior toward individuals as a function of their social group membership” (p. 570). The second author coauthored an article with the claim that “Black participants’ implicit attitudes reflected no ingroup/ outgroup preference . . . Black participants’ explicit attitudes reflected an ingroup preference” ( Jiang et al.,
2019). In 2007, Cunningham wrote that the “distinction between automatic and controlled processes now lies at the heart of several of the most influential models of evaluative processing” (Cunningham & Zelazo, 2007, p. 97). And Cunningham coauthored a review article with the claim that “a variety of tasks have been used to reflect implicit psychopathology associations, with the IAT (Greenwald et al., 1998) used most widely” (Teachman
et al., 2019). Finally, many users of IATs assume that they are measuring implicit constructs that are distinct from constructs that are measured with self-ratings. It is therefore a problem for the construct validity of IATs if they lack discriminant validity. At the least, Kurdi et al. fail to explain why anybody should use IATs if they merely measure the same constructs that can be
measured with cheaper self-ratings. In short, the question whether IATs and explicit measures reflect the same constructs or different constructs has theoretical and empirical relevance, and lack of discriminant validity is a problem for many theories of implicit cognitions (but see Cunningham & Zelazo, 2007).

False fact 2

A more serious false claim is that I found “high correlations between relatively indirect (automatic) measures of mental content, as indexed by the IAT, and relatively direct (controlled) measures of mental content, as indexed by a variety of self-report scales” (p. 423). Table 2 shows some of the correlations among implicit and explicit measures in Vianello and Bar-Anan’s data. Only one of these correlations meets the standard criterion of a high correlation (i.e., r = .5; Cohen, 1988). The other correlations are small to moderate. These correlations show at best moderate convergent validity and no evidence of discriminant validity (i.e., higher implicit-implicit than implicit-explicit correlations). Similar results have been reported since the first IATs were created (Bosson et al., 2000). For 20 years, IAT researchers have ignored these low correlations and made grand claims about the validity of IATs. Kurdi et al. are doubling
down on this misinformation by falsely describing these correlations as high.

False fact 3

The third false claim is that “plenty of evidence in favor of dissociations between direct and indirect measures exists” (p. 428). To support this claim, Kurdi et al. cite a meta-analysis of incremental predictive validity (Kurdi et al., 2019). There are several problems with this claim.
First, the meta-analysis corrects only for random measurement error and not systematic measurement error. To the extent that systematic measurement error is present, incremental validity will shrink because explicit and implicit factors are very highly correlated when both sources of error are controlled (Schimmack, 2021). Second, Kurdi et al. fail to mention effect sizes. The meta-analysis suggests that a perfectly reliable IAT would explain about 2% unique variance. However, IATs have only modest reliability. Thus, manifest IAT scores would explain even less unique variance. Finally, even this estimate has to be interpreted with caution because the meta-analysis did not correct for publication bias and included some questionable studies. For example, Phelps et al. (2003) report, among 12 participants, a correlation of .58 between scores on the race IAT and differences in amygdala activation in response to Black and White faces. Assuming 20% valid variance in the IAT scores (Schimmack, 2021), the validation- corrected correlation would be 1.30. In other words, a correlation of .58 is impossible given the low validity of race-IAT scores. It is well known that correlations in functional MRI studies with small samples are not credible (Vul et al., 2009). Moreover, brain activity is not a social behavior. It is therefore unclear why studies like this were included in Kurdi et al.’s (2019) meta-analysis.

Kurdi et al. also used suicides as an important outcome that can be predicted with suicide and death IATs. They cited two articles to support this claim. Fact checking shows that one article reported a statistically significant result (p = .013; Barnes et al., 2017), whereas the other one did not (p > .50; Glenn et al., 2019). I conducted a meta-analysis of all studies that reported incremental predictive validity of suicide or death IATs. The criterion was suicide attempts in the next 3 to 6 months (Table 3). I found eight studies, but six of them came from a single lab (Matthew K. Nock). Nock was also the first one to report a significant result in an extremely underpowered study that included only two suicide attempts (Nock & Banaji, 2007). Five of the eight studies showed a statistically significant result (63%), but the average observed power to achieve significance was only 42%. This discrepancy suggests the presence of publication bias (Schimmack, 2012). Moreover, significant results are all clustered around .05, and none
of the p values meets the stricter criterion of .005 that has been suggested by Nosek and others to claim a discovery (Benjamin et al., 2018). Thus, there is no conclusive evidence to suggest that suicide IATs have incremental predictive validity in the prediction of suicides. This is not surprising because most of the studies were underpowered and unlikely to detect small effects.
Moreover, effect sizes are bound to be small because the convergent validity between suicide and death IATs is low (r = .21; Chiurliza et al., 2018), suggesting that most of the variance in these IATs is measurement error.

In conclusion, 20 years of research with IATs has produced no credible and replicable evidence that IATs have incremental predictive validity over explicit measures. Even if there is some statistically significant incremental predictive validity, the amount of explained
variance may lack practical significance (Kurdi et al., 2019).

False fact 4

Kurdi et al. (2021) object (p. 424) to my claim that “most researchers regard the IAT as a valid measure of enduring attitudes that vary across individuals” (Schimmack, 2021, p. 397). They claim that “the overwhelming theoretical consensus in the community of attitude researchers.
. . is that attitudes emerge from an interaction of persons and situations” (p. 425). It is instructive to compare this surprising claim with Cunningham and Zelazo’s (2007) definition of attitudes as “relatively stable ideas about whether something is good or bad” (p. 97). Kurdi and Banaji (2017) wrote that “differences in implicit attitudes . . . may arise because of multiple components, including relatively stable components [emphasis added]” (p. 286). Rae and Greenwald (2017) stated that it is a “widespread assumption . . . that implicit attitudes are characteristics of people, almost certainly more so than a property of situations” (p. 297).
Greenwald and Lai (2020) stated that test–retest reliability “places an upper limit on correlational tests of construct validity” (p. 425). This statement makes sense only if we assume that the construct to be measured is stable over the retest interval. It is also not clear how it would be ethical to provide individuals with feedback about their IAT scores on the Project Implicit website, if IAT scores were merely a product of the specific situation at the moment they are taking the test. Finally, how can the suicide IAT be a useful predictor of suicide if it cannot not measure some stable dispositions related to suicidal behaviors?

In conclusion, Kurdi et al.’s definition of attitudes is inconsistent with the common definition of attitudes as relatively enduring evaluations. That being said, the more important question is
whether IATs measure stable attitudes or momentary situational effects. Ironically, some of the best evidence comes from Cunningham. Cunningham et al. (2001) repeatedly measured prejudice four times over a 3-month period with multiple measures, including the race IAT. Cunningham et al. (2001) modeled the data with a single trait factor that explained all of the covariation among different measures of racial attitudes. Thus, Cunningham et al. (2001) provided first evidence that most of the valid variance in race IAT scores is perfectly stable over a 3-month period and that person-by-situation interactions had no effect on racial attitudes. There have been few longitudinal studies with IATs since Cunningham et al.’s (2001) seminal study. However, last year, an article examined stability over a 6-year interval (Onyeador et al., 2020). Racial attitudes of more than 3,000 medical students were measured in the first year of medical school, the fourth year of medical school, and the second year of medical residency.
Table 4 shows the correlations for the explicit feeling thermometer and the IAT scores. The first observation is that the Time-1-to-Time-3 correlation for the IAT scores is not smaller than the Time-1-to-Time-2 or the Time-2-to-Time-3 correlations. This pattern shows that a single trait factor can capture the shared variance among the repeated IAT measures. The second observation is that the bold correlations between explicit ratings and IAT scores on the same occasion are only slightly higher than the correlations for different measurement
occasions. This finding shows that there is very little occasion-specific variance in racial attitudes. The third observation is that IAT correlations over time are higher than the corresponding FT-IAT correlations over time. This finding points to IAT-specific method variance that is revealed in studies with multiple implicit measures (Cunningham et al., 2001; Schimmack, 2021). These findings extend Cunningham et al.’s (2001) findings to
a 6-year period and show that most of the valid variance in race IAT scores is stable over long periods of time.

In conclusion, Kurdi et al.’s claims about person-by-situation effects are not supported by evidence.

Conclusion

Like presidential debates, the commentaries and my response present radically different views of reality. In one world, IATs are valid and useful tools that have led to countless new insights into human behavior. In the other world, IATs are noisy measures that add nothing to the information we already get from cheaper self-reports. Readers not well versed in the literature are likely to be confused rather than informed by these conflicting accounts. Although we may expect such vehement disagreement in politics, we should not expect it among scientists.
A common view of scientists is that they are able to resolve disagreement by carefully looking at data and drawing logical conclusions from empirical facts. However, this model of scientists is naive and wrong.

A major source of disagreement among psychologists is that psychology lacks an overarching paradigm; that is, a set of fundamentally shared assumptions and facts. Psychology does not have one paradigm, but many paradigms. The IAT was developed within the implicit social-cognition paradigm that gained influence in the 1990s (Bargh et al., 1996; Greenwald & Banaji, 1995; Nosek et al., 2011). Over the past decade, it has become apparent that the empirical foundations of this paradigm are shaky (Doyen et al., 2012; D. Kahneman quoted in Yong, 2012, Supplemental Material; Schimmack, 2020). It took a long time to see the problems because paradigms are like prisons that make it impossible to see the world from the outside. A key force that prevents researchers within a paradigm from noticing problems is publication bias. Publication bias ensures that studies that are consistent with a paradigm are published, cited, and highlighted in review articles to provide false evidence in support for a paradigm
(Greenwald & Lai, 2020; Kurdi et al., 2021).

Over the past decade, it has become apparent how pervasive these biases have been, especially in social psychology (Schimmack, 2020). The responses to my critique of IATs merely confirms how powerful paradigms and conflicts of interest can be. It is therefore necessary to allocate more resources to validation projects by independent researchers. In addition, validation studies should be preregistered and properly powered, and results need to be published whether they show validity or not. Conducting validation studies of widely used measures could be an important role for the emerging field of meta-psychology that is not focused on new discoveries, but rather on evaluating paradigmatic research from an outsider, meta-perspective (Carlsson et al., 2017). Viewed from this perspective, many IATs that are in use lack credible evidence of construct validity.

References
*References marked with an asterisk report studies included in
the suicide IAT meta-analysis

Banaji, M. R., & Greenwald, A. G. (2013). Blindspot: Hidden
biases of good people. Delacorte Press.

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity
of social behavior: Direct effects of trait construct and
stereotype activation on action. Journal of Personality
and Social Psychology, 71(2), 230–244. https://doi.org/
10.1037/0022-3514.71.2.230

*Barnes, S. M., Bahraini, N. H., Forster, J. E., Stearns-Yoder, K. A.,
Hostetter, T. A., Smith, G., Nagamoto, H. T., & Nock,
M. K. (2017). Moving beyond self-report: Implicit associations
about death/ life prospectively predict suicidal
behavior among veterans. Suicide and Life-Threatening
Behavior, 47, 67–77. https://doi.org/10.1111/sltb.12265

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A.,
Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B.,
Brown, L., Camerer, C., Cesarini, D., Chambers, C. D.,
Clyde, M., Cook, T. D., Boeck, P., De, Dienes, Z., Dreber,
A., Easwaran, K., Efferson, C., . . . Johnson, V. E. (2018).
Redefine statistical significance. Nature Human Behaviour,
2, 6–10.

Bosson, J. K., Swann, W. B. Jr., & Pennebaker, J. W. (2000).
Stalking the perfect measure of implicit self-esteem:
The blind men and the elephant revisited? Journal of
Personality and Social Psychology, 79, 631–643. https://
doi.org/10.1037/0022-3514.79.4.631

Buhrmester, M. D., Blanton, H., & Swann, W. B., Jr. (2011).
Implicit self-esteem: Nature, measurement, and a new way
forward. Journal of Personality and Social Psychology,
100(2), 365–385. https://doi.org/10.1037/a0021341

Carlsson, R., Danielsson, H., Heene, M., Ker, Å., Innes, Lakens,
D., Schimmack, U., Schönbrodt, F. D., van Assen, M., &
Weinstein, Y. Inaugural editorial of Meta-Psychology. Meta-
Psychology, 1. https://doi.org/10.15626/MP2017.1001

Chiurliza, B., Hagan, C. R., Rogers, M. L., Podlogar, M. C., Hom,
M. A., Stanley, I. H., & Joiner, T. E. (2018). Implicit measures
of suicide risk in a military sample. Assessment, 25(5),
667–676. https://doi.org/10.1177/1073191116676363
Cohen, J. (1988). Statistical power analysis for the behavioral
sciences (2nd ed.). Erlbaum.

Cunningham, W. A., Preacher, K. J., & Banaji, M. R. (2001).
Implicit attitude measures: Consistency, stability, and
No Evidence for Construct Validity of IAT 441
convergent validity. Psychological Science, 12(2), 163–170
https://doi.org/10.1111/1467-9280.00328

Cunningham, W. A., & Zelazo, P. D. (2007). Attitudes and
evaluations: A social cognitive neuroscience perspective.
Trends in Cognitive Sciences, 11, 97–104. https://
doi.org/10.1016/j.tics.2006.12.005

Cvencek, D., Meltzoff, A. N., Maddox, C. D., Nosek, B. A.,
Rudman, L. A., Devos, T., Dunham, Y., Baron, A. S.,
Steffens, M. C., Lane, K., Horcajo, J., Ashburn Nardo, L.,
Quinby, A., Srivastava, S. B., Schmidt, K., Aidman, E.,
Tang, E., Farnham, S., Mellott, D. S., . . . Greenwald, A. G.
(2020). Meta-analytic use of balanced identity theory to
validate the Implicit Association Test. Personality and
Social Psychology Bulletin, 47(2), 185–200. https://doi
.org/10.1177/0146167220916631

Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012).
Behavioral priming: It’s all in the mind, but whose mind?
PLOS ONE, 7(1), Article e29081. https://doi.org/10.1371/
journal.pone.0029081

Falk, C. F., Heine, S. J., Takemura, K., Zhang, C. X., & Hsu,
C. (2015). Are implicit self-esteem measures valid for
assessing individual and cultural differences. Journal of
Personality, 83, 56–68. https://doi.org/10.1111/jopy.12082

*Glenn, C. R., Millner, A. J., Esposito, E. C., Porter, A. C.,
& Nock, M. K. (2019). Implicit identification with death
predicts suicidal thoughts and behaviors in adolescents.
Journal of Clinical Child & Adolescent Psychology, 48,
263–272. https://doi.org/10.1080/15374416.2018.1528548

Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition:
Attitudes, self-esteem, and stereotypes. Psychological
Review, 102(1), 4–27. https://doi.org/10.1037/0033-
295X.102.1.4

Greenwald, A. G., & Farnham, S. D. (2000). Using the Implicit
Association Test to measure self-esteem and self-concept.
Journal of Personality and Social Psychology, 79, 1022–1038
https://doi.org/10.1037/0022-3514.79.6.1022

Greenwald, A. G., & Lai, C. K. (2020). Implicit social cognition.
Annual Review of Psychology, 71, 419–445. https://
doi.org/10.1146/annurev-psych-010419-050837

Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998).
Measuring individual differences in implicit cognition:
The Implicit Association Test. Journal of Personality and
Social Psychology, 74, 1464–1480.

*Harrison, D. P., Stritzke, W. G. K., Fay, N., & Hudaib, A.-R.
(2018). Suicide risk assessment: Trust an implicit probe
or listen to the patient? Psychological Assessment, 30(10),
1317–1329. https://doi.org/10.1037/pas0000577

Jiang, C., Vitiello, C., Axt, J. R., Campbell, J. T., & Ratliff, K. A.
(2019). An examination of ingroup preferences among
people with multiple socially stigmatized identities. Self
and Identity. Advance online publication. https://doi.org/
10.1080/15298868.2019.1657937

Kurdi, B., & Banaji, M. R. (2017). Reports of the death of
the individual difference approach to implicit social cognition
may be greatly exaggerated: A commentary on Payne,
Vuletich, and Lundberg. Psychological Inquiry, 28,
281–287. https://doi.org/10.1080/1047840X.2017.1373555

Kurdi, B., Ratliff, K. A., & Cunningham, W. A. (2021). Can
the Implicit Association Test serve as a valid measure of
automatic cognition? A response to Schimmack (2021).
Perspectives on Psychological Science, 16(2), 422–434.
https://doi.org/10.1177/1745691620904080

Kurdi, B., Seitchik, A. E., Axt, J. R., Carroll, T. J., Karapetyan,
A., Kaushik, N., Tomezsko, D., Greenwald, A. G., &
Banaji, M. R. (2019). Relationship between the Implicit
Association Test and intergroup behavior: A meta-analysis.
American Psychologist, 74(5), 569–586. https://doi.org/
10.1037/amp0000364

*Millner, A. J., Augenstein, T. M., Visser, K. H., Gallagher, K.,
Vergara, G. A., D’Angelo, E. J., & Nock, M. K. (2019). Implicit
cognitions as a behavioral marker of suicide attempts in
adolescents. Archives of Suicide Research, 23(1), 47–63.
https://doi.org/10.1080/13811118.2017.1421488

*Nock, M. K., & Banaji, M. R. (2007). Prediction of suicide ideation
and attempts among adolescents using a brief performance-
based test. Journal of Consulting and Clinical
Psychology, 75(5), 707–715. https://doi.org/10.1037/0022-
006X.75.5.707

*Nock, M. K., Park, J. M., Finn, C. T., Deliberto, T. L.,
Dour, H. J., & Banaji, M. R. (2010). Measuring the suicidal
mind: Implicit cognition predicts suicidal behavior.
Psychological Science, 21(4), 511–517. https://doi
.org/10.1177/0956797610364762

Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening
scientific communication. Psychological Inquiry, 23(3),
217–243. https://doi.org/10.1080/1047840X.2012.692215

Nosek, B. A., Hawkins, C. B., & Frazier, R. S. (2011). Implicit
social cognition: From measures to mechanisms. Trends
in Cognitive Sciences, 15(4), 152–159. https://doi.org/
10.1016/j.tics.2011.01.005

Onyeador, I. N., Wittlin, N. M., Burke, S. E., Dovidio, J. F.,
Perry, S. P., Hardeman, R. R., Dyrbye, L. N., Herrin, J.,
Phelan, S. M., & van Ryn, M. (2020). The value of interracial
contact for reducing anti-Black bias among non-Black
physicians: A Cognitive Habits and Growth Evaluation
(CHANGE) study report. Psychological Science, 31(1),
18–30. https://doi.org/10.1177/0956797619879139

Phelps, E. A., Cannistraci, C. J., & Cunningham, W. A. (2003).
Intact performance on an indirect measure of race bias
following amygdala damage. Neuropsychologia, 41(2),
203–208. https://doi.org/10.1016/s0028-3932(02)00150-1

Rae, J. R., & Greenwald, A. G. (2017). Persons or situations?
Individual differences explain variance in aggregated
implicit race attitudes. Psychological Inquiry, 28, 297–300.
https://doi.org/10.1080/1047840X.2017.1373548

*Randall, J. R., Rowe, B. H., Dong, K. A., Nock, M. K., &
Colman, I. (2013). Assessment of self-harm risk using
implicit thoughts. Psychological Assessment, 25(3), 714–721
https://doi.org/10.1037/a0032391

Schimmack, U. (2012). The ironic effect of significant results
on the credibility of multiple-study articles. Psychological
Methods, 17(4), 551–566. https://doi.org/10.1037/a0029487

Schimmack, U. (2020). A meta-psychological perspective on
the decade of replication failures in social psychology.
Canadian Psychology/Psychologie canadienne, 61(4),
364–376. http://doi.org/10.1037/cap0000246

Schimmack, U. (2021). The Implicit Association Test: A method
in search of a construct. Perspectives on Psychological Science, 16(2), 396–414. https://doi.org/10.1177/1745691619863798

Teachman, B. A., Clerkin, E. M., Cunningham, W. A., Dreyer-
Oren, S., & Werntz, A. (2019). Implicit cognition and
psychopathology: Looking back and looking forward.
Annual Review of Clinical Psychology, 15, 123–148.
https://doi.org/10.1146/annurev-clinpsy-050718-095718

*Tello, N., Harika-Germaneau, G., Serra, W., Jaafari, N., &
Chatard, A. (2020). Forecasting a fatal decision: Direct
replication of the predictive validity of the Suicide–
Implicit Association Test. Psychological Science, 31(1),
65–74. https://doi.org/10.1177/0956797619893062

Vianello, M., & Bar-Anan, Y. (2021). Can the Implicit Association
Test measure automatic judgment? The validation continues.
Perspectives on Psychological Science, 16(2), 415–421.
https://doi.org/10.1177/1745691619897960

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009).
Puzzlingly high correlations in fMRI studies of emotion,
personality, and social cognition. Perspectives on
Psychological Science, 4(3), 274–290. https://doi.org/10
.1111/j.1745-6924.2009.01125.x

Walker, S. S., & Schimmack, U. (2008). Validity of a happiness
implicit association test as a measure of subjective wellbeing.
Journal of Research in Personality, 42, 490–497.
https://doi.org/10.1016/j.jrp.2007.07.005

Yong, E. (2012 October 12). Nobel laureate challenges
psychologists to clean up their act. Nature. https://doi
.org/10.1038/nature.2012.11535

An Honorable Response to the Credibility Crisis by D.S. Lindsay: Fare Well

January 3, 2020Credibility Crisis, Psychological Science, Replication Crisis, Z-Curve, ZcurveUlrich Schimmack

We all know what psychologists did before 2012. The name of the game was to get significant results that could be sold to a journal for publication. Some did it with more power and some did it with less power, but everybody did it.

In the beginning of the 2010s it became obvious that this was a flawed way to do science. Bem (2011) used this anything-goes to get significance approach to publish 9 significant demonstration of a phenomenon that does not exist: mental time-travel. The cat was out of the bag. There were only two questions. How many other findings were unreal and how would psychologists respond to the credibility crisis.

D. Steve Lindsay responded to the crisis by helping to implement tighter standards and to enforce these standards as editor of Psychological Science. As a result, Psychological Science has published more credible results over the past five years. At the end of his editorial term, Linday published a gutsy and honest account of his journey towards a better and more open psychological science. It starts with his own realization that his research practices were suboptimal.

Early in 2012, Geoff Cumming blew my mind with a talk that led me to realize that I had been conducting underpowered experiments for decades. In some lines of research in my lab, a predicted effect would come booming through in one experiment but melt away in the next.
My students and I kept trying to find conditions that yielded consistent statistical significance—tweaking items, instructions, exclusion rules—but we sometimes eventually threw in the towel because results were maddeningly inconsistent. For example, a chapter by Lindsay
and Kantner (2011) reported 16 experiments with an on-again/off-again effect of feedback on recognition memory. Cumming’s talk explained that p values are very noisy. Moreover, when between-subjects designs are used to study small- to medium-sized effects, statistical
tests often yield nonsignificant outcomes (sometimes with huge p values) unless samples are very large.

Hard on the heels of Cumming’s talk, I read Simmons, Nelson, and Simonsohn’s (2011) “False-Positive Psychology” article, published in Psychological Science. Then I gobbled up several articles and blog posts on misuses of null-hypothesis significance testing (NHST). The
authors of these works make a convincing case that hypothesizing after the results are known (HARKing; Kerr, 1998) and other forms of “p hacking” (post hoc exclusions, transformations, addition of moderators, optional stopping, publication bias, etc.) are deeply problematic. Such practices are common in some areas of scientific psychology, as well as in some other life
sciences. These practices sometimes give rise to mistaken beliefs in effects that really do not exist. Combined with publication bias, they often lead to exaggerated estimates
of the sizes of real but small effects.

This quote is exceptional because few psychologists have openly talked about their research practices before (or after) 2012. It is an open secrete that questionable research practices were widely used and anonymous surveys support this (John et al., 2012), but nobody likes to talk about it. Lindsay’s frank account is an honorable exception in the spirit of true leaders who confront mistakes head on, just like a Nobel laureate who recently retracted a Science article (Frances Arnold).

1. Acknowledge your mistakes.

2. Learn from your mistakes.

3. Teach others from your mistakes.

4. Move beyond your mistakes.

Lindsay’s acknowledgement also makes it possible to examine what these research practices look like when we examine published results, and to see whether this pattern changes in response to awareness that certain practices were questionable.

So, I z-curved Lindsay’s published results from 1998 to 2012. The graph shows some evidence of QRPs, in that the model assumes more non-significant results (grey line from 0 to 1.96) than are actually observed (histogram of non-significant results). This is confirmed by a comparison of the observed discovery rate (70% of published results are significant) and the expected discovery rate (44%). However, the confidence intervals overlap. So this test of bias is not significant.

The replication rate is estimated to be 77%. This means that there is a 77% probability that repeating a test with a new sample (of equal size) would produce a significant result again. Even for just significant results (z = 2 to 2.5), the estimated replicability is still 45%. I have seen much worse results.

Nevertheless, it is interesting to see whether things improved. First of all, being editor of Psychological Science is full-time job. Thus, output has decreased. Maybe research also slowed down because studies were conducted with more care. I don’t know. I just know that there are very few statistics to examine.

Although the small sample size of tests makes results somewhat uncertain, the graph shows some changes in research practices. Replicability increased further to 88% and there is no loner a discrepancy between observed and expected discovery rate.

If psychology as a whole had responded like D.S. Lindsay it would be in a good position to start the new decade. The problem is that this response is an exception rather than the rule and some areas of psychology and some individual researchers have not changed at all since 2012. This is unfortunate because questionable research practices hurt psychology, especially when undergraduates and the wider public learn more and more how untrustworthy psychological science has been and often still us. Hopefully, reforms will come sooner than later or we may have to sing a swan song for psychological science.