For years, psychologists interested in their ‘impact’ could check their citation counts in Web of Science and see an increase. The main reason for this increase was the creation of more journals, including online journals without page limitations. The positive illusion of increasing importance came to a halt in 2021.

Figure 1. Citation count for the journal Psychological Science

The reason is probably that WebOfScience became more selective and delisted some low quality and possibly predatory journals. The end of exponential increases in citations is an interesting phenomenon in itself, but it is not the key focus of this blog post. The main question is whether the popping of the citation bubble also reflects concerns about the credibility of published articles in the wake of the replication crisis.

A useful analogy are stock market bubbles, like the valuation of new internet companies in the late 1990s. A bubble is created because investors buy stocks independent of the true value of a stock. Famous value-investor Warren Buffet quipped “Only when the tide goes out do you discover who’s been swimming naked” (not there is anything wrong with it).

After the crash of the dot.com bubble, companies like  Pets.com lost all of their value. In contrast, (for better or worse) companies like Amazon.com became very valuable because they actually generated revenues and profits.

This blog post examines whether the citation-crash in 2021 affected all researchers equally or whether it discriminated between credible researchers that advanced science and researchers who produced articles with incredible results that are difficult to replicate and do not advance psychological science. The list of researchers is not meant to be representative in any way, but based on their significance during the replication crisis. My personal focus is on personality and social psychology. Readers interested in other research areas with access to WebOfScience can compare results for researchers they are interested in on their own.

Citations of Journals

To create a benchmark, i picked a small set of journals that represent social and personality research. For each journal, I recorded the peak (highest number of citations in a year), and the citations in 2023 and 2024. 2024 citations will still increase a bit over the next couple of month. So, a difference between 2023 and 2024 cannot be interpreted as a further decrease, yet. (I may update the numbers in a couple of months).

Note. JESP = Journal of Experimental Social Psychology, EJSP = European Journal of Social Psychology, EJP = European Journal of Personality, JRP = Journal of Research in Personality, PsySci = Psychological Science

The results are fairly consistent across journals. So far, the citation rates in 2024 are 90% or more of those in 2024. Thus, there is no evidence that citations are decreasing notably. The comparison of citations in 24 with the peak show that citations are now 20% to 30% below the peak (bear market territory in stock market language). The most notable drop is observed for Social Cognition, which may reflect the loss of trust or interest in social cognition research, which was hit hard by the replication crisis. The comparison of peak and citations in 2023, shows a similar pattern with a higher average because 2024 citations are not yet complete. Again, social cognition has the biggest drop. Psychological science which also published a lot of questionable research shows the second biggest drop, but there is no notable difference between social and personality journals in general. The averages provide a benchmark to examine trends for specific researchers. A loss in trust (or value in stock market language) would be revealed by bigger drops from the peak to recent years than the average drop for personality and social psychology journals.

Citation Counts of Prominent Social and Personality Psychologists

To provide context, i am presenting the results in order of the 24/Peak values that reflect the biggest drop in confidence.

1. Diderick Stapel

Diderick Stapel contributed to the loss of confidence in (social) psychology in 2011, when it emerged that he had faked data for numerous articles. After an investigation, 58 of his articles were retracted. It is remarkable that Stapel’s peak citation count is a meager 367, a number that pales in comparison to some of the citation counts below.

Citations have decreased to 27% of the peak in 2024 and 32% in 2023. The comparison of 24 and 23 suggests that citations are still decreasing (85% vs. 94% for the journal average). Evidently, awareness of fraud and retractions have produced a strong signal to neglect Stapel’s articles. However, this decrease is unrelated to the bust of the bubble in 2021. The correction appeared rather quickly after the discovery of fraud in 2011.

2. John A Bargh

John A Bargh is best known for his articles on priming without awareness, like his famous claim that reading a few words related to the elderly makes young students walk slower. This work was featured in the popular book of Nobel Laureate Daniel Kahneman along with other social priming studies. After a failed replication study, Kahneman wrote a letter to Bargh expressing his concerns about the replicability of social priming results. In the following years many of these results could not be replicated, confirming suspicion that these results were obtained with questionable research practices, including hiding of studies that did not show the predicted results (cf. Schimmack et al., 2017).

Citations for Bargh are now only 49% of the peak citations in 2014. Thus, it does not take fraud and retractions to correct citation bubbles based on questionable research. Bargh did not make up data like Stapel. Rather, he used unscientific ways to analyze data that increase the chances of obtaining a statistically significant and publishable result or simply conducting many studies and reporting only those that ‘worked.’

The bubble burst in 2014. While there was a slight uptick in citations in 2019, citations increased notably in the following years. With over 800 citations a year, the absolute number of citations is still high for psychology and suggests that many researchers still cite this work without awareness that social priming results are questionable.

On a positive note, the decrease in citations is notably bigger than the average decrease from peak to 2024, 49% vs. 77%.

3. Fritz Strack

Fritz Strack is another social cognition researcher with a notable presence during the replication crisis in social psychology. For example, he wrote a meta-scientific article in defense of social psychology in the face of replicability problems (Strobe & Strack, 2014). Moreover, he is the author of an article that used a deceptive manipulation of facial muscles to provide evidence for the facial feedback theory of emotions. In 2019, he was awarded the IG Nobel Prize for this work. The results could not be replicated in one large replication attempts. is study failed to replicate in a big replication attempt, but he attributed this to problems with the replication study. However, another replication attempt in a project that he was actively involved in also failed. Another line of work suggested that item-order influences the correlation between life-satisfaction and domain satisfaction (e.g., marital satisfaction) judgments. This result was featured in Daniel Kahneman’s book and also failed to replicate. it is now generally accepted that life-satisfaction judgments are rather stable and based on chronically accessible information about important life domains. He is also a co-author on an article about an experimental manipulation of ease of retrieval that also failed to replicate (Schwarz et al., 1991).

The citation count peaked in 2021 and then decreased notably.

Importantly, the decrease is more pronounced that the average decrease of articles in social and personality psychology (

Given the recency of the peak, it is possible that citations continue to fall faster than the average in the coming years.

4. Dan Ariely

Dan Ariely’s research practices were scrutinized after it was discovered that data in one study of an article with him as author were manipulated. The data were provided by him, but he maintains that the manipulation was carried out by somebody else. Dan Ariely also co-authored article with another researcher, Francesca Gino, who is under investigation for data manipulation.

Ariely may not care much about citation counts, as he is one of those psychologists who have broken out of academia into pop culture. His popular book “Predictably Irrational” about his life and research was turned into a TV show that premiered in 2023 on NBC.

Citations peaked in 2021, when the citation bubble burst. Since then, citations have decreased somewhat faster than the average (62% vs. 77%).

Given the recency of the peak and the ongoing investigation of articles co-authored with Gino, it is possible that citation counts will decrease more than the average article in the coming years.

5. Anthony Greenwald*

* tied with Danna Carney who was involved in power-posing research, but distanced herself from this work. Her other work has not been a topic in the replication crisis and her overall citation count is low compared to the famous authors listed here .

Anthony Greenwald is famous for the invention of the Implicit Association Test (IAT). Millions of people have used IATs to obtain feedback about their unconscious/implicit/automatic associations. The most famous IAT is the race IAT that has been used to claim that the vast majority are more prejudiced against African Americans than they would admit in self-ratings. The test has also been used to suggest that African Americans often hold more favorable attitudes of White people than Black people even if they would state the opposite. A famous example is Malcolm Gladwell.

Numerous authors, including myself, have criticized the interpretation of IAT scores and pointed out that they lack evidence of validity (they may not measure racial attitudes) and are poor predictors of actual behavior. For example, the race IAT failed to predict racial bias in the 2008 US presidential election (Obama vs. McCain). That is, there was racial bias and self-report measures showed it, but IAT scores did not. Even proponents of implicit biases are now critical about the use of the IAT to study these biases. Even Greenwald has walked back unsupported claims in his popular book “Blindspots” and suggests IAT scores should simply be interpreted as scores produced by a method that does not rely on self-reports without specific claims about the meaning of these scores (i.e., it is a method without a clear construct).

Citations peaked in 2021, when the citation bubble burst. The decline in citations is somewhat steeper than the journal average (66% vs. 77%), but the number of citations in 2024 is still likely to break the 2,000 mark. The hype about the IAT is far from over and Project Implicit continues to hide the fact that the test is often invalid and provides false feedback.

6. Francesca Gino

The discovery of data anomalies (lawyer speak for funky data) in an article by Francesca Gino was big news on social media in 2023 (Datacolada, 2023). The accusation of fraud led to an investigation by Gino’s employer (Harvard) and a lawsuit by Gino against Harvard and the Datacolada team. The lawsuit against the investigators was recently dropped (Science, 2024) and the four flagged articles have been retracted. The investigation and law suit have drawn a lot of attention to Gino’s work. Collaborators have tried to separate credible studies from studies tainted by data from Gino’s lab, but this effort has not resulted in a clear separation of articles with and without tainted data. This may have affected citation of articles co-authored by Gino.

Note. Only articles from 2007 or later are used because WebOfScience only uses full first names in those years and using the initial would retrieve too many articles by other authors.

Citation counts peaked in 2021. The decrease from 2021 to 2024 is steeper than the average decrease for social psychology journals (67% vs. 77%). If more articles are retracted, the decrease is likely to accelerate in the coming years.

7. Tom Pyszczynski

IT was easier to search for citations for Tom Pyszczynski than his collaborators Jeff Greenberg and Sheldon Solomon, who invented Terror Management Theory (TMT) because his name is more unique. Terror management theory postulates that reminders of death alter people’s priorities and behaviors. Replicability varies depending on the methods that were used to study TMT. Many influential studies used priming without awareness. These results are difficult to replicate as are other priming studies (see Bargh above). TMT studies were highly influential in the early 2000s, but the topic attracted less attention recently, although TMT studies were not a major focus of discussion during the replication crisis.

Pyszczynski’s citation count peaked in 2021 when the citation bubble burst.

Citations decreased a little bit more than the journal average. However, with nearly 800 citations in 2024, TMT is likely to outlive its creators.

8. Kathleen D. Vohs

Kathleen Vohs’ has two lines of research related to the replication crisis. Her work on money priming is just a variation of priming without awareness studies with money as primes. These studies lack credibility and there is good evidence that published results were obtained with questionable research practices (Schimmack, 2024). The more important work was conducted in collaboration with Roy F. Baumeister on ego-depletion. Ego-depletion research has become another posterchild of replication failures. After a large replication project failed to show the effect, Vohs and colleague conducted another large replication project to demonstrate that the first attempt was flawed. However, even she could not replicate the finding with over 2,000 participants. Meta-analyses of published studies also find clear evidence of bias and little evidence of a real effect after controlling for bias. The general consensus among meta-psychologists is that the large ego-depletion literature has produced little insights into willpower and self-regulation failures (e.g., publishing only significant results although this violates scientific principles). Only a few people, still insist that ego-depletion effects are real. It is not clear where Vohs stands on this issue.

Vohs’ citation count peaked in 2021, the year the citation bubble burst. The decrease is a bit steeper than the journal average, down by 29 percentage points from the peak (71% vs. 77). However, with over 1,700 citations in 2024, there is still a lot of room for a correction that reflects the value of her money-priming and ego-depletion articles.

9. Dan Gilbert

Dan Gilbert made a contribution to the replication debate with some snarky online comments (e.g., calling replication researchers “shameless little bullies”) and an attempt to discredit the embarrassing finding of the reproducibility project that only 25% of social psychology studies could be replicated. The key argument was that replication studies were underpowered, but this argument does not explain how similarly powered studies could have a success rate of over 90% in the published articles.

His own research is often co-authord by Timothy Wilson, who is equally known for disparaging comments about replication researchers (Bartlett, 2014). It was easier to look up citations for Gilbet than for Wilson. Their work covers many topics, but they are probably best known for their studies on affective forecasting. The idea is that we overestimate the effects of negative events on our moods. The effect is replicable, but there is disagreement about the interpretation of the finding (Levine et al., 2012).

The peak citation count was in 2020, a year before the bubble burst. Citation counts decreased a bit faster than the journal average down to 71% versus 77%.

10. Roy F. Baumeister

And last, but not least, is Roy F. Baumeister, the inventor of ego-depletion (with Kathleen Vohs). At some point, Baumeister proposed that will-power is related to blood glucose levels and this claim was of course supported by statistically significant results, no less than 9 to be exact. When I wrote an article that pointed out that it is unlikely to get so many significant results without a non-significant effect even if an effect is real, Baumeister was a reviewer of the manuscript and simply noted that they of course had more studies and didn’t publish the weaker ones. That is just how the field works. At this point, ego-depletion was still accepted as a phenomenon, even if it didn’t rely on glucose levels. Now, even ego-depletion has been debunked and hundreds of significant results provide no insight into self-regulation. Nevertheless, Roy Baumeister continues to write articles that claim ego-depletion is real and all the critics are wrong. Roy F. Baumeister has also written on many other topics with influential review articles. For example, he reviewed credible evidence that men have a stronger sex drive than women.

Roy F. Baumeister has an impressive citation record with a peak citation rate of 7192 citations in 2021. He may be relieved that the decline in the following years is partially due to a general decline.

However, the decline is a bit steeper than the decline for the average journal (73% vs. 77%). To examine whether the results would be different, I also examined only articles that mention ego-depletion as a search term in any field. The key finding was hardly different (71% vs 73%). With nearly 900 ego-depletion related citations, psychology still has a long time to find out that Baumeister was swimming naked.

And here you have it. The most interesting result is that Bargh is the most notable victim of the replication crisis, whereas equally questionable research like the work on ego-depletion has been affected much less by demonstration of bias and major replication failures. While the citation bubble has burst, researchers are not as savvy as value investors to spot junk stocks. The following years will show whether awareness about replication failures spreads.

Some People Can be Better than Average

Some notable researchers have trends above the journal average. Here are three examples.

1. Susan Fiske

Susan Fiske became famous during the replication crisis when she called investigators of research bias “method terrorists.” Her own research has not been the focus of method terrorists. This may explain why her citation count decreased less than the journal average (83% vs. 77%).

2. Ed Diener

Ed Diener is one of the most influential psychologists of his generation, akin to Roy F. Baumeister. His main research has been on life-satisfaction and subjective well-being. This research has flourished since Diener published his seminal review paper in 1984. His Satisfaction with Life Scale (Diener et al., 1985) is still the most widely used multi–item measure of SWB. Many results have been replicated with large, nationally representative samples.

Citation counts peaked once again in 2021, the year the citation bubble burst. However, the decrease is relatively small and less than the journal average (87% vs. 77%).

A Positive Response to the Replication Crisis

My UofT colleague Michael Inzlicht also played a part in the replication crisis. In contrast to many social psychologists, he acknowledged that some of his early work was based on shoddy practices that produced results that cannot be replicated. Just recently, he commented on the demise of stereotype threat, another popular literature that turned out to be junk science (Inzlicht, 2024). After being unable to replicate his earlier findings at UofT, he started work on ego-depletion. After he joined a large replication project that failed to show the effect, he gave up on ego-depletion as well (Inzlicht, 2020). Since then, he started over and has already gathered a citation record that stands on its own without the previous publications.

Citation counts for publications (k = 80) from 2000 to 2014 peaked in 2019 and then decreased sharply. The decrease matches the decrease for the third place (Strack).

However, over the past 10 years, Inzlicht published already over 100 new articles that have a higher peak than his old publications.

Moreover, the decrease is shallow and above the journal average (91% vs. 77%).

There you have it. When you invested in a bubble, it is best to sell and invest again in some new enterprise rather than sticking to your guns and hoping this will turn around.

Your’s Truly

Of course, I also looked up myself. In fact, the post is based on my investigation of the disappointing decrease in my own citations. It is comforting that the decrease is general and not specific to my publications.

Rather the decrease is just slightly below the journal average (75% vs. 77%).

Conclusion

Citation counts matter even if they are only a poor reflection of research quality. They show what people are studying or reading and they are used to allocate resources for future research. The availability of citation counts makes it easy to conduct some meta-scientific studies on the popularity of topics and researchers. Whether the average number of citations goes up or down is a matter of journal space and the tracking of articles in an index. In WebOfScience citation counts have decreased. Researchers who look up themselves should be aware that the decrease is not a reflection of their popularity. More important is the relative increase compared to some benchmark or other researchers. Here we see some interesting differences. Most notably, citation counts of people who used fraud (Stapel) or questionable research practices (Bargh) to produce results that cannot be replicated have decreased notably. This is probably more important than the absolute number of citations or the H-Index of a researcher. The H-Index does not track whether a researcher made a positive contribution to science. It merely shows that a researcher was popular at some point in time. The hallmark of science is that it is self-correcting. This means that bubbles will be deflated eventually and popular but false claims will stop being citated. It is good to see some science that psychology acts like a science, but there is still a lot of room for further corrections in the future. So, do not be disappointed if your citation count pales in comparison to some of the giants in the field or decreased this year. No need for imposter syndrome. Not everybody can be better than average and some of the giants did more harm than good. Better to be just average.

Happy New Year
Ulrich Schimmack

I should have done this before I submitted a ms. for publication, but later is better than never. After waiting for a couple of months, I received a rejection for a ms. on the relationship between personality and life-satisfaction.

The rejection decision was based on a form letter by the action editor without any acknowledgement that they read the paper or comments on the main point of the ms. and two reviews.

.

One review was by the author who’s data I reanalyzed and showed that the published results are not supported by my reanalyses. You can guess how favorable that review was or you can read it below. The other review made no substantial comments, probably because the reviewer is not familiar with structural equation modeling and unable to read the MPLUS output files that were shared on OSF. Anyhow, these reviews provide no helpful feedback, but they served the purpose of allowing the editor to reject the ms.

Unhappy, or rather frustrated, I decided to ask ChatGPT for a CRITICAL review of the manuscript and just pasted the manuscript in the dialogue box. Less than a minute later, i had a objective review that showed understanding of the issue, acknowledged strength, and pointed out several limitations that can be used to strengthen the manuscript. Wow. This is a gamechanger. Not only can editors save time and get valuable reviews. Authors can get the feedback before they submit a ms. and improve it. We all knew peer-review is shit because it is driven by nepotism, defensiveness, motivated bias, incompetence, and lack of time and motivation to do a good job. I have written my fair share of terse reviews without much care for the feelings of the authors. ChatGPT doesn’t have a bone in the fight. It may not be ready to replace peer-review, but it surely can elevate it.

ChatGPT Review

Critical Peer Review of “Personality and Life-Satisfaction: A Hierarchical, Multi-Rater Perspective”

Title and Abstract The manuscript addresses a critical and enduring question in psychology: the relationship between personality traits and life satisfaction. The title effectively conveys the scope of the research, and the abstract provides a concise overview of the methodology, results, and implications. However, the abstract could better highlight the novelty of the hierarchical factor analysis approach used, as this seems to be a significant contribution of the study.

Strengths of the Manuscript

1. Methodological Rigor: The use of hierarchical confirmatory factor analysis to separate variance components represents a significant methodological strength. This approach allows the study to address measurement errors and identify unique contributions of specific traits.
2. Multi-Rater Datasets: Incorporating multiple datasets with self-ratings and informant ratings enhances the reliability and generalizability of the findings. The inclusion of domain satisfaction measures in four of the datasets is also commendable.
3. Findings on Neuroticism: The consistent identification of the depressiveness facet of Neuroticism as a primary predictor of life satisfaction is a valuable contribution to the literature. The mediation effect of domain satisfaction provides novel insights into the mechanisms underlying this relationship.
4. Contextual Moderation: The manuscript explores variations across populations (e.g., students vs. parents) and cultural contexts, which adds depth to the findings.

Weaknesses and Areas for Improvement

1. Theoretical Framing:
   • While the manuscript builds on Costa and McCrae’s hierarchical model, the discussion of how this model advances or challenges existing theories of personality and subjective well-being (SWB) could be expanded. For example, the implications of the findings for alternative models, such as dynamic equilibrium theory, are not sufficiently explored.
   • The manuscript heavily focuses on Neuroticism and Extraversion. It briefly mentions other Big Five traits (e.g., Openness) but does not delve into their potential roles or reasons for their weaker effects. This omission may leave readers with an incomplete picture.
2. Facet-Level Analysis:
   • While the study underscores the importance of the depressiveness facet of Neuroticism, the results for Extraversion and Cheerfulness are inconsistent. The manuscript could include a more thorough discussion of why these inconsistencies arise and what they suggest about the role of positive affect in life satisfaction.
   • There is little exploration of why specific traits (beyond Depressiveness and Cheerfulness) fail to consistently predict life satisfaction. Providing more nuanced interpretations could enrich the discussion.
3. Domain Satisfaction and Top-Down Effects:
   • The findings on domain satisfaction are compelling but underexplored. The “top-down” influence of personality traits on domain evaluations is mentioned, but the mechanisms underlying this influence remain vague. For instance, how does Depressiveness lead to negative evaluations of specific domains? This area warrants further elaboration.
   • The manuscript could provide more detail on the “bottom-up” integration of domain satisfaction into global life satisfaction judgments. Are certain domains (e.g., relationships, financial stability) consistently more influential across samples?
4. Methodological Concerns:
   • Measurement of Life Satisfaction: The use of a single-item measure of life satisfaction in some datasets raises validity concerns, as acknowledged by the authors. A more detailed justification for this choice and a discussion of its potential limitations are needed.
   • Sample Representativeness: The reliance on student samples in some studies limits generalizability. While this limitation is noted, it could be addressed more explicitly in the discussion.
   • Reliance on Multi-Rater Data: The manuscript assumes that variance shared among raters reflects valid variance. However, it does not adequately address potential biases introduced by informant ratings, such as social desirability effects or shared environmental influences.
5. Presentation and Accessibility:
   • The manuscript’s extensive use of technical language and complex statistical concepts may limit accessibility for non-specialist readers. Including simplified explanations or visual aids (e.g., diagrams of hierarchical models) would enhance readability.
   • The tables summarizing results are dense and lack accompanying narrative summaries that highlight key patterns. For example, Table 1 and Table 2 could benefit from clearer labels and textual integration into the results section.

Suggestions for Future Research

• The manuscript highlights the need to identify additional predictors of life satisfaction beyond Depressiveness and Cheerfulness. Future studies could explore traits or facets not captured by traditional Big Five measures, such as self-compassion or grit.
• Longitudinal studies that disentangle the causal relationships between personality, domain satisfaction, and life satisfaction would address limitations of the current cross-sectional datasets.
• Cross-cultural studies that examine the moderating effects of societal factors (e.g., individualism vs. collectivism) on the personality-life satisfaction relationship could provide valuable insights.

Conclusion The manuscript makes a meaningful contribution to the literature on personality and subjective well-being by quantifying the effects of personality traits and exploring their mediation through domain satisfaction. However, the study’s potential impact could be enhanced by addressing the theoretical, methodological, and presentation-related issues outlined above. Overall, the research is robust, and its findings are valuable, but greater clarity and depth in certain areas would strengthen its contribution to the field.

Overall Recommendation: Revise and resubmit, addressing the identified weaknesses and incorporating the suggested improvements to enhance the manuscript’s clarity, accessibility, and theoretical depth.

In an ideal world, humans would curb self-interest for the greater good. In reality, psychology has shown that human information processing is riddled with self-serving biases. A great achievement of human cultural evolution is the development of tools that can reduce these biases such as logical thinking and objective empirical observations. Since the invention of computers, it has also become easier to use simulations to put intuition to a test. However, motivated self-interest can bias simulations to produce desirable outcomes. For example, Uri Simonsohn made unreasonable assumptions to claim that p-curve performs well even with heterogeneity in power (Schimmack, 2018). It does not (Brunner,

An anonymous reviewer, henceforth known as Reviewer A (which may stand for anonymous or another word starting with A), recently accused us that we also used biased simulations to make the false claim that z-curve provides useful estimates of power with good coverage of confidence intervals (Brunner & Schimmack, 2020; Bartos & Schimmack, 2022; Schimmack & Bartos, 2023). The same reviewer previously made numerous false claims about z-curve and estimation of true power that we addressed elsewhere (Brunner, 2024). In our response to Reviewer A’s earlier comments, we also challenged them to provide a simulation that shows when z-curve breaks down.

Reviewer A was able to do so. The question is how they did it and whether Reviewer A’s results challenge our simulation results. Here is Reviewer A’s simulation.

To show that z-curve breaks down without hodgepodge heterogeneity, let us consider a situation of unconditional power of value .25   We have a one-sample Cohen’s d population value ranging from 0.2 to 2.2 (by increments of .05 to end up with 40 values), that is accompanied by sample size ranging from 167 to 2 that is calculated to be associated with a power of .25.

I generated data from each combination of Cohen’s d and sample size and fit a paired-samples t-test to obtain 40 p-values. These p-values are associated with a (“unconditional”) power value of .25. The expected discovery rate should be .25 (which is the power associated with the design of these observed results). The output I obtain from the zcurve package for the 40 estimated p-values is 0.05 – very far from the true value of .25.

To be clear, an estimate of 5% power (no evidence against the null-hypothesis) when the true power is 25% is horrible. So, we need to examine the conditions that lead to this horrible outcome. Here is Reviewer A’s code.

R code
d <- seq(.2,2.2,by = 0.05) # range of population Cohen’s d value
ssize<- matrix (0, ncol=40,nrow=1) #placeholder for sample size
pow = .25 #change this to change level of true power
for (i in 1:40){ #obtain sample size associated with Cohen’s d value for chosen level of power.
  ssize [i] <- pwr.t.test(n = NULL, d = d[i], sig = .05, power = pow,
                          type=”one.sample”, alternative=”two.sided”)$n
}
ssize_ <- round(ssize,digits=0) #round up the values
pp <- matrix(0,ncol=40,nrow=1)  #placeholder for estimated p-values
#let’s generate data and collect p-values
for (i in 1: 40){
  dat <- rnorm(n=ssize_[i], mean=d[i], sd=1)
  pp[i] <- t.test(dat, paired=FALSE, alternative=”two.sided”)$p.value
}
*let’s use zcurve to see whether EDR reproduces the specified power value
zcurve(p=as.vector(pp))
==

Let’s first address an annoying side issue in this simulation. The simulation is based on 40 simulated studies or test results. With power of 25% only about 10 of those are expected to be significant and useful for a z-curve analysis to estimate power. We have warned that z-curve estimates with small k (k = 10; 10 p-values below .05) are too variable to be meaningful. They also have wide confidence intervals that Reviewer A does not bother to report. However, this is a side-issue. We can simply increase the number of tests from 40 to 100,000 and see the large-sample bias in z-curve estimates. This confirms that z-curve severely underestimates power in this simulation. So, let’s take a closer look at the scenario that is being simulated.

The simulation starts with effect sizes ranging from small (d = .2) to effect sizes that are very large and rarely observed in real studies (d = 2.2). It is well known that power increases with larger effect sizes. Thus, to maintain low power of 25%, we have to reduce sample sizes. The sample sizes implied by this simulation are as follows.

N freq. perc.
2 11 26.8%
3 13 31.7%
4 4 9.8%
5 3 7.3%
6 1 2.4%
7 2 4.9%
9 1 2.4%
10 1 2.4%
12 1 2.4%
15 1 2.4%
20 1 2.4 %
28 1 2.4%
43 1 2.4%

These results show that only 10% of studies had sample sizes of 15 or more participants and over half of the simulated studies had 2 or 3 participants. As the simulation focused on a one-sample t-test, a study with N = 2 has 1 degree of freedom. This important information was hidden from the editor who is supposed to make decisions based on peer-reviewers’ comments.

Reviewer A could have just simulated studies with sample sizes of 2 or 3 participants to show that z-curve does not work well with these sample sizes, but maybe the editor would have noticed that this is not a reasonable assumption because most studies have more than 3 participants. In fact most studies have more than 20 participants. So, the only plausible reason to simulate effect sizes, when sample size is the driving factor is to hid explicit information about sample sizes from readers who do not understand t-distributions very well.

That being said, it is interesting to examine whether small sample sizes that are actually found in research articles still bias z-curve estimates. For example, John Bargh’s infamous elderly priming study that could not be replicated had only n = 15 participants in the control and experimental group for a total of N = 30 and 28 degrees of freedom. Would z-curve estimates underestimate power with these small sample sizes? Before I can present the results, it is important to point out that z-curve provides to estimates of power. One is the estimated power (including power of 5% for studies where the null-hypothesis is true) of all studies that were conducted and produced significant and non-significant results. This is called the expected discover rate. The second estimate is the power of the subset of studies that produced a significant result (including false positive results with power of 5%, if alpha is set to .05). This is called the expected replication rate because it predicts how many significant results would be obtained if only the studies with significant results were replicated exactly, including the original sample sizes. When power is fixed, the true EDR and ERR are the same, but estimates and biases can differ because estimating the EDR is more difficult.

We can use Reviewer A’s code to determine the effect size that is needed to get significance in a simple between-group study with n = 15 per group (Bargh also had a covariate which increases power, but that is not relevant here). I simulated power of 50%.

#Note. pwr uses n of a single group. with n=15 and type=two.sample, the total N=30 and df =28
d <- pwr.t.test(n = 15, d = NULL, sig = .05, power = .50,
                          type=”two.sample”, alternative=”two.sided”)$d

With N = 30 total sample size we need a large effect size of d = .74 to have 50% power.

Figure 1 shows the t-distribution from which test statistics of individual studies are sampled (green). It also shows the standard normal distribution that is implied by 50% power (black). Visual inspection shows that the two distributions are similar but not identical.

The approximation of the asymmetrical non-central t-distribution with the standard normal distribution introduces some bias in z-curve estimates. For this example, the true power of 50% is underestimated by 4 percentage point. More problematic is that the EDR is underestimated by 13 percentage points. This finding suggests that estimates of the EDR and the FDR, which is simply a transformation of the EDR, are biased in sets of studies with small sample sizes (N < 30), even if we disregard silly sample sizes of N = 2.

Is there a solution to this problem? Indeed there is one and maybe we should have thought about it before, but as they say “better late than never.” There is an alternative approach to ‘convert’ t-values into z-scores. (or F-values with df = 1, t = F^2, i.e., t is the square root of the F-value or F-values are simply squared t-values). The alternative is to simply use the t-value as an estimate of z-scores.

ChatGPT: How to convert t-values into z-scores?

Find the cumulative probability of the t-value: Use a t-distribution table or statistical software to find the p-value associated with the t-value given your specific degrees of freedom. This p-value represents the area under the curve to the left of your t-value (for a one-sided test) or half the area in a two-tailed test. Once you have the p-value, use the inverse of the standard normal distribution (Φ⁻¹) to find the corresponding z-score.

Alternative Formula (if df > 30): For degrees of freedom over 30, the t-distribution closely resembles the normal distribution, so you can directly approximate the z-score by using the t-value.

It is clear that this approach will lead to overestimation of power (EDR, ERR) because uncorrected t-values with small degrees of freedom are always larger than the corresponding z-scores. The question is how big this bias is when we use this approach to conduct z-curve analyses. Here are the results for the same data, but the input are the uncorrected t-values.

The results for the ERR are as expected: the true power is overestimated. Interestingly, the bias is as strong as for the transformation approach, but in the opposite direction. This suggests a possible way to quantify the amount of bias, by using both approaches and use half of the difference in estimates as an estimate of the amount of bias. The results for the EDR are a bit more surprising. There is no bias in the estimate. The reason is that bias is introduced by the wide tail of the t-distribution and this tail has a weaker effect on power for all tests. This is a very encouraging finding and suggests that it is preferable to use this approach to submit t-values to z-curve.

Large sample bias is often hard to detect when the set of studies is small. Moreover, confidence intervals of z-curve estimates are adjusted to allow for small systematic biases. It is therefore interesting to compare the coverage of confidence intervals for both approaches. To do so, I split the 200,000 t-values into 1,000 sets of 200 observations and ran z-curve with confidence intervals. I then checked the percentage of confidence intervals that included the true parameter.

For the ERR, 94.7% of confidence intervals included the true parameter. This is just 0.3 percentage points more than we would expect from a 95% confidence interval. For sample sizes greater than 30, this would imply good coverage. For the EDR, 97.7% of confidence intervals included the true power, indicating good coverage for the 95% confidence interval.

We will follow up on these preliminary results with more extensive simulations, but the results suggest that it is preferable to use t-values as estimates of z-values rather than using the transformation by means of p-values. We also suggest to limit z-curve analysis to studies with at least N = 30 participants for now.

Conclusions about Z-Curve

Z-curve was developed as a statistical tool that estimates the average power of a set of studies. It does so for two populations of studies. One population is all studies that were conducted independent of the result. The other population is the subset of studies that produced a statistically significant result. Alternative methods exist, but z-curve is the only method that can be used when studies differ in power (heterogeneity) without having to make assumptions about the distribution of true power (Brunner & Schimmack, 2020).

As all methods that rely on samples to make claims about populations, z-curve cannot reveal the true average power. It can only provide estimates of average true power in a population of studies. There are two sources of uncertainty in these estimates. One is ordinary sampling error. The other is systematic bias that can be introduced by approximating test-statistics from different designs with z-values. Z-curve provides confidence intervals that take both sources of error into account. Simulation studies suggest that 95% confidence intervals contain the true parameter at least 95% of the time in many realistic scenarios.

Reviewer A’s simulations showed that this is no longer the case when sample sizes are small. A simple solution to this problem is not to include studies with very small sample sizes in z-curve analyses. At present, I would suggest to exclude studies with N < 30 or to be mindful if studies with smaller sample sizes are part of the set of studies. I also recommend a new approach to include t-values and F-values with one degree of freedom. instead of converting them to p-values, t-values should be directly used as estimates of z-values.

Conclusions about Peer Review, Scientific Integrity, and Reviewer A

After presenting misleading claims about statistical power that we have carefully examined and shown to be misguided (Brunner, 2024), Reviewer A uses the results of their dishonest simulation study to claim that z-curve is unable to estimate the true power of a set of studies.

In sum, I do not think that the z-curve delivers estimates of the expected discovery rate (and its sister concept of expected replication rate) on a conceptual basis. The arguments for using estimates of “unconditional power” seem not to reasonably justify making a claim on the discovery rate of a set of publications (why not just count up the p-values if p-values signal discovery?). Even if my conceptual points are swept under the rug (again), perhaps the simulated illustration showing that z-curve does not provide an estimate near to the true value of the “expected discovery rate” would be convincing. Why does it break? Well, that goes back to the conceptual issues I have pointed out about consistency and efficiency of observed power, and the hodgepodge problem of combining lots of things together and hoping all the bad stuff averages out.  One might choose to argue that z-curve can run but cannot walk (that is, it performs well enough in a complex case but fails miserably in simple cases). I would not be convinced of such an argument that ignores first principles.

To be fair, Reviewer A states “I do not think,” which suggests that they are open to the idea that they may be wrong. However, it is unclear why reviewer A continues to ignore the evidence presented in peer-reviewed articles and the opinions of several reviewers who did not see the fundamental problems that they see.  When there is differences in opinions about factual statements, it is important to examine the underlying thought processes and evidence. Reviewer A failed to do so (“I do not think”). Rather, Reviewer A has set up a biased simulation that confirms their suspicion and accuses us of doing the very same thing they were doing. (If this sounds familiar, you know that it can be an effective strategy to deceive people).

Why does z-curve break in the simulations? As I have shown in this blog post, it breaks down in the simulations by Reviewer A when we use it with sample sizes of N = 2 or 3, it is still biased with sample sizes of N = 30, and it does well with large sample sizes. Reviewer A hides the missing moderator (a.k.a., hidden moderator), sample size, which they are well aware off because they knew hot to break z-curve with sample sizes of N < 10. However, they falsely generalize from this unrealistic scenario to all applications with larger sample sizes and ignore that many simulations with larger sample sizes have shown that z-curve performs well.

This deception is not different from the questionable research practices that original researchers sometimes use to present statistically significant results without any real effects (Bem, 2011). They know more than they are telling their audience to present misleading scientific evidence for false claims. Just like Reviewer A’s dishonest behavior, questionable practices in peer-reviews are not rare exceptions, but common occurrences that are based on human’s struggle to overcome their own biases. This would not be a problem, if there were an open exchange of arguments between authors and reviewers that works out the causes of disagreement. Here it is easy to show that the original simulation and Reviewer A’s simulations are both correct and that sample size is the moderator. Once this is clarified, the editor can decide whether simulations based on N = 2 or N = 100 are more relevant. However, often editors are quick to reject articles based on the expert opinions of reviewers, especially in flashy journals that pride themselves on high rejection rates. It is not surprising that the quality of articles in these journals is not better than in other journals because experts will use their power to favor articles that agree with their opinions and be hypercritical about articles that do not. It is well known that pre-publication peer-review is very subjective and reviewers often disagree even about ratings of the quality of a literature review.

How can we improve peer-review? The answer is simple. Make them open! Open science requires transparency of all steps of the production of a scientific article, which includes peer-review. Some innovative journals have implemented open peer-reviews. We are proud that two z-curve articles have been published in the leading journal Meta-Psychology (conflict of interest declaration: I am co-founder of this journal with Rickard Carlsson who has been main editor since its inception). Reviewer A’s opinions do not just clash with our own opinions. They are also inconsistent with reviewers of z-curve who put their name next to their reviews. In contrast, reviewers in legacy journals hide their identity from the authors and the public. Just like I challenged Reviewer A to present a simulation to break z-curve, I challenge them here to an open exchange about the ability of z-curve to estimate the true power of a set of studies. Open exchange of arguments in real time (like in a chess game) in front of an audience needs to be added to the open science practices. Let’s make a badge for that and I will be happy to earn a few of those.

In meta-psychology, social priming has become the posterchild of junk science. Researchers conducted many cheap studies with small samples and published the results only when they supported their predictions. After Nobel Laurate Daniel Kahneman published some of these results in his bestselling book “Thinking: Fast and Slow,” he became concerned about the robustness of these results. He send an open letter to the leading social priming research Bargh, asking for replication studies. Bargh and other prominent social priming researchers declined. However, many younger researchers answered the call and reported replication failures. Anybody familiar with the replication crisis in social psychology is well aware of these problems and would not cite social priming studies as scientific evidence unless the studies were preregistered and conducted with reasonable sample sizes to detect small real effects.

However, many psychologists and other social scientists seem to be unaware of the replication crisis or willfully ignore the fact that articles by leading priming researchers provide no credible evidence for the claims made in these articles. As a result, a decade of replication failures has failed to correct the scientific literature. This blog post uses money priming as an example.

The simple idea behind money priming is that some manipulation that makes people think about money will change their attitudes and behaviors in ways that make people more materialistic, selfish, and less altruistic. The original article by Vohs et al. (2006) published 9 studies to provide evidence for this claim. Ironically, an article with 9 successful studies should not make us believe in the effect because even well-designed studies with real effects will occasionally be unsuccessful; that is, produce a p-value above .05 (Schimmack, 2012). Thus, an article that features only successful studies tells us nothing about the actual effect because it is unclear how many attempts were made to get the 9 significant results (Sterling, 1995).

In response to the replication crisis, my colleagues and I have worked on statistical methods to detect selective publishing of confirmatory evidence, which is sometimes called a questionable research practice, but clearly violates the spirit of scientific integrity and undermines the credibility of science and the scientific community. I am focussing on money priming here because I have used Vohs et al.’s (2006) article to train students in bias detection (video). Even 9 studies are sufficient to show that the evidence in this article omits studies that failed to support the mone-priming effect.

I was curious to see whether criticism of priming research and concerns about money priming in general influenced citations of the article. A search in WebOfScience suggests that citations are decreasing. However, citations of psychology articles have been decreasing in 2023 in general and the article is still cited about 20 times a year, which is rather high for psychology. Clearly money priming is not dead yet, and criticizing the work is not akin to flogging a dead horse.

I then looked at articles to see how they cited Vohs et al. (2006). The article “Effects of money priming on sustainable consumption attitudes” caught my attention because the title suggested that it reports results of new money priming research. Indeed, the article reports the results of two successful studies. I will focus on those results a bit later, but first I want to focus on Table 1 in the article that carefully listed results of 19 money priming studies with sample sizes and test statistics (t-tests, F-tests). This is information is sufficient to examine the presence of selection bias in the broader money-priming literature.

I was even able to copy past the table directly into excel. I just needed to add the information about test results in a way that excel could use it and add a few formulas.

The top row shows the important information. 95% of the results were significant if “marginally significant” results (p < .10) are counted as successes. This is consistent with success rates in psychology journals since 1959 (Sterling, 1959). Could this just be due to the fact that money priming is real and that the studies had high statistical power; that is a high probability to get p < .05? The answer to this question can be found by computing the exact p-values for the various test-statistics and converting them into z-scores. These z-scores can be used to compute the observed power of each study based on the effect size estimate in a single study. It is well known that this information is not useful for a single study, but it becomes useful when we have sets of studies and can compute the average power. The average observed power is 66.9%. Thus, we should have gotten about 67% significant results, not 95%. However, observed power is just an estimate of power. Maybe power is really higher? The problem with this argument is that observed power is inflated when selection bias is present. So, the 70% estimate is an overly optimistic estimate and the true average power of the studies is likely to be lower. How much lower? A simpel way to estimate the true average power is to subtract the difference between the success rate and average observed power because the inflation of the estimate increases the more selection bias there is. With an inflation of 94.7% – 67% = 25%, the estimated true power is only 40%. Whether this is sufficient evidence to wonder about the number of studies that failed to show the effect and were not reported is of course a subjective judgment. As the saying goes, a sucker is born every minute.

My colleagues and I have developed a more powerful method to examine these kinds of data called z-curve (Bartos & Schimmack, 2022; Brunner & Schimmack, 2021). The method fits a model to the distribution of the z-values implied by test statistics. Visual inspection of the distribution also provides clear evidence that the distribution of z-scores could not have been produced by random sampling error. It is just not possible to get so many results that are just significant (z = 2 implies p = .05) and no non-significant results.

Based on z-curve, we can estimate a number of statistics. The estimated replication rate (ERR) predicts how many of the 95% studies with significant results would produce a significant result again if these studies were repeated exactly with the same sample sizes. It is like asking Bargh to do his study again and show us that he can get the same results again. When asked to do so, he said nobody would believe him anyways if he would report significant results again. The ERR estimate of 35% is less important than the 95% confidence interval around this estimate. The predicted success rate in a set of exact replication studies could be as low as 10%. It could also be as high as 64%, but we just don’t know how replicable these results are. Thus, the 18 (out of 19) significant results tell us practically nothing about the ability to replicate these results.

The other statistic is the expected discovery rate. The expected discovery rate is an estimate of the percentage of significant results that we would find if researchers had kept all of their results and we could find the missing results and see how many significant results we get. The point estimate is very low. We would expect 5% just by chance alone. To get 7% is hardly better than chance. Again, we need to consider the uncertainty in this point estimate. It could be up to 17%, which would imply that researchers get about 1 significant result in 5 attempts (20%). Would you trust a researcher who hides 80% of their results? Moreover, it is also possible that the EDR is 5%, which is chance level. This means that all significant results were obtained by chance alone without any real money priming effect.

To be clear. This is not my data. I didn’t select studies or code studies. The data come from true believers in money priming who used these data to motivate their own studies.

Despite the wealth of adverse effects documented in the literature on money priming in relation to specific behaviors (as outlined in Table 1), there is a noticeable gap in research regarding the impact of money priming on sustainable consumption. (Koruk & Mandrik, 2024, p. 309).

I only plugged the data into stats programs that can look beyond the evidence we see to see whether we can trust the evidence and the answer is that these 19 studies provide no convincing evidence that money priming caused the significant results in these 18 studies. It could just have been chance and selective reporting of significant results.

Now take a moment and predict the outcome of the new studies in this article. Money priming was manipulated with a scrambled sentence task that made participants rearrange words that were either related to money (experimental group) or not (control group) (Exp1) or a paragraph writing task and a picture of money.

The outcome variabel was the average rating to the following three items.

(1) I am concerned about wasting the resources of our planet.
(2) I will make an effort to use products that do not harm the environment.
(3) It is important to change my consumption patterns (use less or avoid buying products) in order to protect the environment

Scroll down to see the results.

Results

Experiment 1 Effect size d = 1.44, z = 9.73
Experiment 2 Effect size d = 1.55, z = 9.18

Let’s just say that these results are very surprising. The effect sizes are very large for results in psychological research in general and the priming literature specifically. A difference of 1.5 standard deviations is as big as the difference in height between men and women.

Due to these large effect sizes, the test-statistic is of the chart. Z-scores of 9 have a probability of 1 out of a gazillion to occur by chance. These are not chance finding. These results are also not consistent with the evidence in Table 1, which showed much lower z-scores for most studies.

There are many possible explanations for these surprising results that include computational errors, demand effects among others (wink). I don’t really care about these results because priming studies are problematic even if they show real effects. First, the manipulation is artificial and may not correspond to real world situations in which we think about money. Second, ratings on a scale do not imply that people would really change their actual behaviors. However, it might still be an easy study to replicate and see whether the results are replicable. Even smaller samples would be sufficient to produce these results again, given the strong effect sizes reported in this article.

The main point of this blog post is that we need to look at results in published articles differently. We cannot just see how many significant results we see in journals. We already know the answer to this question. The published literature tends to have over 90% significant results. This is not an empirical finding that can be used to evaluate evidence. The real question is always how many non-significant are missing. Bias tests can be useful to answer this question. Thus, if you want to be a scientist and make scientific claims you need to examine the amount of bias in the studies that you are using. “Studies show…” is not a scientific claim. Studies also show that extraverts can sense pornography before it is even presented (Bem, 2011). The real question is how many studies really show an effect.

One of the greatest meta-psychologists was Jacob Cohen. He was concerned about the risk that psychologists might waste resources on studies that had a low probability to provide evidence for a true hypothesis. Following Neyman and Pearson this error is called a type-II error. It can also be called a false negative result.

Psychologists typically rely on null-hypothesis testing to provide evidence for their predictions. They set the criterion value for a statistically significant result to 5%. This means that there is only a 5% probability to get a significant result without a real effect. This is called a type-I error or a false positive result. In this approach, a type-II error occurs when a prediction is true (a treatment is effective), but the p-value is above .05.

Cohen (1961) warned psychologists that many studies have a high risk of producing false negative results, especially when effect sizes are statistically small. Even when effect sizes are statistically around the average effect size in psychological studies, the risk of a false negative result was about 50%. Follow-up studies showed that this situation had not changed in the following decades (Sedlmeier & Gigerenzer, 1989).

One might assume that psychologists simply have little control over the false negative risk in their studies. However, that is not true. A simple way to decrease the false negative risk is to increase sample sizes. Thus, one has to wonder why psychologists did not increase sample sizes in response to evidence that they are conducting studies with high risk of a false negative result?

Imagine a gambler who can play two slot machines. One has a 50% chance of winning, the other one has an 80% chance of winning? Which machine would you pick? The answer is obvious. The situation for a researcher is a bit different. Fist, they have to pay more (invest more resources in larger samples) to play the higher odds of winning (i.e. avoiding a false negative result). Second, they do not know the actual odds of winning. They merely know that the odds of winning are higher when they invest more resources. Cohen (1988) tried to help researchers to make decisions that reduce the false negative risk without paying too much for larger sample sizes. It took 50 years for power analyses to become more popular in psychology in the past decade.

While better control of false negative results may seem desirable to all, a recent peer-reviewed article by Pek, Pitt, and Wegener (2024) suggests that power analyses are useless. They claim in the title “Uncertainty limits the use of power analysis.” In the article, they wonder “Isn’t use of power better than not using power at all?” and their answer is not a simple “yes” (p. 11). They say it is also not a simple “No”, but they provide no examples where power analysis is better than drawing a random number from a hat to determine the sample size of a study. In fact, they go on to state that “we recommend that researchers place limited confidence when using power to design experiments, or not use it at all as a direct justification for determining N” (p. 11). If that does not mean “power analysis is useless”, they do a very good job at hiding the benefits of power analysis.

It is remarkable that such a harsh criticism of Cohen can be published in a leading psychology journal without even mentioning Cohen’s work. It is also remarkable that they never mention false negative results / type-II errors, although power is defined as the probability of avoiding a type-II error (beta = type-II error, power = 1 – beta). So, we do not know what Pek et al.’s (2024) suggestion for researchers is when they get a non-significant result. Maybe somebody should write to them. “Hey I just did a study with a randomly generated sample size and got a non-significant result. What now?”

Two other giants in the history of psychology wrote an article in 1971 about the problem with small samples that often have a high false negative risk (Kahneman & Tversky, 1971). They wrote “We refuse to believe that a serious investigator will knowingly accept a .50 risk of failing to confirm a valid research hypothesis.” (p. 110).

What Pek et al. do not tell there readers is the real reason why psychologists ignored false negative results and continued to use small samples. The reason is not that they rarely have false negative results. The reason is that they invest relatively few resources in their studies so that they can conduct many studies or many tests within a study to get at least one significant result. The non-significant results are simply discarded. This is known as using questionable research practices because researchers are not disclosing all results. This increases the risk that the published results are false positives. If a researcher tests 20 false hypotheses, they can expect to get one p-value below .05. If they do not disclose that they ran 20 tests, readers cannot see that the 1 significant result was expected by chance alone.

Pek et al. also do not tell readers why power analysis has become more popular in the past decade. The reason is that a high rate of false negative results makes significant results less informative. Imagine that researchers test 100 true hypotheses and 100 false hypotheses. The 100 false hypotheses are expected to produce 5 significant results. This is implied by the use of the 5% criterion. If the 100 true hypotheses are tested with only 10% power, we have 10 true findings and 90 false negative results. Now we publish only the significant results, which means there are 15 results, 5 are false findings (a medicine does not work) and 10 are true findings (a medicine works). This means one third of published findings are false. Cohen recommended to plan studies with 80% power. This would mean we get 80 true findings and 5 false findings. As a result, only 6% of the published results are false. Would you still believe Pek et al. (2025) that power analyses are useless or would you rather wonder whether the average power in psychology is closer to 10% or 80%?

The key argument in Pek et al.’s article is that power is a hypothetical construct because we do not know whether the predicted effect is small, medium, or large. First of all, this is not true of all science. Some sciences have theories that make quantitative predictions. Even psychologists may have some idea whether they are testing a weak, moderate, or strong effect. However, we do not even need to know what the true effect size is. We can conduct a power analysis based on an effect size that is theoretically interesting. For example, the question whether money buys happiness or not is a silly question. A more interesting question is how much happiness money can buy. Let’s say that money is only important for a theory of happiness, if the correlation between money and happiness is at least r = .1, what Cohen calls a weak effect. Power analysis not only helps us to determine a reasonable sample size to look for this correlation, it can also help to make non-significant results informative. For example, if we power a study to have a 95% chance to be significant with a correlation of r = .1, and we obtain a non-significant result, the chance that this result is a false positive result is less than 5%. We may therefore be willing to accept the hypothesis that the correlation is less than .1 and conclude that money has a negligible influence on happiness (BTW the true correlations tend to be between 1. and .3). This is valuable information that can only be obtained by considering the risk of a false negative finding. Finding a non-significant result in a study with N = 20 people does not warrant the conclusion that money does not matter much for happiness because the false negative risk is too high. Pek et al. (2024) ignore all of this useful information that power analyses can provide even if there is great uncertainty about the true power of a study. Thus, researchers can easily track the power of their studies by keeping track of their success record.

If a researcher conducts 20 statistical tests and finds only 4 significant results, the average power is about 25%. According to Kahneman and Tversky (1971) any serious researcher would have to wonder whether they are just testing a lot of false hypotheses or whether they produce a lot of false negative results. No serious researcher should just continue doing what they are doing and just publish the 4 significant results and call it a day. However, that is what social psychologists like Pek’s co-author Duane Wegener have been doing for decades, while ignoring power analyses. This has led to the replication crisis in social psychology that has uncovered many false findings. At least Noble Laureat Daniel Kahneman had the humility to recognize his mistake. “What the blog gets absolutely right is that I placed too much faith in underpowered studies” (Kahneman, 2017).

Kahneman (2017) also points out that we need science to make new discoveries and to correct false beliefs, but that science can only serve this function when all relevant results are published. And that was not the case in social psychology. Non-significant results were ignored and only significant results that confirmed even the most implausible predictions were published. This bias is evident in the high percentage of significant results in psychology journals (Motyl et al., 1997; Sterling, 1959; Sterling et al., 1995). With success rates of 90%, honest reporting would imply that psychologists only test true hypothesis with high power. Ironically, this would mean that psychologists do not need power analysis because they miraculously never obtain false negative results. The real reason for 90% success rate is rather different. A replication project found only 25% significant results in replication studies of social psychologists (Open Science Collaboration, 2015), suggesting that most studies are well below the 50% criterion for serious researchers (Kahneman & Tversky, 1971). These are well known facts that Pek et al. (2024) and the editor who published this article simply ignore and are hiding from readers who are not familiar with the history of power analysis.

Finally, Pek et al.’s (2024) concern about the uncertainty about the true power of a study is irrelevant for the usefulness of power analyses. The true power of a study is less important than the truthful reporting of results. Uncertainty about true power implies that even researchers who conduct power analyses will sometimes conduct studies that produce false negative results. First, Cohen’s recommendation to aim for 80% power implies that 20% of tests of a true hypothesis will produce false negative results. Second, power analyses can overestimate true power and the false negative risk can be even greater than 20%, This is not a problem if the results are published and combined with other evidence that can correct false negative results. This is what researchers in medicine do. Here, studies have only about 30% power on average, but non-significant results are reported and meta-analyses can reduce the risk of false conclusions. Thus, the biggest threat to psychology as a science is uncertainty about the honest reporting of results and not uncertainty about true power. The advantage of conducting power analyses is that it is more likely that we have honest and credible evidence when researchers conduct a few studies with high power than many studies with low power. This is what Cohen meant when he said “Less is more, except for sample size.”

In conclusion, if you are new to statistical power and its role in psychological science, I recommend to read Cohen (1988, 1992) and to ignore Pek et al.’s (2024) useless article. A simple truth about power is that the percentage of significant results in a set of studies is an estimate of the mean power of studies. If you see a set of studies with over 90% significant results, you have to ask yourself: did these studies really test only true hypothesis with high power, or did researchers not report studies that failed to support their claims ” (Schimmack, 2012). I trust you to come to the right conclusion, but you can also use power calculations to test for the presence of selection bias. But that is a story for another day.