Pek, J., Hoisington-Shaw, K. J., & Wegener, D. T. (2024). Uses of uncertain statistical power: Designing future studies, not evaluating completed studies.. Psychological Methods. Advance online publication. https://dx.doi.org/10.1037/met0000577

In their latest article, Pek et al. continue to spread misinformation about the use of average power to estimate the replicability of a set of published hypothesis tests. The strong claim that they are spreading misinformation is not based on my own biased assessment of their article. It is based on a logical analysis by a neutral third party: ChatGPT after I presented some counterarguments in a discussion with ChatGPT. You can read the full discussion here.

Below is a ChatGPT generated summary of the discussion. It reflects poorly on Pet et al. that their arguments do not hold up under scrutiny by an AI. Their false arguments either reflect willful ignorance or a level of intelligence that is below that of an AI. On the other hand, the capabilities of ChatGPT to reveal bullshit suggest that psychology may advance by incorporating AI in the review process.

Summary of the Discussion on Average Power and Replicability

The discussion critically examined the arguments against using average power to evaluate published research, particularly as presented by Pek et al. (2020). The key points and conclusions are as follows:

1. The Ontological Error Argument is Invalid

🔹 Claim by Pek et al.: Computing average power from completed studies is an “ontological error” because probabilities should only apply to future experiments, not past data.
✅ Counterargument:

• This claim is incorrect because inferential statistics routinely estimate probabilities from observed data.
• Z-curve and similar methods use observed effect sizes to estimate a population parameter, which is standard statistical practice.
• Applying probability models to past data (as done in Bayesian inference and meta-analysis) is not a logical error.

🔹 Conclusion:
🚫 The ontological error claim is misleading and does not justify ignoring average power estimates.

2. Average Power Can Predict Replication Outcomes (When Used Correctly)

🔹 Claim by Pek et al.: Average power should not be used to estimate replicability because it does not predict replication success.
✅ Counterargument:

• If replication studies are conducted under the same conditions (same sample size, design, effect size), average power should correspond to expected replication success.
• In reality, selection bias inflates effect sizes in original studies, leading to overestimation of replication rates.
• Z-curve corrects for this bias and provides expected replication rates (ERR), making it a useful predictor of replication success under realistic conditions.

🔹 Conclusion:
🚫 The argument that average power does not predict replication is too strong and misleading.
✅ Bias-corrected estimates (like Z-curve) provide meaningful replication estimates.

3. The Argument About Imprecision is Overstated

🔹 Claim by Pek et al.: Average power estimates are too uncertain to be useful.
✅ Counterargument:

• All statistical estimates have uncertainty, including effect sizes, confidence intervals, and meta-analytic results.
• Z-curve provides confidence intervals to account for this uncertainty.
• The precision of estimates depends on sample size (number of studies, k).
• Small k (e.g., <30 studies) → more uncertainty.
• Large k (e.g., 100+ studies) → reliable power estimates.

🔹 Conclusion:
🚫 Saying power is “too uncertain to use” is incorrect—uncertainty is expected in all statistics and can be quantified with confidence intervals.

4. Publication Bias is a Real Problem, But Z-Curve Adjusts for It

🔹 Claim: Publication bias distorts average power estimates.
✅ Counterargument:

• Yes, publication bias inflates power estimates if uncorrected.
• Z-curve accounts for this bias using the distribution of z-scores.
• The Expected Discovery Rate (EDR) helps detect selective reporting.

🔹 Conclusion:
✅ Publication bias is an issue, but Z-curve explicitly corrects for it, making it more reliable than naive power estimates.

5. Should Z-Curve Be Ignored?

✅ Final Verdict:

• Z-curve is a useful tool for evaluating replicability.
• The arguments by Pek et al. are either incorrect, misleading, or overstated.
• Z-curve’s expected replication rate (ERR) provides valuable insights into research credibility.
• Publication bias correction makes Z-curve one of the best available methods for evaluating replicability.

🚫 Pek et al.’s recommendation to ignore Z-curve is not justified.

6. Could I Have Been “Tricked” by the Discussion?

✅ Highly unlikely.

• My conclusions are based on logical reasoning, empirical evidence, and statistical best practices.
• The ontological error argument is demonstrably false.
• Power-based replication estimates are widely used (e.g., Ioannidis & Trikalinos, 2007; Francis, 2012).
• Uncertainty in power estimation is expected and accounted for in proper statistical methods.

🚫 Pek et al.’s critiques do not hold up under scrutiny.

Final Thoughts

Z-curve is a valuable method for assessing replicability, and dismissing it based on the arguments by Pek et al. would be a mistake. The discussion confirms that power estimation, when properly corrected for bias, remains a meaningful and useful tool for evaluating research credibility.

Meta-science uses scientific methods to study science and meta-psychologists study psychological science. Although meta-psychology emerged only recently, psychologists have reflected on their science since its beginning (Cohen, 1961; Tversky & Kahneman, 1971).

From 1960 to 1990, the most influential meta-psychologist was Jacob Cohen. Cohen made two important observations about statistical practices in psychology. First, Cohen observed that psychologists often conducted studies without considering the a priori probability that their study would produce the correct result; that is, a statistically significant result that is based on an actual relationship between two variables. Apparently, they were as happy to conduct studies with a 20% chance of getting significant results as doing studies with an 80% chance of a significant result (Tversky & Kahneman, 1971). Second, Cohen conducted the first meta-analysis of statistical power in psychology that suggested an average power of 50% (Cohen, 1961). This finding has been replicated in other studies and power does not seem to have increased from 1960 to 2010 (Maxwell, 2004; Sedlmeier & Gigerenzer, 1989; Schimmack, 2012; Smaldino & McElreath, 2016).

In the 1990s, the American Psychological Association assembled a task force to examine statistical practices in psychological science. A key concern was the high number of false negative results. In response to this concern, Psychological Science tried to abandon the use of statistical significance as a criterion for publication (Cutting, 2006). Although another solution to the problem would have been to demand higher-powered studies, power remained neglected (Maxwell, 2004).

The 2010s saw a dramatic shift in meta-scientific concerns about research practices in psychological science. A new concern was that that many hypotheses in psychology might be false; that is, the nil-hypothesis is actually true, and that many published results are false positive results. A similar concern had been raised a bit earlier for medical research (Ioannidis, 2005), but psychologists only started worrying about false positive results in 2011. The crisis of confidence was triggered by a social psychological article that claimed implicit priming effects work even when the prime is presented after the behavior (Bem, 2011). Few psychologists were willing to accept time-reversed causality and considered Bem’s results to be false positive results. This sensational finding triggered questions about the credibility of other findings that were produced with the same research methods. Suddenly, a major concern was that many, if not most published results are false positive results.

Concerns about false positive results were stocked by demonstration that a few statistical tricks may help researchers to produce significance when the nil-hypothesis is true (Simmons, Nelson, & Simonsohn, 2011). These tricks are called questionable research practices because the undermine the objectivity of data analysis (John et al., 2012). Concerns about false positive results are intimately tight to concerns about questionable research practices because the propoer use of significance testing produces significant results in only 1 out of 20 attempts and if the sign has to be consistent, it only produces significant results in 1 out of 40 attempts with the standard criterion of alpha = 5%, two-sided (z = 1.96). It seems implausible that a single researcher could conduct 40 studies to produce 1 significant result. Even if independent researchers are testing the same hypothesis, it is implausible that a field with dozens of significant results in a meta-analysis could have produced these results, if the nil-hypothesis was true. This logic is consistent with Rosenthal’s fail-safe N statistic for meta-analysis. The statistic computes the number of non-significant results that would be required to nullify a significant meta-analytic result. Typically, fail-safe N is much higher than any reasonable number of studies would be. Thus, it is irrational to postulate a high false positive risk without assuming the use of questionable research practices that increase the chance of producing false positive results. In sum, the 2010s saw a shift in concerns about research practices in psychological science. The new concern was that psychological science is not very different from other human attempts to understand the world.

“Instead of a naïve scientist entering the environment in search of the truth, we find the rather unflattering picture of a charlatan trying to make the data come out in a manner most advantageous to his or her already-held theories” (Fiske & Taylor, 1984).

The past decade has demonstrated conclusively that questionable research practices inflate the percentage of significant results in psychology journals. First, statistical comparisons of power and success rates show that success rates exceed statistical power (Francis, 2012; 2014; Schimmack, 2012; 2020). Second, success rate of replication studies without QRPs are lower than success rates in original articles (Open Science Collaboration, 2015). Third, success rates in published articles based on dissertations are higher than success rates in the dissertations (reference). Thus, there is no doubt that questionable research practices are used and increase the risk of false positive results. However, it remains unclear whether QRPs merely inflate true effect sizes or whether they often produce false positive results.

At present, concerns about a high rate of false positive results in psychology rests mostly on speculations. For example, Smaldino and McElreath (2016) speculate that most effect sizes are small, d = .2, which leads to the assumption that power is low (24%), and that this explains why “false discoveries are common” (p. 5). However, with low power it is impossible to distinguish between true small effects and true nil-hypotheses and there is no empirical evidence to support the claim that false discoveries are common. The key problem with these speculations is that it is practically impossible to distinguish between false hypotheses where the nil-hypothesis is true and true hyptheses, where the effect size is very small and may not be practically significant (Brunner & Schimmack, 2020). Thus, even a high rate of replication failures does not warrant the conclusion that most published results are false positives (Schimmack, 2020).

Dreber et al. (2015) speculated that the false discovery risk is high because psychologists often test false hypotheses. Based on a prediction market of replication outcomes they estimated that only 9 out of 100 hypotheses in psychology are true. Along with low power, this would suggest that the actual number of discoveries in psychological laboratories is below 10% (90 * .05 + 10 * .50 = 9.5 out of 100). That is, whenever a psychological researcher presses a key to get a p-value, 9 out of 10 times the result is a p-value greater than .05. Researchers then use QRPs to inflate the rate of significant results in publications to 95% (Sterling, 1995; Sterling et al., 1995).

Without strong empirical data, meta-scientists are at risk to be no different than other scientists. That is, they will fit their assumptions and simulations to predict the state of affairs they believe to be true. To avoid this problem, I am presenting the results of objective analyses of empirical data. The empirical data are test results from articles published in psychology journals that cover a wide range of psychological disciplines. The dataset also covers all years from 2010 to 2020, which makes it possible to examine any changes in research practices over the past decade. The test statistics were automatically extracted from text files of the published articles. This ensures that the coding of articles is objective (that is not influenced by my own motivated biases) and a large sample size, which is helpful to obtain precise parameter estimates. The method also has some limitations that are discussed in detail in the discussion section, but the dataset is superior to previous attempts to estimate false discovery risks in psychology. For example, Dreber et al.’s (2015) estimate was based on 44 tests. In contrast, the present analysis is based on over 1 million test statistics.

To provide quantitative information about the use of questionable research practices, power, and false positive risks, I used a new statistical tool called z-curve.2.0 (Bartos & Schimmack, 2021; Brunner & Schimmack, 2020). To standardize information from various test statistics, z-curve first transforms test statistics into exact p-values (or log p.values), and then converts the two-sided p-values into absolute z-scores. For example, a p-value of .05 (two-sided) is converted into a z-score of 1.96. Each observed z-score has two components. One component is the non-central z-score that is expected based on the sample size and effect size of a test. The other component is sampling error. In reality, each z-score can have a different non-central z-score. However, the distribution of z-scores can be approximated with a finite mixture model that has only a few non-central z-scores as components. Z-curve approximates the observed distribution of z-scores by assigning different weights to each component to optimize fit between the observed distribution and the distribution that is predicted by the z-curve model. Each component of the model corresponds to power to reject the nil-hypothesis. For example, the component z = 2 has approximately 50% power to reject the nil-hypothesis. The key parameters that are estimated by z-curve.2.0 is the average power of all studies (before selection for significance) and the average power of studies with a significant result (after selection for significance). The focus here is on the estimate of power before selection for significance. This parameter is called the estimated discovery rate (EDR).

The EDR is crucial to answer three questions about research practices in psychology. First, the EDR is an estimate of the typical power of psychological studies. This power estimate is different from a priori power estimates that postulate a (non-zero) effect size. Here power is an estimate of average power across a set of studies, not power of a single study. Moreover, because it is not possible to determine the number of false hypotheses that were tested, the average also includes an unknown percentage of studies with power equivalent to alpha. Thus, power is not conditioned on the presence of an effect. To distinguish it from the typical use of power, I call it unconditional power. Unconditional power provides an estimate of the amount of times psychologists press a key and get a significant result, p < .05. Unconditional power can be low for two reasons. Either psychologists test many false hypotheses and power is 5% or they test many true hypotheses with low conditional power.

An estimate of the discovery rate can be used to quantify the influence of questionable research practices that inflate the observed discovery rate. For example, if psychology journals publish 95% significant results with only 20% unconditional power, we see a large discrepancy between observed power and estimated power that can only be explained with QRPs. However, if the observed discovery rate is 60% and the estimated discovery rate is 50%, the influence of QRPs is milder. Finally, if the observed discovery rate matches the expected discovery rate, there would be no evidence that QRPs were used. Thus, the discrepancy between observed and expected discovery rates provide quantitative information about the use of QRPs.

The estimated discovery rate is also useful to assess the risk that published significant results are false positive results. The reason is that the risk of false positive results decreases as the discovery rate increases (Soric, 1989). Figure 1 shows the relationship between the discovery rate and the maximum rate of false positive results, which I call the false discovery risk.

The relationship is non-linear. With just 10 discoveries, the false discovery risk is 50%. Thus, claims that most published results are false require fewer than 10 out of 100 significant results (Dreber et al., 2015; Ioannidis, 2005). On the other hand, if just over 50% of all tests are significant, the risk that significant results are false positives is only 5%. In contrast, to speculations about the number of true or false hypotheses that are being tested, Soric’s focus on observable discovery rates makes it possible to study false discovery risks empirically. The only proble is that observed discovery rates in psychology journals are not credible estimates of the actual discovery rates becuase they are inflated to an unknown degree by the use of QRPs. Z-curve makes it possible to answer questions about QRPs and false discovery risk because it provides estimates of unconditional power that correct for the influence of QRPs.

Data

PDF files of published articles from 120 psychology journals (journal list) from 2010 to 2020 were downloaded and converted into text files using the commercial software PDFzilla. The text files were searched for F-tests, t-tests, z-tests, and chi-square test statistics that were reported in the text. Results presented in other forms or in tables were not included. Chi-square tests were limited to tests with up to 5 degrees of freedom to exclude mode fit results from structural equation modeling studies. Results from t-tests that were reported without degrees of freedom were treated as z-tests. The data are available to reproduce the results (data.file).

Results and Discussion

Figure 2 shows the z-curve plot for all test statistics. First, the percentage of results that are significant with alpha = .05 (two-sided) is 69%. This is notably lower than previous estimates (Sterling, 1959; Sterling et al., 1995). The main reason for this discrepancy is that journal articles are often written around a significant result as the key finding (Kerr, 1998). Thus, non-significant results are not entirely missing from published articles, but the focus is on significant results because they can be interpreted, whereas non-significant results are inconclusive in the standard nil-hypothesis framework.

The second observation is that the EDR is 47% with a fairly tight confidence interal ranging from 42% to 53%. This finding is consistent with Cohen’s assumption that most studies in psychology test true hypotheses and that average power to detect true effects is around 50%. In contrast, an EDR around 50% is inconsistent with previous claims that average power is 23%, which limits the EDR to 23% if only true hypotheses were tested. An EDR of 47% implies that the false discovery risk is limited to 7%, with a fairly tight confidence interval ranging from 5% to 7%. This finding is inconsistent with Ioannidis’ s speculation that most published results in science are false. A similar analyses for medical journals also failed to support Ioannidis’s prediction (Jager & Leek, 2014). Finally, the results are inconsistent with Deber et al.’s estimate that psychologists test only 10% true hypotheses, which limits the predicted EDR to 14.5.

The third observation is that the estimated discovery rate is only 47%. Given the large sample size, the 95%CI around this estimate is fairly tight ranging from 42% to 53%. The discrepancy between the observed discovery rate and the estimated discovery rate shows clear evidence that the success rate in journals is inflated by QRPs. The discrepancy is not as large as some have feared (Simmons et al., 2011), but it is not trivial either. These results corroborate the findings from anonymous survey studies that QRPs are being used to produce significant results (John et al., 2012; Nosek et al., 2021).

In conclusion, a z-curve analysis of statistical results in psychology journals provides some clarification about the prevalence of false positives and false negative results in psychology. The results strongly support the concerns of the 1990s that psychological studies are often underpowered and bear a high risk to produce false negative results. In contrast, the risk of false positive results is relatively small. Psychologists are much more likely to miss detecting statistically small effects than to produce significant results without a true effect. Before the implications of these new findings are discussed in detail, it is important to examine potential moderator variables.

Time Effects

To start the analyses in the year 2010 is arbitrary. With some additional work it is possible to extend the dataset to previous years. However, it is unlikely that results have changed in the time before 2010 because critical reflections on research methods showed little effect (Sedlemeier & Gigerenzer, 1989; Smaldino & McElreath, 2016). The more interesting question is whether research practices changed in response to the crisis of confidence (Nosek et al., 2021). The present study provides the first empirical answer to this question. To do so, I estimated the ODR and EDR separately for each year.

The results in Figure 3 show that ODR decreased over time and EDR increased over time. The ordinal trends are highly significant, p < .001. An analysis with quadratic trends suggests that the increase in the EDR is not linear. Visual inspections suggests that EDR did not change much from 2010 to 2015, but did increase after 2015. This would be consistent with a delayed response to the credibility crisis that emerged in 2011.

In absolute terms the changes are not that impressive. The ODR decreased from 72% to 68%. The EDR increased from 45% to 53%. Thus, the general results hold even for 2010. Average power was around 50% and the false discovery risk was only 6%.

Discipline as Moderator

The credibility crisis has focussed heavily on social and cognitive psychology (Open Science Collaboration, 2015). In contrast, research where recruitment of participants is difficult and costly has seen fewer replication studies. As a result, little is known about variability in the use of QRPs and false discovery risks across scientific disciplines.

I first focus on the major areas that have multiple journals, namely social psychology (k = 21), cognitive (k = 16), developmental (k = 14), clinical (k = 14), biological (k = 13), personality (k = 6), and applied (k = 17).

The results show that all disciplines use QRPs as the ODR is always significantly higher than the EDR. The confidence intervals are 83% CI to allow easier comparisons of EDR’s across disciplines, but the typical 95%CI also never included the ODR. There is statistically significant variation in the ODR which are estimated with high precision, but in terms of effect sizes the variation is small ranging from 65% for Clinical to 72% for Personality. Variation in the EDR is more substantial, ranging from 31% for Applied to 54% for Cognitive Psychology. Most confidence intervals overlap, but cognitive psychology has a higher EDR than social psychology and applied psychology. In addition, applied differed significantly from personality and development.

The result qualitatively validate the finding in the reproducibility project that results in cognitive psychology that replication studies without QRPs produced more significant results for cognitive psychology than social psychology. The present results also suggest that this difference is driven by both disciplines. Social psychology is less replicable and cognitive psychology is more replicable than other fields.

However, quantitatively the present results are not consistent with the OSC reproducibility project. While the success rate for cognitive psychology (50%) is in line with the EDR estimate of 54%, the success rate for social psychology was lower (25%) than the EDR (40%). A number of factors can contribute to this discrepancy, including the low number of studies in the reproducibility project and some problems with conducting actual replication studies in social psychology (van Bavel et al., 2016; Inbar, 2016). A larger sample of actual replication studies is needed to explore this discrepancy further.

Overall, the results show only modest heterogeneity in research practices across disciplines in psychology. The most notable exception are results published in applied psychology journals. This is a novel and concerning finding as applied research has more impact on individuals’ lives. However, even for applied psychology results the 95%CI for the false discovery risk ranges from 6% to 17%. Thus, there is no indication that most published results in psychological science are false in any of the major disciplines.

Time by Discipline Interactions

Most of the discussion about research practices has occurred among social psychologists (Nosek et al., 2021; Schimmack, 2020), and social psychologists have been overrepresented in the Open Science movement. It is plausible to assume that changes in research practices among social psychologists disproportionally contribute to the overall increase in the EDR. It is less clear, whether other disciplines have changed over the past decade. To answer this question, I also examined time trends for each of the major disciplines.

All disciplines, except cognitive psychology, show an increasing trend. Confidence intervals are not shown because they are too wide to show annual differences. Another way to compare time trends of disciplines is to regress the EDR and to compare the slopes.

The wider 95%CIs show that all disciplines except cognitive and developmental psychology improved. However, the CIs are wide and it is possible that some increases occurred that are not yet statistically reliable. Using the narrower 83% confidence intervals shows that only the large increases for social and personality psychology are statistically significantly different from cognitive psychology. In terms of practical significance, the changes for social and personality psychology are meaningful and show some robust improvements in these two disciplines.

The next figure shows the results after EDR scores are transformed into FDR scores. the results are predicted scores based on linear regression of EDR scores on year. The regression was carried out on EDR scores and predicted scores and confidence interval boundaries were transformed to FDR scores. The most important finding is that the false discovery risk is much lower than meta-scientists feared (Ioannidis, 2005; Simmons et al., 2011). Even in social psychology in the 2010s, the FDR is only 14% and the upper limit of the 95%CI is 18%.

In 2020, several areas have point estimates below 5%, but the 95%CI include 5%, making it impossible to conclude that the FDR in any area is below 5%. However, all areas except applied psychology have an FDR below 10%. Overall, these results are consistent with Cohen’s assumption that psychologists infrequently test true nil-hypotheses.

Adjusting Alpha to Control the Journal-wide False Discovery Risk

In the final empirical part of this article, I focus on one specific journal to illustrate how z-curve analyses can be used to examine research published in a single journal. The focus on journals is useful because journal editors have control over the research that is published in their journals. So far, most editors have treated all p-values below .05 as equivalent. However, z-curve analyses of significant p-values shows that not all p-values below .05 are equal. Some provide stronger evidence for a hypotheses than others, and while some p-values that are just significant are expected, too many just significant results are a sign that QRPs were used (Lindsay, 2013 editorial).

To illustrate the use of z-curve to assess individual journals I picked the journal Psychological Science for several reasons. First, it is the flag ship journal of the Association for Psychological Science that publishes articles from all areas of psychology. Second, the journal was chosen for the reproducibilty project and showed a low success rate in replication attempts without QRPs (Open Science Collaboration, 2015). Third, the journal has responded to the crisis of confidence by implementing reforms such as awarding badges for open science practices. Fourth, it is one of two journals that showed a significant increase in the EDR from 2010 to 2020 with alpha = .01. Thus, it is one of the few journals that shows signs of increasing unconditional power (the ERR shows improvement for more journals because estimates are more reliable). I extended the analysis to all years since 2000. I did not use the first volumes in the 1990s because they had relatively few test statistics to analyze.

A simple plot of the median z-score shows that the signal/noise ratio was a bit elevated at the beginning of the 2000s. It is noteworthy, that Psychological Science implemented a new statistic, p-rep, in 2005 and softened the strict p < .05 criterion to claim discoveries. We see how this policy coincides with a reduction in median z-scores. However, even when this policy was abandoned in 2010, evidence remained weak and only strengthened again with a new editor in 2015. However, this increase is relatively mild and the median z-score is still within 2 and 2.8, which corresponds to p = .05 and p = .005, respectively.

To increase the robustness of results, I conducted z-curve analyses for the years 2000-2006, 2007-2016, and 2017-2020, which roughly mirrors the pattern of median-z scores in the previous figure.

The results for 2000-2006 show clear evidence that QRPs were used because the 95% confidence interval of the EDR, 21% to 54%, does not include the observed discovery rate of 77%. With a point-estimate of the EDR of 32%, the results imply that up to 11% of the significant results could be false positives, but the upper limit of the 95%CI suggests that no more than 20% of significant results are false positives.

The next results show that the small dip in the median z-scores has no practical relevance for the credibility of published results. The estimated discovery rate is 33% compared to 32% for the previous years. It is interesting to compare the EDR of 33% to the success rate of actual replication studies in the reproducibility project for studies published in Psychological Science. The actual success rate was 38%, which is fairly close to the EDR, but much lower than the estimated replication rate (62%), which will be discussed in the General Discussion section.

One way to use the information about the EDR and the implied false discovery risk is to lower the criterion for significance (alpha-level) to reduce the false discovery risk. With alpha = .01, the false discovery risk is 4%. Thus, readers of older Psychological Science articles can adjust alpha to .01 (t ~ 2.5, F ~ 8), to minimize the risk of interpreting a false positive result. This adjustment is empirically justified and less stringent than the adjustment suggested by a large group of meta-scientists who called for alpha = .005 to reduce the risk of false positives (Benjamin et al., 2017).

The results for the last four years show an increase in the EDR from 33% to 54%, which lowers the false discovery risk to 4%. Thus, alpha = .05 offers the same protection against false discoveries as alpha = .01 did in previous years. The adjustment of alpha according to empirically estimated false discovery risks provides an incentive for journal editors to maintain a reasonably high discovery rate. However, even the results for 2017-2020 show that QRPs are still being used. Thus, readers may still be cautious in interpreting just significant results in Psychological Science and editors should aim to further reduce the gap between the observed discovery rate and the estimated discovery rate.

In sum, z-curve analyses of individual journals can be used to examine the type of research that a journal publishes. The results for all 120 journals are published in the annual replicability rankings (Schimmack, 2021). Here I showed how this information can be used to create an incentive for editors to reduce publication bias and false discovery risk by maintaining an unbiased discovery rate of at least 50%.

General Discussion

With a few exceptions (Cohen, 1989; Sedlmeier & Gigerenzer, 1989), meta-psychologists have relied on assumptions to speculate about research practices and false discovries in psychology. Survey studies demonstrated that researchers use questionable practices, but it remained unclear how much these practices undermine the credibility of published results (John et al., 2012). Empirical investigations with replication studies provided an alarming finding that only 25% of significant results in social psychology could be replicated, but it remained unclear whether this low replication rate is limited to social psychology or also holds for other research areas. As a result, there remains a lot of uncertainty and room for speculations among meta-psychologists. This study provides much needed empirical data based on an analyses of over 1 million test statistics from over 100 journals that cover a broad range of psychological disciplines. David Funder observed that some data are often better than no data. Based on this simple heuristic, the increase from N = 0 to N > 1 million is a step in the right direction. Although the present study is not without limitations, it provides much needed empirical evidence about the use of QRPs and the false discovery risk in psychological research across a broad range of disciplines. Below I provide a more detailed discussion of the key findings.

Questionable Research Practices

The first finding is that there is clear evidence that questionable research practices are common in all disciplines of psychology. This finding supports survey results that show not only that psychologists use these practices, but also do not see them as problematic (John et al., 2012). However, survey results have been questioned and are subject to numerous biases that may lead to an underestimation of the prevalence of QRPs. Here I used an objective method to quantify the impact of QRPs on the credibility of published results. The evidence is conclusive that journals publish too many significant results. The only remaining question is why QRPs are being used.

One common claim among meta-psychologists is that QRPs are often used without awareness when researchers explore their data. However, this explanation is unsatisfactory for two reasons. First, data exploration also produces many non-significant results. Thus, researchers must be actively choosing to report the significant ones and not to report the non-significant ones to produce to many non-significant results in the journals. It makes little sense to postulate that these choices occur without awareness. A more plausible explanation is that psychologists are aware how they analyze their data, but lack (or at least lacked) awareness that their practices are questionable. In support of this hypothesis, many researchers considered questionable practices (except fraud) to be acceptable (John et al., 2012).

One possible explanation for this nonchalant attitude towards QRPs is that many psychologists may falsely believe that statistical significance ensures a low false discovery risk (reference). It is therefore important to educate psychologists that alpha is not equivalent to the false discovery risk and that the false discovery risk increases when they fish for significance in their data and find only a few significant results in a large number of tests. This does not mean that fishing expeditions are wrong, but reporting the results of these exploratory studies as if only a few confirmatory tests were conducted is wrong. Honest reporting of all tests is a fundamental aspect of a sound science. This message was also emphasized by Bem (2000) in his tuturial for graduate students.

“the integrity of the scientific enterprise requires the reporting of disconfirming results.”

Thus, there is no excuse for the use of QRPs that inflate the rate of confirming results. Z-curve analyses of published studies make it possible to detect the use of QRPs and researchers who use QRPs should be held responsible for practices that threaten scientific integrity.

Average Power of Studies in Psychology

A seminal study of statistical power by Cohen (1961) suggested an average power of around 50%. Most commentators have argued that 50% power is low and Nobel Laurate Kahneman and his equally famous colleague Tversky called it “ridiculously low” (Tversky and Kahneman, 1971, p. 107). They echoed Cohen’s recommendation to conduct a priori power analyses and suggested that researchers should not conduct studies if power is less than 50%. “We refuse to believe that a serious investigator will knowingly accept a .50 risk of failing to confirm a valid research hypothesis” (p. 110). Yet, Kahneman admitted that he ignored his own advice and placed to much faith in studies with small samples (Kahneman, 2016). The present results suggest that Kahneman is not alone. Psychologists across many disciplines seem to be happy to conduct studies with an average success rate around 50%. A number of meta-psychologists have speculated about this neglect of statistical power (Maxwell, 2004; Schimmack, 2012; Sedlmeier & Gigerenzer, 1989).

Many areas of psychology still publish articles that report the results of a single study. For solo-study articles, a simple explanation for the neglect of power is that studies are often complex enough to allow for multiple hypothesis tests. With an average probability of 50%, the chance of finding one significant result increases considerable as the number of tests increases, especially when these tests are independent. For example, three independent tests with 50% power have 87.5% power to get at least one significant result. This logic implies that non-significant results are type-II errors and that results will be inconsistent across studies. In this scenario, questionable research practices are particularly damaging to a field because there is no real difference between significant and non-significant results that are merely distinguished by the random flip of a coin. If at least all results were published, meta-analyses could use all the data to reveal that the inconsistent results from study to study are due to chance and produce consistent and unbiased results. Thus, honest reporting of all results would solve the problem that is created by selecting for significance with studies that have low power.

Schimmack (2012) pointed out that low power is a bigger problem for articles that report multiple tests of the same hypotheses across several studies, like Bem’s article with 9 demonstrations of time-reversed cognitive abilities. Here low power may seem counterproductive because internal replication failures weaken the evidence in favor of a hypothesis. Most people would be more impressed if all studies are successful than if only 50% of the reported studies are successful. Not surprisingly, most internal replication studies that are published report significant results. I could not find an explanation for low-powered internal replication studies. However, a simulation study showed that it can be more efficient to run small studies and to select only significant ones than to run studies with larger samples to produce a package of successful replication studies (Finkel, @@@). For example, it requires a large sample size to have 80% power to test a small standardized mean difference of d = .2, N = 788). To produce 4 significant results, researchers need to conduct 5 studies which implies a total sample size of N = 3,940 participants. Instead, researchers could test the same effect in severely underpowered studies (10%) with just 46 participants per study. Ten percent power implies that they need to run 40 studies to get 4 significant ones, but they need only 1,840 participants. Thus, it is more efficient to allocate resources to more studies with low power than to a few studies with high power. The problem with this strategy is that significance does not ensure that significant results are true positive results. And here we see the disadvantage of the latter strategy. When we have 4 significant results out of 40 attempts, the discovery rate is only 10%, and the false discovery risk is 50%. However, when we have 4 out of 5 significant results, the discovery rate is 80%, and the false discovery risk is less than 1%. Thus, conducting many studies with low power creates more significant results, but at the cost of a higher risk of false discoveries. This risk was ignored because QRPs masked the low discovery rate. With real-time information about the actual discovery rate in psychology journals, editors and researchers should have a real incentive to increase their discovery rates.

The present results showed that several areas of psychology, especially those where recruitment of participants is fairly easy and inexpensive, have increased statistical power over the past decade. These increases have also reduced the false discovery risk. In light of these positive trends, it would be counter-productive to lower the significance criterion to .005 (Benjamin et al., 2017). This recommendation was based on speculations about extremely high false positive rates in psychology. The present results show that these fears are unfounded. Moreover, the false discovery risk can be reduced further by increasing statistical power. In addition, increasing statistical power has the advantage that it also reduces the percentage of false negative results. Any adjustments to alpha should be based on empirical evidence that the false discovery risk in a discipline is too high.

It is also important to distinguish between previous estimates of conditional power and the present examination of unconditional power. Previous estimates of power assumed effect sizes. In contrast, the present results are based on the actual effect sizes and include an unknown proportion of studies in which the nil-hypothesis was at least approximately true. Ideally, we would want researchers to sometimes test false hypotheses because making risky new predictions is an important part of scientific progress. The presence of some studies in which the nil-hypothesis was true implies that the power estimates here are estimates of the lower bound of conditional probability when the effect size is at least small or larger. Some models have tried to estimate the percentage of actual nil-hypotheses, which would allow to estimate the conditional probability given the presence of an effect size (Jager & Leak, 2014). This is an interesting avenue for future research. Meanwhile, the present results should be considered conservative estimates of power for studies in which an effect size is at least small.

On the other hand, the present estimates of power may overestimate power for focal hypothesis tests in confirmatory studies. The reason is that power estimates are based on all test-statistics that are reported in the text of an article. This often includes manipulation checks and auxiliary tests. A comparison of automatically extracted test statistics and focal hypothesis tests tends to show lower EDRs for focal tests. The solution to this important caveat is to conduct studies with hand-coding of focal hypothesis tests (Motyl et al., 2016). Thus, the way forward is to get more data, but not to return to arm-chair speculations and criticism.

False Discovery Risks in Psychological Science

Concerns about false discoveries in psychology are not new (Rosenthal, 1979). The main method to address these concerns were meta-analyses. Meta-analysts were well aware that questionable research practices, typically called publication bias, undermine the validity of meta-analyses. However, psychologists relied heavily on the fail-safe N statistic to convince themselves that publication bias is often not a major problem. A large fail-safe N statistic was even used to argue that extrasensory perception is real (Bem & Honorton, 1994). The problem with meta-analysis is that they do not properly correct for publication bias. As a result, hundreds of meta-analyses are invalid and provide no evidence whether an effect is real or not, let alone provide credible effect size estimates.

The sanguine belief that most results in psychology are real was shattered when Simmons et al. (2011) published an article titled “False Positive Psychology.” The article has been cited thousands of times to suggest that producing false positive results is easy and common. However, once more this claim was based purely on simulations and a demonstration how massive use of questionable research practices produced an implausible significant result. The present results show that the simulated scenario does not represent the typical behavior of psychological researchers. If this were the case, we would have expected a much higher estimate of the false discovery risk. The huge impact of the false positive article shows how easy opinions can be swayed in the absence of hard facts. The present results provide clear evidence that researchers use QRPs, but they do not support the claim that most published results are false positives. In this way, the present results confirm Jager and Leaks’ findings for medical journals, where the estimates false discovery rate was 15% and not 50% as predicted by Ioannidis (2005). It is therefore important to correct the widespread belief that scientists are just (p-value) hacks. While there is need for improvement, social scientists are producing knowledge that is grounded in empirical observations. We can improve and we need to protect our science from practices that undermine it, but a few stunning replication failures in social psychology do not imply that psychology as a whole is not trustworthy.

Outlook

Psychology as a science has come a long way since it took off in North America after the second world war. Pioneers worked with limited resources and were probably well aware of the limitations of their studies. The study of human behavior is becoming easier as technology advances and more and more of human behavior is mediated by computers. In contrast, statistical practices in psychology have not changed and still follow Fisher’s original approach of significance testing with alpha = .05. This approach to data analysis has been criticized for decades, but not much has changed. The reason is that the approach is not fundamentally flawed. Real effects, especially those with meaningful effect sizes, are more likely to produce a p-value less than .05 than nil-hypotheses or trivial effect sizes. Thus, the call for radical reforms seems misguided. Instead, psychologists could benefit from focussing more on effect sizes, conducting a priori power analyses, and most important, reporting all of their results. The honest reporting of results has become much easier with unlimited publication space, online supplements, and online sharing of data. While vanity journals may continue to limit publication space like exclusive restaurants and night clubs, science is best served by journals that merely ensure scientific integrity. The emergence of journals that are free for authors and free for readers creates an infrastructure that makes honest reporting of research results possible. The only remaining barrier are motivated biases by researchers to protect their theories from falsification and their original discoveries from replication failures. However, the past decade has shown that it is no longer possible to suppress replication failures. I am therefore optimistic that a real reform of psychology is underway and the positive trends in the EDR support this. This optimistic conclusion may surprise some readers who have followed my work over the past decade, but it is informed by an empirical assessment of the evidence. My first analyses of trends in 2016 showed little evidence that things were changing, some journals and disciplines have shown some real improvements. The present results hopefully will accelerate this change because no editor wants their journal to be the last refuge for p-hackers.

It is also important to realize that rejecting a false nil-hypothesis is only the beginning of a scientific exploration. Defenders of nil-hypothesis testing have pointed out that a correct rejection of the nil-hypothesis with a two-sided test provides at least information about the direction of an effect. This is more than nothing. Knowing that an intervention, on average, reduces depression rather than increasing depression is valuable information. However, the next step has to be estimation of effect sizes and specification of boundary conditions. Moreover, demonstrating real effects requires valid measurement and psychology has neglected validating measures. Thus a lot of work remains to improve psychology, but at least we are not starting from ground zero.

References

Bem, D. J., & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115(1), 4–18. https://doi.org/10.1037/0033-2909.115.1.4

Smaldino P.E. & McElreath R. (2016). The natural selection of bad science.
Royal Society Open Science, 3, 160384. http://dx.doi.org/10.1098/rsos.160384

To reduce bias, I created this review with the help of ChatGPT. ChatGPT evaluates the manuscript within the limited framework of the question examined in this article. My own review focusses on broader issues and the practical relevance of these simulations.

ChatGPT

Overview

The manuscript offers a thorough examination of how confirmatory factor analysis (CFA) models with perfect simple structure perform when fitted to data generated by exploratory factor analysis (EFA) structures with various numbers of cross-loadings. The authors combine clear theoretical derivations with extensive simulation studies—implemented via a novel ShinyApp—to demonstrate that, under certain proportionality constraints, EFA models can be covariance-equivalent to their CFA counterparts. This work challenges the conventional wisdom that acceptable EFA solutions are invariably rejected by CFA models, and it raises important questions about the sensitivity of standard SEM fit indices (RMSEA, CFI, and SRMR) to omitted cross-loadings.

Strengths

1. Theoretical Rigor and Novel Insights:
   The manuscript provides a detailed algebraic treatment showing that when cross-loadings in an EFA model are proportional to the main loadings, the covariance structure is invariant under rotation—yielding a CFA model with perfect simple structure. This theoretical contribution clarifies the mathematical basis for covariance equivalence and underlines why commonly used fit indices may fail to detect misspecification in many practical scenarios.
2. Comprehensive Empirical Investigation:
   By systematically varying the number and magnitude of cross-loadings—and by considering both constant and random loading scenarios—the simulations offer a nuanced picture of when misfit is detectable by SEM fit indices. The exploration of both positive-only and mixed-sign cross-loadings adds depth to the analysis, illustrating that the sensitivity of fit indices depends not just on the number of cross-loadings but also on their sign and variability.
3. Practical Tool (ShinyApp):
   The introduction of a ShinyApp for exploring the sensitivity of fit indices provides a valuable resource for researchers. It allows users to dynamically assess how different parameter settings affect model fit, making the theoretical results more accessible and practically useful.
4. Integration with Existing Literature:
   The review of prior simulation studies (e.g., Chen et al., 2008; Ferrando & Lorenzo-Seva, 2000; Marsh et al., 2013, 2020) situates the work within the broader discourse on CFA versus EFA. The authors do a commendable job reconciling conflicting findings and explaining why commonly accepted “folk wisdom” about CFA model misfit may need to be revisited.

Weaknesses and Areas for Improvement

1. Limited Scope of Models:
   While the manuscript’s focus on 2- and 3-factor models makes the simulation study more manageable, the generalizability of the findings to models with a higher number of factors is less clear. Although some exploratory results for larger kk are mentioned, a systematic investigation into higher-dimensional models would strengthen the paper’s implications for applied research.
2. Reliance on Population-Level Analyses:
   The study emphasizes population values of fit indices to eliminate sampling fluctuations. Although this approach clarifies the asymptotic behavior of the indices, it may understate the impact of sampling variability. Future work could benefit from incorporating finite-sample analyses or confidence intervals to better bridge the gap between theory and practice.
3. Exclusivity of Fit Indices Considered:
   The focus on RMSEA, CFI, and SRMR is justified for clarity, but many practitioners use a broader range of fit indices. Expanding the discussion to include alternative measures or discussing how other indices might behave under similar conditions would add practical value.
4. Complexity and Accessibility:
   The manuscript is densely technical and might be challenging for readers who are not already familiar with the nuances of factor analysis and SEM. Simplifying some of the algebraic derivations or providing additional intuitive explanations could enhance accessibility without sacrificing rigor.
5. Implications for Practice:
   While the paper offers clear recommendations for simulation studies and theoretical model selection, it could further discuss how practitioners should balance statistical fit with theoretical justification. Emphasizing strategies for validating the proportionality constraints in real-world data would be beneficial.

Conclusion

Overall, the manuscript makes a significant contribution to our understanding of model fit in factor analysis by revealing that CFA models with perfect simple structure can exhibit excellent fit—even when the true data-generating process is far more complex—due to covariance equivalence. The blend of theoretical and empirical work is commendable, as is the development of a practical tool for researchers. Addressing the limitations related to model scope, sample variability, and accessibility would further enhance the impact of the work. This study is a valuable resource for methodologists and applied researchers alike, prompting a re-examination of standard practices in model evaluation.

ChatPGT rating 8 out of 10

Ulrich Schimmack

The main problem with this article is that it has limited (to be generous) relevance for applied researchers who want to use factor analysis to examine the pattern of correlations among items. The key problem is the confusion of measurement and structural parts in a confirmatory/structural equation model. A measurement model relates theoretical constructs to measures of these constructs. For example, the item “I am sociable” is used to measure the personality disposition to be more or less sociable across different social situations. The purpose of examining correlations among items that measure different constructs is to evaluate discriminant validity (Campbell & Fiske, 1959). Take the constructs sociability, assertiveness, and cheerfulness as an example. To measure the three constructs, researchers would write items that reflect differences in these three personality traits. A good item would have high loadings on the intended factor that represents the theoretical construct and low loadings on the other factors. For example, the item “I am sociable” might have a loading (correlation) with the sociability factor of .7 and loadings of 1. and .2 on the assertiveness and cheerfulness factors. A common rule of thumb in factor analysis is that good items have loadings of .6 or higher on the intended factor and loadings of .3 or lower on the other factors. Importantly, researchers have full control over the loading pattern by selecting good items and removing bad items. The goal is to end up with items with high primary loadings (on the intended factor) and low secondary loadings) on the other factors. If the analysis does not show sufficient items that show the expected relationships, researchers could and should write new items and validate them in a new set of data. Ultimately, a good measurement instrument would show the expected simple structure, a term used for the desired pattern of primary and secondary loadings.

The problem with Savalei and Huang’s simulations is that they treat the pattern of correlations as a natural given that needs to be explored and described. However, measurement models are not a product of nature. They are a product of researchers’ ability to write good items and good items will show a simple structure.

It is also a common mistake to use EFA to examine the structure of item-item correlations because EFA cannot reveal all problems of items that reduce their validity. Most importantly, EFA fails to reveal that some items are more strongly correlated to each other than the EFA model predicts. These additional relationships are called correlated residuals. Only CFA can reveal that two or more items have correlated residuals (Schimmack, 2025, Padova Winter School). Thus, the focus on secondary loadings in this article ignores a major problem in item validation, namely demonstrating that residual variances of items are only weakly related.

Savalei and Huang might argue that EFA is sometimes used to examine theoretical questions. For example, before researchers used EFA to validate items to measure the Big Five traits, EFA was used to explore the structure of personality, which led to the creation of the Big Five model. The problem here is that it is unreasonable to expect nature to produce simple structures. For example, anger is related to high Neuroticism and low Agreeableness and Cheerfulness is related to high Extraversion and low Neuroticism. These theoretical relationships are not under the control of researchers by selecting good measures because the relationship between specific and broad traits is a theoretical one. This is called the structural model. A structural model needs theoretical justification. This is also the reason why EFA cannot be used to validate structural models. To do so, it is necessary to specify a theortical model and use SEM to fit the model to the data. It is totally irrelevant whether data generated with an EFA model can fit a CFA model or not. The real question is whether a theory fits the data and to examine this the theory needs to be specified as a set of structural equations and fitted to the data.

The final problem is that EFA is limited in the exploration of structural models for a number of reasons. First, it assumes that each item represents a distinct construct and that the correlation between items can be explained by their relationship to factors. This is a problem because single items have a lot of measurement error. Multi-rater studies suggest that only 30% to 50% of the variance in a personality item is valid variance. The large amount of measurement error makes it difficult to study structural relationships between constructs. A solution to this problem is to build hierarchical models that represent constructs with multiple items, but EFA cannot do this. Thus, CFA is a superior method to study structural models without measurement error.

Another problem of EFA is that it is a dogmatic causal model. Tha is, it assumes that constructs are only related to each other because they are influenced by unobserved third variables (factors) and cannot influence each other. This dogmatic assumption will sometimes be false (e.g., heigt and weight are not corelated due to the shared influence of an unobserved third variable), but factor analysis cannot represent the direct effect of heitht on weight. In contrast, SEM forces researchers to justify their causal assumptions when they build a model. Some correlations may be explained by unobserved third variables and others by direct causal effects. Researchers cannot hide behind a dogma to solve the third variable problem (Schimmack, 2025, Padova Winter School).

In short, the relationship between EFA and CFA models has limited practical importance for researchers interested in measurement or exploration of substantive structural models. Measurement models relate items to constructs and CFA is needed to demonstrate that items have good psychometric properties (i.e., high primary loadings, low secondary loadings, & low correlated residuals). Structural studies require validated measurement models to examine the relationship among constructs without dogmatic assumptions that hide the third variable problem. To make progress in 2025, we need to introduce a new generation of students to SEM rather than examining the relationship between EFA and CFA.

Rating 2 / 10

Patil, P., Peng, R. D., & Leek, J. T. (2016). What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science. Perspectives on Psychological Science, 11(4), 539-544. https://doi-org.myaccess.library.utoronto.ca/10.1177/1745691616646366

Review by ChatGPT (2/15/2025) [Rating 7/10]

Conclusion

Patil, Peng, and Leek (2016) provide a thought-provoking statistical defense of psychological replication efforts, arguing that expectations for replication need to align with statistical principles. Their prediction interval analysis offers an important counterpoint to the oversimplified “36% replication rate” narrative.

However, the paper does not fully address deeper issues in psychological science, including publication bias, p-hacking, and the overall credibility of original findings. While statistical consistency is valuable, it does not necessarily mean that the original findings are valid or practically meaningful.

Ultimately, their work highlights the need for a more nuanced conversation about replication—one that considers both statistical expectations and scientific integrity in equal measure.

Comment by Ulrich Schimmack (25/02/15)
Rating 3/10

I only saw this article today. I was surprised by the conclusions in this article, because the senior author Leek had conducted an empirical study of the false discovery risk in medicine and found a relatively low risk of 13%. Ten years later, we (Schimmack & Bartos, 2024) replicated this finding with a better statistical method. The authors of this article could have used their method with the original data from the Open Science Reproducibility Project that is the target of this article. Instead, they use the finding from medicine to suggest that things are also fine in psychology. We have shown with z-curve that medicine and psychology, especially social psychology, are dramatically different. Medicine is much more credible than experimental social psychology was before the replication crisis. Thus, their article is true and meaningless. Focusing on the wide confidence intervals in original studies due to small sample distracts from the finding that a replication rate of 37% with a discovery rate of 90% or more in original articles suggests massive publication bias and a high false positive risk. Disappointing, but it may explain why Leek never replied to my emails.

Introduction

The article by Patil, Peng, and Leek (2016) offers a statistical perspective on the replicability crisis in psychology, particularly in response to the Reproducibility Project: Psychology (Open Science Collaboration, 2015). The authors challenge the simplistic interpretation that only 36% of the studies successfully replicated and argue that a deeper statistical understanding—particularly using prediction intervals—paints a different picture of reproducibility.

Their core claim is that 77% of the replication effect sizes fall within the 95% prediction interval of the original study, suggesting that most replications are statistically consistent with expectations, even if effect sizes differ.

Strengths of the Paper

1. A More Nuanced Statistical Perspective

One of the paper’s biggest contributions is its emphasis on prediction intervals, rather than a binary success/failure measure of replication. The authors highlight how:

• Statistical variation can naturally lead to different effect sizes in replication attempts.
• Wide confidence and prediction intervals in original studies mean that a range of replication results may still be “consistent.”
• The expectation of near-identical effect sizes across studies is unrealistic, given factors like sampling variability and measurement error.

This argument is a valuable contribution to the debate on replication, as it shifts the focus from absolute reproducibility to statistical consistency.

2. Critique of Media and Public Interpretation

The authors provide a strong critique of how the 36% replication success rate was widely publicized without appropriate context. They argue that such a stark percentage misrepresents the complexity of replication, especially when original studies have high uncertainty.

This is an important point, as oversimplified media narratives can lead to public distrust in science and misunderstanding of how replication works in practice.

3. Focus on Imprecision in Original Studies

The paper draws attention to a crucial issue: many original psychological studies have wide confidence intervals and low statistical power. This means that:

• Many reported effects may be overestimated (a common issue due to publication bias).
• Even when a replication effect is statistically consistent, it may not provide strong evidence of a real effect.

By emphasizing the uncertainty in original studies, the authors shift the focus from whether studies “replicate” to whether they provide reliable and precise estimates.

Criticisms and Limitations

1. Over-Reliance on Prediction Intervals

While prediction intervals provide a broader statistical perspective, they do not necessarily solve the problem of false positives and publication bias. Some key concerns include:

• A replication result being within the prediction interval does not mean the effect is “real.” If the original study had a biased or inflated effect size, then replications that cluster around it may still be misleading.
• Prediction intervals assume no systematic bias in the original study. Many psychological studies suffer from issues like questionable research practices (e.g., p-hacking, HARKing), which may inflate effect sizes. Simply computing prediction intervals without accounting for this can be misleading.
• Wide intervals can be uninformative. If the prediction interval is very large, a wide range of replication outcomes may appear “consistent,” even if they are practically meaningless.

2. Downplays the Importance of Failed Replications

While the authors argue that replications are often “statistically consistent” with the original studies, they do not fully address the practical significance of the findings. If a replication finds an effect size much smaller than the original, even if statistically within expectations, it may challenge the real-world impact of the original finding.

For example, if an original study found an effect size of 0.5 and a replication finds 0.1, the latter may technically fall within the prediction interval but still indicate a much weaker or negligible effect.

3. Lack of Discussion on Systematic Biases

The paper treats the issue as primarily statistical, but psychological science’s replication crisis is also methodological:

• Publication bias favors studies with significant results, leading to inflated effect sizes.
• P-hacking and researcher degrees of freedom can produce misleading original findings.
• Lack of pre-registration means that many “original” findings may be the result of selective reporting.

By focusing mainly on prediction intervals and statistical expectations, the paper does not fully engage with these deeper concerns about scientific rigor and transparency.

Implications for Psychological Science

The authors offer a valuable statistical critique of how replication is interpreted, but their conclusions also raise further questions:

1. How should researchers interpret replications where the effect size is much smaller but still statistically “consistent”?
2. Should psychology journals adopt stronger standards for original studies, such as larger sample sizes, pre-registration, and stricter statistical thresholds?
3. How can the media better communicate nuanced statistical findings without oversimplification?

While this paper challenges the narrative that the Reproducibility Project showed a “crisis,” it does not fully exonerate psychological science from the replication challenges it faces.

We are running out of terms. Personomics, Personmetrics, and Personometrics have already been used. Fortunately, the term Personometrics is used by engineers to talk about people as data sources. I hope they are fine with me using the same term to describe the marriage of personality science with psychometrics.

Psychologists are of course used to fake novelty and it seems fair to accuse me of just trying to be original by finding a new term for something that is not novel at all. Don’t we already have journals with titles like “Journal of Personality Assessment,” or “European Journal of Psychological Assessment?” However, a WebOfScience search with “personality” and “measurement” retrieves only 624 articles and the article with the 7th highest citation count is a article by situationist Walter Mischel from 1977 with the title “The future of personality measurement.” For people unfamiliar with personality psychology, it is necessary that Mischel is best known for his claim that personality traits do not exist, which makes the measurement of personality unnecessary. The lack of influential and foundational articles on personality measurement suggests that personometrics is indeed a neglected research topic.

Another indication that personality research and mesurement are separate fields is an article by a psychometrician that criticized personality psychologists for their lack of understanding of modern psychometrics (Borsboom, 2006). Borsboom’s article attacked McCrae, Zonderman, Costa, Bond, and Paunonen (1996) for their conclusion that modern psychometrics should not be used to study personality because these methods do not support their theory of personality structure. You do not have to be a rocket scientist to realize that rejecting a method because it fails to support your theory is not science. Science requires testing theories and revising theories when a theory does not predict empirical observations. Thus, McCrae et al.’s rejection of modern measurement theory is unscientific. If would not be necessary to mention this article, if it were not symptomatic of the attitude towards psychometrics among personality psychologists even today. Twenty years after McCrae published the infamous article, he wrote to me and argued “Why in the world do you think that “CFA is the only method that can be used to test structural theories”? If that were true, I would agree with your position. But the major point of our paper was to offer an alternative confirmatory approach using targeted rotation” (McCrae, personal communication, 2019).

This response shows that McCrae still did not understand the fundamental difference between the classic and outdated methods they used and modern latent variable models that are now commonly used in psychometrics (Borsboom, 2006). He also seems to be unaware of Borsboom’s criticism of the 1996 article. As a result, a peer-reviewed psychometric evaluation of Costa and McCrae’s model of personality is still lacking in 2025 (see Schimmack, 2019a, 2019b, 2019c, 2024, for pre-prints).

Other personality psychologists share this attitude towards psychometrics. In another personal communication, Lew Goldberg wrote “Isn’t it the case that one problem with CFA is that one must start with an ending?  That is, one must have a structural representation in mind before one can “confirm” it.  EFA might be used to provide the initial structure in novel domains, and then one might use CFA as a more “causal” representation?”

The statement also reveals a fundamental lack of understanding of psychometric models like Confirmatory Factor Analysis. First, it is an entirely different model than EFA that does not allow for hierarchical structures or correlated residuals. Thus, EFA may never produce results that can fit the data with a CFA analysis. Second, it is not a problem that CFA requires an idea about the structure in the data. In fact, this is the the main strength of CFA. It may confirm or disconfirm a theory depending on the fit to the data. In contrast, EFA does not test theories and fails to alert users that the simplistic model does not fit the data. The main problem that McCrae et al. (1996) encountered was not a lack of a theory, but the lack of fit of their theory to the data.

Bill Revelle also sent me an email to inform me that EFA is fine and, contrary to my claims, can fit hierarchical models. “Inspired by your blog on how one needs to use CFA to do hierarchical models (which is in fact, incorrect), I prepared the enclosed slides.”

It is not necessary to discuss Revelle’s EFA approach to the study of hierarchical structures because he has never attempted to use this approach to study personality traits. Instead, he has warned about the use of latent variable models which includes EFA and CFA and is now advocating the study of personality items rather than personality traits. This is like studying thermometers without a theory of temperature.

I hope these three examples of prominent personality researchers support my claim that personality psychologists have shown little interest in using statistical methods developed by psychometricians to test theories and evaluate measures. Thus, Borsboom’s attack has had no effect on research practices in personality research.

Borsboom also contributed to the lack of progress in personality measurement because he convinced himself that personality traits do not exist. In a series of papers, Borsboom examined crucial concepts like latent variables and construct validity. He came to the right conclusoin that measurement is based on a causal effect of a factor that exist independent of a measure on an instrument that was created to reflect variation in the factor. Cronbach and Meehl (1955) used temperature and thermometers as an example. Temperature is a physical attribute of the material world and thermometers are valid if variation in the reading on the thermometer corresponds to the variation in temperature.

The same assumption is made in personality psychology. The problem is that the influence of personality traits on observable behaviors is not deterministic. Behavior is always a function of situational factors and personality factors. This makes it harder to measure personality. Another problem is that the internal causes of behavior are not directly observable because they are rooted in complex neurological processes. This makes it difficult or impossible to experimentally manipulate personality traits to study their effects on behavior and to validate personality measures. These difficulties, however, do not justify disbelief in the reality of personality traits. Twin studies, for example, have provided strong evidence that some internal causes of behavior are heritable. It is also possible to observe the influence of personality by demonstrating consistency in behavior across different situations. Borsboom ignored all of this evidence from decades of personality research to claim that personality traits are not real and to claim that it is wrong to model item correlations with latent variables. Thus, he also dismissed the use of psychometric measurement models to study personality (Cramer et al., 2013).

Thus, here we are in 2025 without empirically tested theories of personality traits and validated measures of these traits. This problem is hidden by the fact that personality researchers pretend that they study personality traits and routinely claim that their personality scales are valid. The problem is that the term validity itself has no meaning (Schimmack, 2020). A classic article by Cronbach and Meehl (1955) introduced the concept of construct validity. A measure has construct validity, if it measures what it is supposed to measure. To evaluate whether a measure is a valid measure of a construct, it is necessary to define a construct. For example, to evaluate the construct validity of a measure of Extraversion, it is necessary to define Extraversion. The problem for personality researchers is that 100 years of research on Extraversion has not produced an answer to the question of the nature of extraversion. What is extraversion?

The lack of a theory of extraversion or any other personality trait is not a problem for personality researchers because they operationalize construct. That is, the concept is defined by the measure. Extraversion is whatever a specific Extraversion scale measures. Typically, this means that Extraversion is defined by a specific set of items that a personality researcher wrote. This classic approach to the study of personality is illustrated in Figure 1a.

Figure 1 uses the graphic language of structural equation modeling to illustrate the difference between classic personality research that is grounded in classic test theory and psychonometrics, which is rooted in modern psychometrics.

In classic test theory, constructs are operationalized by specific operations that produce scores for individuals. The typical operation in personality research is to ask participants for a rating on a Likert scale. In Figure 1, there are three items. To reduce random measurement error, the scores to the three items are summed or averaged. In the picture this operation is illustrated by the arrows pointing from the items to the sum score. Evidently, scores on the sum score depend on the specific items. Different sets of items will produce different sum scores. Operationalism is captured in saying like “Intelligence is whatever the intelligence test measures.” As there is no theoretical construct that exists independent of the specific items, it is meaningless to question the construct validity of a sum score. A sum score is a sum score is a sum score.

Operationalism is often masked in the cargo cult of classic personality research by naming their sum scores with worlds that are used in everyday language (Anxiety) or can be interpreted using everyday language (Positive Affect, Psychological Wellbeing). This practice is problematic because it seems to imply that sum scores measure constructs and that the construct is captured by the everyday meaning of the labels. The problem with operationalism is that there is no way to examine construct validity; that is, does a sum score really measure what it label suggests. As a result, personality item sum scores have unknown construct validity (Borsboom, 2006; Schimmack, 2010).

To give an example, I am using three of the 10 items from the Positive Affect (PA) scale of the Positive Affect Negative Affect Schedule (PANAS; Watson, Clark, & Tellegen, 1988). The items are strong, attentive, and active. We sum the scores on the three items and now we call the sum score Positive Affect. Using this sum score, we find lower scores of PA on weekends than during the week. The result is used to provide scientific evidence for the claim that people have higher Positive Affect during the week than on the weekend. If you think this is a problem, you made the mistake to assume that a Positive Affect sum score measures something like happiness or pleasure based on your interpretation of the worlds Positive Affect. The correct interpretation is that people have more of whatever the PA sum score measures during the week than on the weekend. At best we can look at the item content and say that people are more likely to be active, attentive, and strong during the week than on the weekend. However, even this conclusion may be wrong because the sum score is compatible with different patterns for the individual items. Maybe only attentiveness and activity are higher during the week, but strong shows the reverse pattern. To be sure, we would have to examine each item individually and then the result for the sum score follows directly from the pattern of the differences for each item.

The next problem of classic personality research is that it blurs the distinction between prediction and causation. The reason is that personality research often uses correlational evidence because it is difficult or unethical to manipulate personality traits. Psychology has a strong bias towards experimental evidence as evidence to support causal claims. To justify the use of artificial and simplistic laboratory experiments, correlational studies are often criticized for not being able to prove causality. As a result, personality researchers are very reluctant to talk about causality. Even when they want to make causal claims, they often avoid causal language or editors tell them to avoid causal language. As no evidence of causality is needed, correlational results are considered sufficient. Even when causality is implied, it is sufficient to claim that some unspecified experiment is needed to provide evidence of causality in the future.

Personometrics requires causal assumptions. First, the measurement model implies that variation in an unobserved personality traits causes variation in item responses to use item responses as valid measures of a personality trait. Second, personometrics makes causal claims because it assumes that variation in personality traits causes variation in actual behaviors (Figure 1b). The assumption that correlations between item responses and behaviors are caused by a common personality trait makes it possible to test construct validity of personality measures. For example, to demonstrate that a shyness questionnaire measures shyness, researchers can examine behavioral indicators of shyness in a controlled laboratory setting (Asendorpf & Banse, 2002).

Figure 2 is a different visual representation of the psychonometric model in Figure 1b. It highlights that there is no structural distinction between real world behaviors and responses to test items in a survey. Every observable variable can serve as an indicator that reflects the influence of a personality trait. A shyness item is a good measure of shyness if it shares variance with variation in actual shyness behaviors. The stronger the relationship is, the higher is an item’s validity. The common cause model is the fundamental principle in Cronbach and Meehl’s (1955) seminal article on construct validity.

The validation with specific behaviors or feelings in a specific situation (states) is useful because it is much easier to define constructs of observable behaviors. For example, while the internal mental and neurological processes that produce variation in helpfulness are difficult to study, it is a lot easier to define behaviors like helping. Even without insights into the inner processes, it is then possible to measure trait differences by observing behaviors in controlled situations or as cross-situational consistency across independent situations in real life (Schimmack, 2010). Self-report measures mainly serve the purpose of making the measurement of these traits easier and cheaper. That is, once a set of items has been validated as a measure of a personality trait, variation in the measure can be used to test theories of the causes and consequences of personality traits.

The model in Figure 2 makes it clear that latent variables are needed to build theories of personality that can explain behavior. The reason is that item-responses are not causes of behavior. The only justification for the use of item-responses as predictors of behavior is the assumption that item-responses reflect unobserved causes of actual behaviors. This fundamental assumption requires validation by demonstrating that correlations among items and correlations of items with actual behaviors and other relevant variables are consistent with a theory of internal causes of behavior.

Some personality psychologists have argued that latent variable models have many problems that make it less appealing to use them (McCrae et al., 1996). Many of these criticism are outdated and based on lack of expertise. While latent variable models may have a steep learning curve, they offer a lot of advantages over classic correlational studies with sum scores. First, it is well known that short scales with a few items have low reliability. To address this problem with sum scores researchers often use scales with 4, 8, or 10 items. However, this limits the amount of traits that can be measured. Latent variable models correct for unreliability and it is possible to measure a trait with just three items. It is also known that self-report items are biased by response styles. Correcting for these response styles with sum scores is difficult and so called lie scales do not work. In contrast, latent variable models make it possible to correct for systematic biases and to measure them with just a few items (Anusic et al., 2009). Finally, latent variable models are needed when personality traits are measured with multiple methods or ratings by multiple raters. A sum score will reduce rater-biases, but it will not eliminate the individual biases. In contrast, a latent variable model makes it possible to separate the shared variance among raters that is more likely to reflect the true trait from rater-specific variance that is more likely to be measurement error. In short, latent variable models are needed to move personality research forwards towards personality science. The hallmark of science is to use methods that subject theories to empirical tests and can force researchers to revise and improve their theories. Personometrics takes this fundamental aspect of science seriously. Traditional personality researchers like McCrae, Goldberg, and Revelle may not want to embrace modern methods, but hopefully new psychologists who are interested in internal causes of behavior will break with the anti-science dogma and embrace falsification of untested personality theories like the Big Five model.

References

Asendorpf, J. B., Banse, R., & Mücke, D. (2002). Double dissociation between implicit and explicit personality self-concept: the case of shy behavior. Journal of personality and social psychology, 83(2), 380–393.

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440. https://doi.org/10.1007/s11336-006-1447-6

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219. https://doi.org/10.1037/0033-295X.110.2.203

Cramer, A. O. J., Van der Sluis, S., Noordhof, A., Wichers, M., Geschwind, N., Aggen, S. H., Kendler, K. S., & Borsboom, D. (2012). Dimensions of normal personality as networks in search of equilibrium: You can’t like parties if you don’t like people. European Journal of Personality, 26(4), 414–431. https://doi.org/10.1002/per.1866

Source:
Bryan, C. J., Yeager, D. S., & O’Brien, J. M. (2019). Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences USA, 116, 25535–25545. http://dx.doi.org/10.1073/pnas.1910951116

This review was created “with the help of” ChatGPT to minimize bias. My own comments are highlighted in bold italics with my initials US.

Critical Review of the Article

1. Summary of the Article

The authors argue that the current methods used to conduct and publish replication studies in psychology are flawed due to “replicator degrees of freedom”—choices made during experimental design and data analysis that can lead to misleading failures to replicate. They suggest that flexibility in replication studies allows researchers to inadvertently (or deliberately) produce false negatives while maintaining an appearance of methodological rigor. Using a case study of replication attempts on a voter turnout intervention, they demonstrate how variations in experimental design and statistical analysis can obscure an effect that was originally found to be significant.

The authors claim that:

• Replication studies are subject to biases, much like original studies.
• Flexibility in replication design (e.g., slight changes in timing, context, or participant selection) can result in misleading null findings.
• Even with strict methodological controls, standard analytical choices can be used to favor non-significant results.
• Statistical tools such as specification-curve analysis can be used to evaluate the robustness of replication findings and expose selective reporting of statistical results.

2. Strengths of the Article

a. Important Contribution to the Replication Debate

This paper addresses a key issue in psychological science: the replication crisis. Many scholars focus on p-hacking and questionable research practices in original studies, but this article shifts attention to the biases that can arise in replication attempts. The authors argue convincingly that failures to replicate should not be taken at face value without scrutiny.

b. Use of Statistical Rigor

The authors employ specification-curve analysis, a robust statistical technique that tests multiple model specifications to reveal how different analytical choices impact results. They also apply Bayesian Causal Forest (BCF), a machine-learning approach for detecting heterogeneity in treatment effects. These methods strengthen their argument that the replication study they critique was affected by data analysis choices.

US. Data analytic choices can influence results, but these analyses do not show that these choices reduced the power of the replication studies to produce significant results. A better way to do so would be an estimation of power and an examination of bias (i.e., fewer significant results than power of studies allows).

c. Challenges Assumptions About the Objectivity of Replicators

A key assumption in replication research is that replicators are unbiased, but the authors argue that they, too, have incentives to produce certain results (e.g., publishing null findings to challenge existing theories). By pointing out that replication studies may be subject to their own version of p-hacking (“null hacking”), the paper encourages a more balanced perspective on the replication process.

US. Once more, a comparison of statistical power and success rates can reveal bias in both directions. If we get only 20% significant results with 50% power, we have evidence that non-significant results were selected. If we get 90% significant results with only 50% power, we have evidence for the typical selection bias in favor of significant results. Claims about bias should be supported by empirical evidence of bias.

d. Calls for Improved Methodological Standards

The article provides practical recommendations for improving replication methodology, including:

• Considering contextual factors in replication attempts (e.g., timing and population differences).
• Avoiding overly rigid statistical criteria that make significant results difficult to obtain.
• Using pre-analysis plans and specification-curve analysis to ensure robustness in replication testing.

3. Weaknesses and Criticisms

a. Potential Confirmation Bias

Although the authors present strong statistical evidence, they are not neutral parties in the debate. They were involved in the original research on voter turnout, and their analysis is designed to defend the original findings. While this does not invalidate their arguments, it raises concerns about potential confirmation bias. They acknowledge this issue but do not fully address how it might have influenced their methodological choices.

b. Selective Case Study

The authors focus on one specific replication debate (voter turnout and linguistic framing). While they argue that their findings generalize to other replication studies, this claim is speculative. More evidence across different areas of psychology would strengthen their argument.

US. Not only is the claim speculative, it is also contradicted by empirical evidence. Bias tests have routinely demonstrated selection FOR significance in original studies. In contrast, there is no empirical evidence of selection for non-significant results in replication studies so far.

c. Overreliance on Specification-Curve Analysis

Specification-curve analysis is useful for identifying how analytical choices impact results, but it does not necessarily prove which analysis is correct. The authors suggest that replicators’ choices were misleading, but one could argue that their own analysis is simply another subjective selection of model specifications. While their statistical methods are sophisticated, they do not completely eliminate the possibility of bias.

d. Downplaying the Importance of Larger Sample Sizes

The authors critique the idea that larger replication samples should be given more weight, arguing that flawed design choices can still produce misleading results. While this is true, larger samples generally produce more reliable effect size estimates than small, underpowered studies. The original study they defend had a much smaller sample than the replication attempt, yet they do not fully acknowledge how that might have influenced the discrepancy in findings.

US. Agreed. The strongest evidence to counter replication failures would be for the original authors to replicate their own findings using a preregistered analysis plan with a sample sizes that ensures high power to demonstrate that the effect is real when the right methods and analyses are used. Interestingly, this option is used rarely by original researches when they respond to replication failures.

e. Lack of Discussion on Preregistration’s Role

The article suggests that preregistration does not fully solve the problem of analytical flexibility. However, it does not discuss how preregistration can help mitigate some of these issues. A more nuanced discussion of how preregistration could be improved (rather than dismissed) would strengthen their argument.

4. Broader Implications

This article highlights a critical issue in the replication crisis: the field’s tendency to accept replication failures as definitive proof that an original effect was spurious. The authors argue that context matters, and replication attempts that fail to recreate key aspects of the original study may be invalid. This raises broader questions for psychology:

• Should replication studies always be considered the gold standard for scientific validity?
• How can the field create better methodological guidelines for replication attempts?
• Should psychology focus on understanding effect variability rather than labeling effects as either “real” or “false”?

The paper encourages a more nuanced view of replication, one that takes into account experimental context, researcher incentives, and statistical flexibility. While it does not fully resolve the debate, it provides a strong argument for revisiting past replication failures with greater scrutiny.

US. A common problem in discussion of replication failures is the misinterpretation of replication failures as evidence that an original effect was spurious. The problem is that

5. Conclusion

Overall, this article makes an important contribution to the ongoing replication debate in psychology. It highlights the role of replicator degrees of freedom in shaping replication outcomes and calls for more rigorous methodological standards in replication studies. However, its arguments are somewhat self-serving, and its critique of replication studies could be more balanced. Despite these limitations, it raises valuable points that should be considered as psychology continues to refine its approach to replication research.

Final Rating: 8/10

US. The rating is way to high, but may be the AI is American. There is no evidence that replication failures are caused by shady practices of replication researchers to produce negative results.

✅ Strengths:

• Important contribution to the replication crisis discussion
• Strong statistical analysis (specification-curve & BCF)
• Highlights biases in replication studies
• Calls for better methodological standards

❌ Weaknesses:

• Potential confirmation bias
• Focuses on a single case study
• Overreliance on specification-curve analysis
• Underestimates the value of large sample sizes

This paper should spark productive discussions on how replication studies are designed, analyzed, and interpreted. While it is not the final word on the topic, it is a thought-provoking challenge to conventional wisdom in psychological science.

US The problem is that there are now way more articles that discuss the replication crisis than empirical studies that actually explain it. This article adds no substantial evidence to understand replication failures outside of the specific studies examined here, and even for these there is no conclusive evidence that the original results were credible and the replication failures were caused by unintentional or intentional biases to produce non-significant results.

Final Rating 2/10

Like a bad marriage, I have tried to make it work for years, but it is finally time to throw in the towel and get a divorce. I was never in love with personality psychology. My first passion was emotion research (states not traits) and social psychology. I soured on social psychology well before the replication crisis that showed many results were obtained with dishonest research practices because artificial lab studies don’t really tell us anything about real human behavior.

My interest in emotions in real life led me to daily diary and experience sampling studies, and I was fortunate to join Ed Diener’s Subjective Wellbeing (SWB) lab to conduct some of these studies. I became interested in SWB and life-satisfaction. For better or worse, SWB research was part of personality psychology because social psychologists focussed on artificial laboratory studies rather than real-world outcomes. Social psychologists only contribution was to make false claim about the validity of life-satisfaction judgments (Schwarz & Strack, 1999) that were amplified by Nobel Laureate Daniel Kahneman to propose a return to hedonism. He later admitted his mistake, but never really apologized to Ed Diener for his invalid criticism of life-satisfaction judgments as the best measure of human happinesses (there is not one happiness for social scientists to discover. Everybody has to define it for themselves, and scientists can only measure how happy people are given people’s own criteria for a good life. They cannot define happiness for them).

Personality psychologists were happy to include life-satisfaction measures in their studies to demonstrate that personality has real world consequences. However, the justification of personality psychology with prediction of real world outcomes also created an incentive to inflate effect sizes of personality and to dismiss other influences on life-satisfaction. This is the paradigm trap. Once you look at a question only from one perspective, you lose sight of other factors. As much as personality psychologists hated the slogan by social psychologists “the power of the situation” to make fun of small effect sizes in personality psychology, personality psychologists started to do the same thing to prove the “power of personality.”

A number of articles on wellbeing by personality psychologists illustrate the fallacy to think that people’s life-satisfaction is determined by their personality. You can find articles with titles like “Happiness is a personal(ity) thing” (Weiss et al., 2008) or “Most people’s life satisfaction matches their personality traits” (Mõttus et al., 2024). Most prominently, Lykken and Tellegen’s (1996) article “Happiness is a stochastic phenomenon” argued that stable personality traits explain 80% of the variance in wellbeing and Lykken claimed that trying to be happier is like trying to be taller. I hated this article and my motivated bias made me think there was an error in their computation of explained variance. I was wrong, but they made several other mistakes. First, they used a bunch of items from a personality questionnaire, averaged them, and then labeled this personality item sum score (PISS) “wellbeing”. The name implies that it is a measure of happiness or wellbeing, but it is just a name for the average of some personality items that were designed to measure personality, not happiness or life-satisfaction. Thus, an invalid measure of wellbeing was used to tell people that they cannot be happier.

The same mistake was repeated just recently when three items from a personality questionnaire were separated from the other personality items (“Am happy with my life,” “Feel that my life lacks direction,” “Have a dark outlook on the future”), averaged, and the 3-item PISS was labeled life-satisfaction, as if it is a standard life-satisfaction measure, which it is not. Other item were called a personality item (“I have a lot of fun” and the fact that this personality item is strongly correlated with the life-satisfaction PISS was used to argue that life-satisfaction is just personality (Mõttus et al., 2024). No evidence is provided that the “fun” item was not just another life-satisfaction item or that the “dark outlook” item is not a personality item. Welcome to shitty personality research in 2024. Rich Lucas, a former Ed Diener student, who knows better did not see any need to publish a comment on this terrible article (you see, why I no longer have psychology friends. You either pledge a paradigm oath, or you are a pariah. I chose pariah because I hate lies, deception, and shoddy pseudo-science.

The fundamental problem of personality psychology is that PISS are given labels that suggest they are measures of some construct. Like the 10-item PISS created by Rosenberg in the 1960s is called self-esteem and it is still used over 60 years later to measure self-esteem, but it is unclear how much scores on this PISS reflect people’s actual self-esteem. If we would not have to worry about measurement error, you could just increase your self-esteem by changing your responses to the items.

An article as old as the self-esteem scale explained how psychologists could validate their measures (Cronbach & Meehl, 1959), but construct validation research is boring and hard to publish. So nobody does it. with a few exceptions. Ed Diener actually did many studies to examine the validity of his 5-item life-satisfaction scale using experience sampling and informant ratings to show convergent validity. Most personality psychologists, however, treat self-report measures as if they are perfectly valid despite evidence to the contrary (self-informant correlations are around r = .4).

The careless use of labels for PISS often impedes real scientific progress. The biggest confusion was created again by Tellegen in his work with his students Watson and Clark. They created two 10-item PISSs, one called Positive Affect and the other called Negative Affect, but the Positive Affect measure does not measure what normal people would call pleasure, positive affect, or enjoyment. It includes items like alert, which you might feel when somebody breaks into your house at 2am. Hundreds of articles about wellbeing have used this misleading measure without every wondering whether the measure has any validity as a measure of happiness. One study even found that watching a distressing scene in the movie “Schindler’s list” increased Positive Affect, but most psychologists rely on labels to interpret results and are not bothered by findings that question construct validity. Construct validity is just not a thing, personality psychologists care about.

That is personality psychology in a nutshell. Write some items, show that the item average is reliable, give the PISS a name, and then correlate the shit out of it with other PISSs. Alternatively you can use exploratory factor analysis, demand two independent factors, call them Positive Affect and Negative Affect and then claim that – contrary to common sense – Positive Affect and Negative Affect are independent. This does not mean that happiness and sadness are independent or any other positive and negative feelings are independent. It simply shows something that was clear from the beginning. If you demand two orthogonal/independent factors, EFA will give you two independent factors. Trying to find a substantive theory for this discovery is silly because you didn’t discover anything. You create two PISSs that are independent by design. The surprising finding that Positive Affect and Negative Affect merely shows that you picked stupid labels for your PISSs and nothing more. Call me crazy, because the article that published this crap is the most highly cited article in the esteemed Journal of Personality and Social Psychology. So clearly, I must be crazy to criticize a measure that is so popular (well, science is not a popularity contest, cf. Galilelo vs. Catholic Church). Actually, their measure is slowly being replaced by a better measure developed by Diener and colleagues.

Another property of personality psychologists beloved exploratory factor analysis is that it never examines whether it fits the data. This is a problem because science advances by falsifying bad theories that do not fit data. So, using a method that always fits cannot advance a science. Fortunately, personality psychology is not a science and it doesn’t care about progress, except the creation of more PISSs that can be correlated with already existing PISSs.

In the 1990s, personality psychologists made the mistake to use an alternative method called Confirmatory Factor Analysis (CFA) that provides information about the fit to actual data. They were disappointed that the method suggested their beloved Big Five model was not consistent with the data. Fortunately, they found a solution to this problem. They decided that CFA should not be used (McCrae et al., 1996). The ban of methods that can reveal bad fit may explain why personality psychology has made no theoretical progress over the past two decades. Some people like to use Big Five PISSs and others use the HEXACO PISSs, and cool new kids use the Dark Triad or Tedrad PISSs. It doesn’t matter as long as the work creates some interesting PISS-PISS correlations that can be published.

Personality psychologists do not even have a clear definition of personality that guides their work. Personality can be a trait (an internal disposition that leads to different responses to the same situation), but it can also be a state that varies from situation to situation. In the end, any item that produces variation in responses across individuals can be a personality item. For example, one personality item is “I don’t like poetry,” but there are no items that ask whether you like rock music, or more specifically the Beatles. See, there is an infinite number of items that personality researchers can create and average. You can even average all items you can find and call it the general personality factor. No kidding, it is a thing among PISS researchers. If you agree strongly with items like “I have the best personality,” “I have many good qualities,” and “I am fantastic” you probably also score highly on the General Personality Factor. And if you score low on this factor, you probably score high on the general personality disorder scale. Again, I am not kidding. Pathological personality psychologists really believe that some people have all the disorders.

So it is time to say good bye to personality psychology. Maybe it was a mistake to become a psychologist. When I made my career choice, I was considering alternative fields like sociology or economics, but ironically I thought psychology is the most scientific way to study human behavior. In the end, I became a meta-psychologist or in German Über-psychologist. Meta-science is a new field that uses the scientific method to study scientists. As scientists are human, we can actually learn a lot about human behavior from studying psychologists. One of the biggest experiments (without a control group) is under way to see whether psychologists are able to recognize their mistakes and improve their research practices. Time will tell.

Saying good bye to personality psychology also does not mean that I will ignore personality traits as predictors of happiness. There is ample evidence that personality and genetic dispositions contribute to variation in life-satisfaction across people, especially in wealthy nations where social policies have reduced variation in people’s ability to fulfill basic human needs (Anusic & Schimmack, 2016). However, there is also ample evidence that social factors influence life-satisfaction ranging from unemployment and underemployment to being overworked and burnout. A happy marriage adds to the wellbeing of some, whereas unhappy marriages, divorce, bereavement, and involuntary singlehood reduce wellbeing. Although it is clear that personality and environmental factors often interact, there is hardly any research on these interactions, again, in part, because personality psychologists think only personality matters.

I am even still teaching a course on personality. This is only possible because I started a long time ago to write my own materials that are now published as a textbook. The book is short because it focusses on the few real scientific contributions that personality research has made. it also avoids the problem of PISS because it teaches students that PISS are imperfect measures of real personality traits that cannot be directly observed using the language of latent variable models. Fittingly, the book is called “The Science of Personality.” Unfortunately, most research that is published fails to follow the rules of science. Writing a book on personality is also easier these days because it has become common to publish open data and some impressive studies with large national representative samples include personality measures. The book uses these data with proper measurement models that avoid the pitfalls of PISS.