Shaky Scientific (Epistemiological) Foundations of Personality Psychology

Personality psychology lacks solid scientific foundations. Behaviorism was impoverished, but at least it was based on clear rules. Only the effects of situational manipulations on observable behavior were admissible empirical observations, and theories were not allowed to invoke metaphysical constructs like thoughts and feelings to explain behavior.

Modern empirical psychology has moved on from behaviorism, but behaviorism still influences the thinking of psychologists without their awareness. Personality psychology survived behaviorism, but it kept one of its rules: causal questions can only be answered with experiments. This is a problem for a research area that cannot run experiments with random assignment to study causality.

Are taller people more confident because of their height? We cannot simply assign people to a tall or short condition, alter their height, and see what happens. Behaviorism simply decided that such questions are not psychological questions. Nowadays, hundreds of articles report correlations of personality traits with each other or other variables, but personality psychologists have not revised the way they think about causality.

For the most part, they have accepted that observational research cannot test causal theories and frame their research in terms of prediction: can my personality item sum score predict something of interest? For example, does a Conscientiousness scale predict GPA? The answer is yes, the correlation is about r = .2, and no definition of conscientiousness or causal theory is needed to explain the correlation.

However, personality psychologists sometimes make causal assumptions, even if these assumptions are not spelled out explicitly to avoid criticism of causal claims based on correlational research.

The most common causal assumption in personality research is that responses to similar items correlate because they measure a common trait. For example, responses to the items “I am talkative” and “I am outgoing” are correlated, and this correlation is used as evidence that both items measure a common trait, a personality disposition. This justifies averaging the responses to create a personality item sum score and calling it sociability. Importantly, the name of a personality item sum score is arbitrary. It could also be called T346, or “that thing that is reflected in the shared variance between responses to items like talkative and outgoing.”

No theoretical definition of a construct and no theory of measurement is needed. Sociability is whatever a specific sociability scale measures. Here lies the contradiction: a causal assumption, a common trait, is used to justify summing the items, and then an operational definition is used to avoid ever testing that assumption.

The problem with this approach to measurement becomes more obvious when we go beyond correlations among items that are synonymous or semantically similar. For example, take the items “I am often in a good mood” and “I am talkative.” Responses to these items are also positively correlated, but the items have distinct meanings. While it may be easier to imagine that people in a good mood talk a lot, it is also easy to imagine people who are in a good mood while absorbed in a solitary activity (flow) without engaging in any social interaction.

Research on the relationships among specific traits like sociability, good humor, anxiety, and anger is nearly 100 years old and took off when Thurstone (1934) developed multiple-factor analysis to study these relationships systematically. In the 1980s, the Big Five model emerged, and it has dominated personality psychology for nearly half a century. However, a clear definition of the Big Five is still elusive.

The reason is that personality psychologists rarely study the Big Five factors. Instead, they create more and more Big Five scales that average a set of items and are given Big Five names. The problem is that these sum scores depend on the items that are summed, and different sum scores produce different results. Thus, Extraversion is not a clearly defined construct. It is a label for scales that tend to correlate moderate to highly with each other because they contain overlapping sets of related items.

These sum scores cannot be used to study the Big Five factors. For example, we can create an Extraversion sum score by averaging the items “I am often in a good mood” and “I talk a lot.” Now we can show that this sum score correlates with a Good Humor scale that contains the item “I am often in a good mood” and with a Sociability scale that contains the item “I talk a lot.” However, these correlations are no longer empirical observations of something interesting. Scales that share items will be correlated, just as 6 and 9 are both divisible by 3 because both are products of 3 and another number (3 × 2 = 6, 3 × 3 = 9). The shared items guarantee the correlation, just as the shared factor of 3 guarantees divisibility.

Unfortunately, many personality psychologists seem unable to realize that these correlations do not tell us anything theoretically interesting or are simply not interested in scientific theories of personality. To examine why Good Humor and Sociability are correlated we need distinct measures of each construct not sum scores that average across items from both measures.

But when faced with a correlation between two observed measures, the alarm bells go off because we are now asking a causal question without experimental evidence. As this is considered improper, the question is never asked.

The simple solution to this problem is to shed the experimental dogma and treat personality psychology like any other science that cannot manipulate the objects it studies. Take astronomy as an example. We resolved the question whether Earth revolves around the sun or vice versa without randomly assigning planets and stars to different experimental conditions. Welcome to personality science, where we test causal theories without random assignment and without directly observable phenomena.

In personality science, Big Five theory makes clear predictions. It assumes that correlations among specific traits are influenced by five broader dispositions. People who are more often in a good mood are also more likely to be talkative because both dispositions reflect a broader disposition called Extraversion. Importantly, Extraversion is just a label, and it is confusing to use the everyday meaning of the term as if we knew what this broader disposition is. A better name might be Factor I, following the numbering of the factors in the lexical tradition (Norman, 1963; Goldberg, 1990), which evokes no associations with bubbly conversation partners.

The nature of Factor I is unknown, but we infer that it exists from the observation that Good Humor and Sociability are correlated. This makes Factor I a causal theory, and the theory could be wrong. Maybe there is another reason why the two traits are correlated. Personality psychologists avoid testing this theory by using sum scores and giving them labels. A sum score is simply a fact about how people answered a set of questions; no theory is needed to call it Extraversion. To keep the two apart, we can refer to factors by number (Factor I) and to scale scores by their labels, together with the instrument used: BFI Extraversion, BFI-2 Extraversion, NEO-FFI Extraversion, and so on. A scale can be used without examining whether it is related to Factor I, or without knowing what Factor I is. How well BFI Extraversion measures Factor I is an empirical question, and the answer may differ for each instrument.

From Exploratory Factor Analysis to Confirmatory Factor Analysis

Thurstone’s original method was exploratory factor analysis (EFA). It did not impose constraints on the relationships between specific traits and broader factors. Researchers chose the number of factors, and the method produced the best possible loadings (the relationships between observed variables and factors) given the correlations among the measures. The model usually did not reproduce all of the correlations, but fit to the data was not a major concern. Often the goal was not to test a theory, but simply to reduce a large number of items to a few sum scores.

Since then, the measurement methods have been developed further in a field called psychometrics. In psychometric measurement models, correlations among items are also explained by unobserved factors (often called latent variables), but these models differ from EFA in two important ways. First, researchers can specify a predicted pattern of loadings and thereby test a theory about why items are correlated; this approach is called confirmatory factor analysis (CFA; Jöreskog, 1969). Second, a good theory should explain the full pattern of correlations, so model fit matters.

Personality psychologists tried these methods to find evidence for Big Five theory, but their models did not fit the data (McCrae et al., 1996). Instead of asking whether their theory might be at least partially wrong, they concluded that confirmatory factor analysis is not appropriate for personality questionnaires. To this day, many of them prefer exploratory methods that never put a specific structural hypothesis to the test.

This is akin to astronomers refusing to use more advanced telescopes and working with the one invented by Galileo. Not surprisingly, little progress has been made. The structural model established in the 1980s has barely been tested, let alone improved.

Using CFA with theoretically constrained loading patterns, it is actually possible to find evidence for the Big Five factors. They do not explain all of the relationships among specific traits, which explains earlier failures, but many traits are related to one or more of the Big Five factors.

This makes it possible to create measurement models of the Big Five without knowing what they are. The shared variance among a set of specific traits is attributed to the causal influence of a broader trait. This does not mean that the Big Five really exist; until more evidence emerges, we can study them as “hypothetical constructs.” For example, we can examine whether Factor I predicts outcomes like life satisfaction or whether Factor III predicts GPA.

Many psychologists, especially older ones socialized during the era of behaviorism, are reluctant to view factors as at least hypothetically real entities (Revelle, 1983, 2016) . However, astronomy shows how a science can make progress by studying hypothetical constructs. Black holes are a useful reminder that some things are real even though we can never see them. Nothing escapes a black hole, not even light, so no telescope will ever show us one. Astronomers know that black holes exist only through their effects on things we can observe, such as stars orbiting an invisible mass or light bending around empty space. No astronomer would say that black holes are unscientific because they are unobservable. Yet this is exactly the rule behaviorism imposed on psychology: if we cannot observe it, it does not belong in science. Personality traits are like black holes. We cannot see them, but we can infer them from their effects on what people say and do. The scientific question is not whether something is observable, but whether the indirect evidence is strong enough to justify the inference. For the Big Five, that evidence is modest, but the reason is that nobody has even tried to find evidence for or against them. If we never test a theory, it’s validit remains unknown. And we cannot test it by correlating more and more personality item sum scores with each other. At some point, we have to study variables that are not directly observable like the Big Five factors with measurement models that rely on the indirect evidence of correlations among specific traits that are easier to measure.

Do Meta-Traits Produce Correlations Among the Big Five?

Some personality psychologists have proposed that there is a set of even broader personality dispositions than the Big Five. This theory was originally proposed by Digman (1997) who found consistent patterns in correlations among Big Five scales. For example, Extraversion scales tended to correlate with Openness scales. These days, the strongest proponent of this theory is Colin DeYoung, who renamed the factors Plasticity and Stability.

The main problem for this theory is that it rests on correlations among Big Five scales, not Big Five factors. Even if BFI-Extraversion and BFI-Openness scales correlate positively, it does not provide evidence that the corresponding factors, Factor I and Factor IV, are correlated with each other. The reason is that scale scores also contain information about the specific traits that were used to create the sum scores. To study the relationship among Big Five factors, we would first need to create a measurement model of the Big Five factors and then examine whether a model with Meta-Traits fits the data and whether it fits the data better than alternative models.

Even the latest article claiming support for the metatraits (DeYoung et al., 2026) did not provide this crucial test. The authors used exploratory factor analysis to compute five factor scores and then treated these scores as measures of the Big Five. But factor scores are not factors. As Revelle (2024) put it, “factor score estimates are not latent variables, they are merely weighted sum scores.” So, it can be both true that Extraversion and Openness sum scores created by DeYoung are positively correlated, but Factor I and Factor V are not.

The fact that 30 years after Digman, an article that uses sum scores to test theories about the correlation among factors underscores my observation that personality psychology as a field is stuck in behavioristic thinking, lacks clearly defined constructs, and lacks measurement models for Big Five scales. My invitation to collaborate on a proper test of Meta-Trait theory with proper measurement models was rejected by DeYoung. Others who are more interested in their own sum scores (inventors of the BFI-2 or HEXACO inventory) similarly showed no interested in testing personality theories.

To turn personality psychology into personality science does require more than just a change of journal names like “Personality Science” or “Social and Personality Science.” It requires a psychometric revolution that takes measurement and model fit seriously. I had little hope that such a revolution would happen during my life time, but AI is revolutionizing sciences as I am typing. So there is some hope that it may also force personality psychologists to become a science with explanatory theories.

Scientific revolutions usually happen when they give new researchers something new to do. I think applying psychometrics to the measurement of personality is timely (or should I say long overdue) and an opportunity for young scientists ready to make a mark. The availability of free software and the new norm to share data provides a unique opportunity for graduate students to do original research and publish articles in journals that are not controlled by guardians of the sum score paradigm.

References

DeYoung, C. G. (2006). Higher-order factors of the Big Five in a multi-informant sample. Journal of Personality and Social Psychology, 91, 1138–1151.

Digman, J. M. (1997). Higher-order factors of the Big Five. Journal of Personality and Social Psychology, 73, 1246–1256.

Goldberg, L. R. (1990). An alternative “description of personality”: The Big-Five factor structure. Journal of Personality and Social Psychology, 59, 1216–1229.

McCrae, R. R., Zonderman, A. B., Costa, P. T., Jr., Bond, M. H., & Paunonen, S. V. (1996). Evaluating replicability of factors in the Revised NEO Personality Inventory: Confirmatory factor analysis versus Procrustes rotation. Journal of Personality and Social Psychology, 70, 552–566.

Norman, W. T. (1963). Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. Journal of Abnormal and Social Psychology, 66, 574–583.

Revelle, W. (1983). Factors are fictions, and other comments on individuality theory. Journal of Personality, 51, 707–714.

Revelle, W. (2024). The seductive beauty of latent variable models: Or why I don’t believe in the Easter Bunny. Personality and Individual Differences, 221, 112552. https://doi.org/10.1016/j.paid.2024.112552

Thurstone, L. L. (1934). The vectors of mind. Psychological Review, 41, 1–32.

An open invitation to personality researchers

Personality psychology has produced numerous taxonomies of personality traits, but we still lack a widely accepted hierarchical measurement model that specifies how narrower traits relate to broader personality dimensions.

Although the Big Five are widely accepted as a descriptive framework, they lack a consensual measurement model. Moreover, other models like the HEXACO model exist without any attempt to create a unified model. This means that personality psychology still lacks a scientific taxonomy of personality traits. Instead, different researchers use different inventories, item selections, and statistical models, often producing different conclusions about personality structure.

How can we establish a common empirical foundation?

Like all other sciences, we have to start with a broad – not necessarily representative or exhaustive – list of objects that we want to classify. For personality psychologists these objects are specific personality traits. The key information for a classification system is how specific traits are related to each other. A key problem is that the standard method used by personality psychologists (EFA, PCA) is limited and produces new structures with every new set of traits. The solution to this problem is to use hierarchical structural equation modeling (or Hierarchical Factor Analysis, HFA) that imposes a theoretical structure on the data that does not change with the item set and correlations, but tests whether the data are consistent with the theoretical model.

I am inviting personality researchers, psychometricians, graduate students, and anyone interested in personality structure to participate in an open, collaborative many-analyst project.

This is a good time for such a project because data are openly available, software is free, and learning HFA is easier than ever with the help of AI. AI can even fit models for you. So, even personality scientists who are not familiar with structural equation models can formulate models and test them.

Participation does not require funding, new data collection, or specialized software. All you need is a computer. Call it theoretical or computational personality science.

The starting point for this project are data from the FacetMAP dataset (Irwing et al., 2024) that contains correlations for 77 facets, each measured with six items.

UlrichSchimmack/Personality-Trait-Taxonomy-PTT-: Developing Hierarchical SEM Models of Personality Traits

I created a GitHub repository that contains the data, syntax files for MPLUS and lavaan, and a model with five independent factors (the classic Big Five model). The model has a well fitting measurement model for the facets. Thus, researchers can focus on alternative models of the structural model that relates the facets to each other. However, modification to the measurement model are also possible. Starting with this model can save a lot of time and prevent convergence problems, but experienced users are free to start from scratch.

The immediate goal is to develop and evaluate measurement models that can help us answer several fundamental questions:

• Which facets can be measured reliably and distinctly?
• How are narrow personality traits related to one another?
• Can these relationships be explained by broader personality factors?
• How much of the shared variance reflects substantive personality characteristics versus method variance?
• Which structural conclusions remain robust across alternative, theoretically defensible measurement models?

Researchers are invited to contribute their own analytical approaches, challenge existing specifications, and propose alternative models.

Rather than trying to establish which researcher or existing taxonomy is correct, we want to identify results that withstand independent scrutiny and develop a measurement model that researchers with different theoretical perspectives can agree on.

All analyses, code, and findings can be shared openly, including results that challenge preferred theoretical models.

The FacetMAP dataset is only the beginning. Once we have established promising models, we can test whether the same structure emerges in other datasets and item pools, including large collections of personality-descriptive adjectives.

The ultimate goal is not another personality inventory. It is a cumulative, empirically grounded understanding of the hierarchical structure of personality traits.

Researchers interested in contributing are invited to explore the GitHub repository, run the existing models, and propose new analyses.

Ulrich Schimmack
University of Toronto Mississauga

Dozens of studies have correlated students’ personality with their GPA (Grade Point Average). The reason is simple. Students are easily accessible, and GPA is an easily measured objective outcome variable. The main finding of these studies is that personality predicts GPA. This is not very surprising to the average reader or students themselves, but in psychology, it was once common to dismiss personality as an explanation for behavior and behavioral consequences. Personality matters!

The more interesting question is which personality traits predict GPA and how much of the variance in GPA they explain. Study after study and meta-analyses of these studies show that Conscientiousness (being orderly, planning ahead, persistence, and low impulsivity and procrastination) is the strongest personality predictor of GPA (O’Connor & Paunonen, 2007; Poropat, 2009). Effort matters, and some students are more able and willing to work hard for a higher GPA. Nobody claimed that psychology is rocket science.

However, it may be surprising how weak the relationship between Conscientiousness and GPA really is. The typical meta-analytic estimate is a correlation of approximately r = .24 (O’Connor & Paunonen, 2007; Poropat, 2009). This means that Conscientiousness explains only about 6% of the variance in GPA.

To make this correlation more meaningful, we can use it to compute the probability that a student who is above average in Conscientiousness also has an above-average GPA. Assuming a bivariate normal distribution, the probability is only about 58%, compared with 50% if Conscientiousness and GPA were unrelated.

This blog post uses data from a published journal article (McCredie & Kurtz, 2020) to examine three additional questions. First, do studies that rely on self-ratings underestimate the true relationship between Conscientiousness and GPA because self-ratings are imperfect measures of personality? Second, are informants, such as parents and peers, better judges of students’ Conscientiousness than students themselves? Finally, do evaluative biases in self-ratings reflect a real personality characteristic that predicts higher GPA, or are they merely rating biases?

The use of multiple raters also makes it possible to examine another interesting question: Does having a positive view of oneself predict higher GPA?

This possibility was suggested by Biderman et al. (2019), who examined evaluative factors in Big Five and HEXACO personality questionnaires. They found that a general evaluative factor predicted GPA even after controlling for the individual personality domains.

[Insert image of Biderman et al. (2019)]

However, their analyses relied exclusively on self-ratings. This makes it difficult to determine whether the evaluative factor represents a substantive personality characteristic or whether it captures valid personality variance that is not adequately separated from evaluative bias.

A multi-informant design provides a stronger test of this question.

Results: How Valid Are Self- and Informant Ratings?

I used structural equation modeling to reanalyze the open data. The model distinguished common personality factors from evaluative factors that were specific to each rater. Here, I summarize the key findings.

All three raters contributed to the measurement of Conscientiousness. The standardized loadings on the common Conscientiousness factor were:

• Self-ratings: .79
• Parent ratings: .75
• Peer ratings: .64

These coefficients indicate how strongly each rating reflects the common Conscientiousness factor. Their squares suggest that the common factor accounted for approximately 62% of the variance in self-ratings, 56% in parent ratings, and 41% in peer ratings.

Thus, self-ratings were at least as strongly related to the common factor as parent ratings, whereas peer ratings showed somewhat weaker agreement.

These findings are broadly consistent with the literature on self–other agreement in personality judgments (Connelly & Ones, 2010). There is no compelling evidence that informant ratings are consistently and substantially more valid than self-ratings. Both methods provide useful information, and their relative validity may depend on the trait, the informant, and the circumstances.

How Strongly Does Conscientiousness Predict GPA?

The common Conscientiousness factor predicted GPA with a correlation of approximately r = .31 (SE = .09).

This estimate is somewhat higher than the typical meta-analytic correlation of r = .24 based on observed personality scores. The difference is consistent with the expectation that measurement error attenuates correlations between observed personality scores and external outcomes.

However, the difference is not dramatic. Even a correlation of .31 explains less than 10% of the variance in GPA.

Thus, correcting for measurement error strengthens the relationship between Conscientiousness and academic performance, but it does not fundamentally change the conclusion: Conscientiousness matters, but most of the variance in GPA remains unexplained.

Do Positive Self-Evaluations Predict GPA?

One of the advantages of the multi-informant model is that it can distinguish common personality variance from rater-specific evaluative variance.

Personality ratings are influenced by the desirability of traits. People generally prefer to describe themselves and others as conscientious, agreeable, extraverted, emotionally stable, and open to experience. As a result, ratings of different personality traits may be positively correlated simply because a rater has a generally favorable impression of the person being rated.

This phenomenon is commonly called a halo effect.

The model identified clear evaluative factors in all three rating sources: self-ratings, parent ratings, and peer ratings.

If these evaluative factors represented a substantive personality characteristic, we might expect them to show positive agreement across raters. For example, people who evaluate themselves especially favorably might also receive especially favorable evaluations from their parents and peers.

However, the correlations among the three evaluative factors ranged from −.45 to .09, and none was statistically significant.

There was no consistent positive agreement across raters. This pattern is consistent with the interpretation that the evaluative factors primarily reflect rater-specific tendencies rather than a common personality characteristic.

The second finding was equally interesting. The evaluative factors did not predict GPA.

In particular, the evaluative factor in self-ratings was not a significant predictor of GPA (β = −.12, SE = .14).

The sample size limits strong conclusions about small effects. Nevertheless, the results provide no evidence that students who evaluate themselves more favorably across personality traits achieve higher grades after controlling for their substantive personality characteristics.

This finding is relevant to Biderman et al. (2019), who reported positive relationships between evaluative factors and GPA in three large samples. Their standardized regression coefficients ranged from approximately .15 to .21.

The present results do not establish that those findings were statistical artifacts. However, they illustrate an important identification problem in single-rater studies. A general evaluative factor extracted from self-report items may contain substantive personality variance as well as evaluative bias. Predicting an external outcome does not, by itself, establish which component is responsible for the relationship.

The multi-informant model provides a way to separate these sources of variance. In the present data, the common Conscientiousness factor predicted GPA, whereas the self-specific evaluative factor did not.

What About the Other Big Five Traits?

Conscientiousness was not the only personality trait associated with GPA in this sample.

Neuroticism was a negative predictor (β = −.22), suggesting that students who were more emotionally unstable tended to have lower grades. Openness was a positive predictor (β = .15).

Extraversion (β = −.05) and Agreeableness (β = −.01) were practically unrelated to GPA.

These findings should not be interpreted as establishing a universal pattern. The effects of personality traits other than Conscientiousness tend to be smaller and less consistent across studies (O’Connor & Paunonen, 2007; Poropat, 2009). The present sample is too small to draw strong conclusions about these additional predictors.

Why Did Peer Ratings Predict GPA?

Perhaps the most interesting finding was that the unique variance in peer ratings of Conscientiousness was related to GPA.

At first glance, this might suggest that peers are better judges of Conscientiousness than students themselves or their parents. However, this interpretation is difficult to reconcile with the lower loading of peer ratings on the common Conscientiousness factor.

If peer ratings were generally more valid indicators of Conscientiousness, we would expect them to show stronger agreement with the other measures, assuming that the common factor captures the valid trait variance.

An alternative explanation is that peers rely heavily on academic performance when evaluating their friends’ Conscientiousness.

For example, students who receive high grades may be perceived as conscientious even if they are disorderly, impulsive, or unreliable in other areas of life. Conscientiousness is a broad personality trait, and students may be highly conscientious about their academic work without being equally conscientious in other domains.

This would make peer ratings useful predictors of GPA without necessarily making them better measures of general Conscientiousness.

Another possibility is that peers observe academically relevant behaviors that are less visible to parents and are not adequately captured by students’ self-ratings.

These explanations are not mutually exclusive. Facet-level data and more detailed information about what peers actually observe would be needed to distinguish them.

The broader lesson is that unique predictive validity does not necessarily imply superior construct validity. A measure can predict an outcome because it captures criterion-specific information rather than because it is a better measure of the intended broad personality trait.

Conclusions

Three results stand out.

1. Self-ratings and ratings by a single knowledgeable informant tend to be about equally valid. Some studies report differences in validity, but there is no consistent evidence that informant ratings are substantially superior to self-ratings. The present results are consistent with this broader conclusion.
2. Personality matters for real-world outcomes, but the effects are not huge. Conscientiousness is a reliable predictor of academic performance, and correcting for measurement error produces a somewhat stronger relationship. Nevertheless, most of the variance in GPA remains unexplained. Personality matters, but it does not determine destiny.
3. Evaluative factors in personality ratings appear to reflect primarily rater-specific bias. The lack of consistent positive agreement among evaluative factors across raters and their failure to predict GPA suggest that much of this variance reflects evaluative tendencies rather than a substantive personality characteristic. Even the evaluative component of self-ratings, which may reflect positive self-evaluation, provided no evidence of incremental validity for GPA.

As always, results from a single study are never conclusive. The sample is relatively small, and some estimates have substantial uncertainty. Nevertheless, the pattern of findings is broadly consistent with the existing literature on self–other agreement and the validity of personality ratings.

References

Biderman, M. D., McAbee, S. T., Hendy, N. T., & Chen, Z. J. (2019). Validity of evaluative factors from Big Five and HEXACO questionnaires. Journal of Research in Personality, 80, 84–96. https://doi.org/10.1016/j.jrp.2019.04.010

Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122. https://doi.org/10.1037/a0021212

O’Connor, M. C., & Paunonen, S. V. (2007). Big Five personality predictors of post-secondary academic performance. Personality and Individual Differences, 43(5), 971–990. https://doi.org/10.1016/j.paid.2007.03.017

Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996

Peer review is supposed to uncover errors in manuscripts. Sometimes it does. Sometimes it provides a rather nice demonstration of the errors humans make while trying to find errors in other people’s work.

I recently submitted a manuscript on publication-bias detection in meta-analysis. The central empirical result was simple: tests of publication bias often have low power when effects are heterogeneous and model assumptions are violated. Consequently, a nonsignificant test for publication bias should not be interpreted as evidence that publication bias is absent. The paper therefore argued for estimating how much publication bias remains compatible with the data, rather than relying on binary significance tests.

The paper was rejected. In his editorial letter, Associate Editor Stephen West wrote that there were “several statistical misstatements throughout the paper which need to be corrected.”

That sounds serious. So I went back through the reviews to identify these errors.

The exercise turned out to be more interesting than expected.

Error 1: What is statistical power?

The manuscript stated:

“Classical power is conditional on the null hypothesis being false and on a specified population effect size being the true effect size.”

Reviewer 2 responded:

“This is false. See a basic introductory statistics textbook such as Testing Statistical Hypotheses by Lehmann and Romano.”

This criticism is particularly interesting because it directly contradicts another criticism of my work that I tried to estimate “average power” without conditioning on a true effect size.

In other words, I was first criticized because power is conditional, I changed the terminology accordingly, and now I am now criticized because defining power as conditional is supposedly false.

Welcome to psycho-science.

Part of the confusion is historical. Lehmann’s Testing Statistical Hypotheses, first published in 1959, uses the classical mathematical-statistics concept of a power function, which gives the probability of rejecting the null as a function of the population parameter and can therefore be evaluated both under the null and under alternatives. That terminology was retained in later editions. In contemporary psychological research, however, statistical power is ordinarily used more narrowly for the probability of rejecting the null conditional on a specified alternative effect size—a usage central to prospective sample-size planning (e.g., Pek et al., 2024).

Maria Soto and I recently published a power primer specifically to disentangle these different usages of power (Soto & Schimmack, 2026). So the problem here is not that I fail to understand conditional probabilities. The problem is that different statistical traditions use the same word differently. It is particularly unhelpful for a reviewer to take one legitimate historical usage and declare another established usage simply “false” and for a lazy editor to repeat this claim without being aware that the term power has no clear definition in the psychological literature.

Error 2: You cannot manipulate the number of significant studies or the FDR

Reviewer 1 objected to two factors in the simulation:

“I don’t see that you can directly vary the number of significant studies or the FDR.”

Of course you can.

A simulation does not have to start with a fixed number of attempted studies and then passively accept whatever number of significant findings happens to emerge. One can condition the simulation on the evidence of interest.

If I want 300 significant results with an FDR of .20, I can generate 60 significant results under the null and 240 significant results under alternatives. This is no more mysterious than fixing the sample size, heterogeneity, or mixture proportions in any other simulation.

Whether this is a useful simulation design is a legitimate question.

Whether it is possible is not.

The reviewer additionally wrote that they did not see the relevance of FDR to meta-analysis. That is a substantive theoretical position. It effectively assumes that a heterogeneous literature should be represented as a continuous distribution of nonzero effects rather than as a mixture that can include true null effects.

But one purpose of the simulation was precisely to avoid making that assumption. The replication crisis has made it entirely plausible that some proportion of published significant findings are false positives. If that possibility is excluded by design, then publication-bias methods are being evaluated only in a world in which every study investigates a real nonzero effect. That is a very strong assumption to build into a simulation of publication bias.

Error 3: Zero effects make high power impossible

The manuscript reported:

“No method approached the conventional .80 benchmark when averaged across the full design.”

Reviewer 1 responded:

“But the full design includes zero effect sizes, doesn’t it? … So no test would have .80 power under those conditions.”

Here the reviewer has confused two entirely different meanings of power.

The quantity being evaluated was the power of a publication-bias test: if a simulated data set is biased, what is the probability that the test correctly rejects the null hypothesis of no publication bias?

It was not the power of the original studies to reject a substantive null hypothesis.

A literature in which every population effect is zero can nevertheless provide overwhelming evidence of publication bias. Suppose only 5% of results would normally be significant, but nearly all nonsignificant findings disappear. With enough observations, a publication-bias test could have very high power to detect that selection.

Thus, including studies with true null effects does not make .80 power impossible. If anything, strong selection is easier to detect when the underlying discovery rate is low, because publication bias produces a large distortion in the observed proportion and distribution of significant results. By contrast, when studies already have high power and most results are significant before selection, removing nonsignificant studies produces a much smaller observable distortion and publication bias becomes harder to detect.

Error 4: 5,184 simulation conditions is a miscalculation

Reviewer 2 wrote:

“The paper reports 5,184 conditions. I count 3 number of studies × 4 effect sizes × 3 heterogeneity values × 3 sample sizes × 4 FDRs × 3 publication bias settings which gets me to 1,728 so off by a factor of three. What am I missing?”

What was missing was one of the simulation factors: the three distributions of population effects.

The design was 3 x 4 x 4 x 3 x 4 x 3 x 3 = 5,184.

There were four, not three, levels of heterogeneity, and three effect-distribution conditions.

There is another small problem. Even the multiplication printed in the review,
3 x 4 x 3 x 3 x 4 x 3 equals 1,296, not 1,728.

So this particular attempt to identify an arithmetic error contains an arithmetic error.

Error 5: A beta distribution cannot have mean zero

Error 5: A beta distribution cannot have mean zero

Here the reviewers found a real problem.

The manuscript described a Beta(1,10) distribution as having mean 0 and standard deviation .4. A raw Beta(1,10) distribution obviously does not have those moments.

The problem, however, was in the description of the simulation, not in the simulation itself. The beta distribution was used to generate the shape of the effect-size distribution and was then transformed to have the intended mean and standard deviation. The manuscript failed to describe that transformation clearly.

In the era of open science, this distinction is unusually easy to resolve. The simulation code was available, so a reviewer who suspected that the reported moments were impossible could have checked whether the error was in the prose or in the actual data-generating process. The code shows that the simulation itself implemented the intended transformed distribution.

So this was a genuine reporting error that peer review correctly detected. But it was not evidence that the simulation results were based on an incorrectly specified Beta(1,10) distribution.

Error 6: The mysterious noncentrality parameter

Another reviewer objected to the statement that a simulated z statistic had a noncentrality parameter of 2:

“There is no noncentrality parameter for a normal distribution. Presumably the mean was 2?”

This is mostly terminology. Under an alternative hypothesis, a normal distribution is centered over a value other than zero. If all studies have 50% power with a significance criterion value of 1.96, the distribution is centered at 1.96 (~ 2).

In power calculations this quantity is routinely described as the noncentrality parameter, although unlike the t, F, and chi-square families there is no separately named “noncentral normal distribution.”

Calling it the mean of the alternative z distribution would avoid the terminological distraction. Call it what you want, but all sampling distributions have a median and when the effect size is not zero, the median is not zero. Whatever you call it, everybody can understand the meaning here and maybe propose a wording change rather than claiming that I invented a statistical entity that does not exist.

Again, this is very different from discovering that the simulation or its conclusions are wrong.

What was actually wrong?

After going through the reviews, I can identify one clear reporting problem: the description of the transformed beta distribution. There are also several places where the manuscript could have been clearer—the z-curve version numbers were particularly unhelpful, and some terminology could be tightened.

Those are useful comments.

Other alleged errors fare less well. The number of simulation conditions was correct. It is perfectly possible to condition a simulation on the number and composition of significant studies. Including zero population effects does not make it impossible for a publication-bias test to have high power. And the definition of power that one reviewer declared “false” is closely related to the very definition that other critics previously insisted I adopt.

None of this proves that the manuscript deserved acceptance. The paper was too long. Its argument could have been more focused. Including a conventional multi-step selection model would strengthen the comparison. The presentation of the various z-curve versions invited confusion.

These are legitimate criticisms.But those are different claims from saying that the paper contained “several statistical misstatements.”

ut those are different claims from saying that the paper contained “several statistical misstatements.”

There is also a simple reason why the manuscript contains no major statistical errors. Before submitting it, I used AI—both ChatGPT and Claude—to challenge my arguments, check calculations, and look for weaknesses. AI did not write the ideas for me, but it was extremely useful as an adversarial reader.

After receiving the reviews, I conducted a small experiment. I uploaded the anonymous manuscript to Claude and presented the reviews as if they were drafts of reviews that I had written and wanted evaluated. Claude was not particularly impressed. (It eventually figured out what was going on because it located the preprint with my name on it.)

Reviewer 1

“Your review raises several valid concerns, but about a third of the specific comments contain an error or overreach that the author could easily rebut. It also misses some issues that are stronger than the ones it leads with. My suggestions are below, in order of importance.”

Reviewer 2

“Reviewer 2 raises the most damaging points in either review: whether z-curve has been properly vetted, and the coverage claim. Several of the supporting comments, though, are factually off or point at the wrong target, and the author could rebut them easily.”

Conclusion

Peer review is commonly described as quality control, but anonymous peer review does not guarantee quality control. It is a human judgment process, and human judges make mistakes. Reviewers misunderstand arguments, overlook information, perform arithmetic incorrectly, import their own theoretical commitments, and sometimes mistake disagreements about terminology or modeling assumptions for factual errors.

Editors then have to evaluate those reviews. If they do not independently check the reviewers’ claims, reviewer errors can become editorial “facts.” In this case, an editorial decision referred to “several statistical misstatements,” even though several of the alleged mistakes were themselves mistakes in the reviews.

This is where AI changes the scientific publishing system. An AI system can read a manuscript, inspect a review, check arithmetic, distinguish a factual error from a modeling disagreement, and compare a criticism with the actual contents of the manuscript in seconds. It will also make mistakes, of course. But unlike traditional peer review, those judgments can be generated repeatedly, compared across systems, and audited immediately.

The interesting question is therefore no longer whether AI should be allowed into peer review. It is already better than pretending that two anonymous humans constitute a reliable error-detection system.

Journals may increasingly use AI to assist editors and reviewers. Authors will certainly use it to evaluate reviews. And eventually the importance of journal gatekeeping itself may decline. Scientific papers increasingly function as inputs into larger information systems—search engines, databases, systematic reviews, and now AI models—that integrate evidence across sources rather than treating publication in a particular journal as the final certificate of truth.

Peer review will probably survive. But its monopoly on deciding what counts as credible scientific information may not.

AI may eventually kill us, but I hope it will kill bad peer reviews first.

Editorial Letter and Reviews

Alesina, A., Carlana, M., La Ferrara, E., & Pinotti, P. (2018). Revealing stereotypes: Evidence from immigrants in schools (NBER Working Paper No. 25333). National Bureau of Economic Research. https://doi.org/10.3386/w25333

Abstracts are short and cannot tell the whole story. A previous working paper and data in this article show that the IAT was also used to predict grading and that there was no evidence that IAT scores predicted lower grades for immigrant students.

The study

Alesina, Carlana, La Ferrara and Pinotti asked whether Italian middle-school teachers’ implicit stereotypes about immigrants, measured with an immigrant/Italian-name IAT, show up in how they grade immigrant students. They also asked whether telling teachers their IAT scores changes their grading.

The design has real strengths:

• 1,384 teachers in 102 schools took the IAT.
• Grades came from administrative records, so teachers didn’t know their grading was being observed.
• Blind-graded national tests (INVALSI) provide a performance benchmark.
• A randomized field experiment varied when teachers received their IAT feedback.

The paper circulated as NBER Working Paper 25333 (December 2018). It was published in the American Economic Review in July 2024. Between the two versions, the evidence on the central predictive-validity question changed little, but the presentation changed a lot.

The predictive-validity test

The key analysis regresses end-of-year grades on an Immigrant × Teacher-IAT interaction, controlling for INVALSI scores and teacher fixed effects. Grades are from the 2011–2016 cohorts. If the IAT measures something that drives discriminatory grading, teachers with higher scores should show a larger immigrant penalty at equal test performance.

Working paper (2018). The authors analyzed math and literature separately.

• For math, one SD higher on the IAT (0.26 D-units) was associated with a 0.033-point larger immigrant penalty (SE 0.019). This was significant only at p < .10 in every specification.
• For literature, the estimate was zero (−0.001, SE 0.017).
• The literature null was explained after the fact by lenience toward first-generation immigrants. The supporting interaction was itself not significant.

Published version (2024). The subjects are pooled, standard errors are clustered by teacher, and the IAT is entered in raw D-units.

• The interaction is −0.075 (SE 0.050), p ≈ .13. The authors state that it is not statistically significant.
• Per SD of IAT, that is about 0.02 grade points, or about 0.02 SD of grades.
• The 95% CI excludes any effect larger than about 0.04 SD per SD of IAT.

So with a smallest effect of interest of d = .10, the data are evidence of no meaningful predictive validity, not merely a lack of evidence for it.

What changed between versions

1. The main test became a footnote to subgroup analyses. After the pooled null, the AER version splits students by INVALSI score.

• For high-ability students: −0.139 (SE 0.080), p ≈ .08.
• For low-ability students: −0.031 (SE 0.060).
• No test of the difference between subgroups is reported.

Even the high-ability estimate implies at most about 0.07 SD per SD of IAT at the CI bound. This split does not appear in the working paper. Yet the abstract-level summary in the introduction now says that higher IAT scores are associated with lower grades for high-performing immigrants. The conclusion calls the pattern “strongly suggestive of bias.”

2. Measurement error was invoked, and a new measure was introduced. The authors attribute the null partly to noise in teacher-level bias estimates. In an online appendix, they show that a “naïve” teacher-level bias measure is not significantly related to the IAT. An empirical-Bayes-shrunken version of the same measure is. This alternative outcome appears only after the preregistered-looking specification fails, which is a classic forking path.

3. The measurement error that matters most is in the IAT itself. The AER version reports a new fact (footnote 6): the male-name and female-name immigrant IATs correlate only 0.28. That implies a reliability of roughly .44 for the averaged score. The authors present this as reassuring because 76% of teachers received consistent categorical feedback. But a test-retest correlation of .28 places a low ceiling on any criterion correlation. It also means many teachers were told they were “moderately” or “strongly” biased on the basis of noise.

4. The figures and text overstate the tables. The text describing Figure 5 says higher IAT is associated with significantly lower grades for immigrants. Table 3, which quantifies the same relationship, says it is not significant.

5. The framing moved from prediction to intervention. The working paper’s abstract led with the IAT–grading correlation. The AER abstract omits it and emphasizes the experiments. The working paper’s key moderator result—that the feedback effect was driven by teachers without explicit anti-immigrant views—has moved to an online appendix.

6. The IAT critique literature is acknowledged, then set aside. The AER version cites the predictive-validity critiques, including Blanton et al. (2009), Oswald et al. (2013), and Schimmack (2021). It responds that those studies used small samples and lacked real-world behavior. This study has a large sample and real-world behavior, and it reproduces the weak-to-null pattern those critiques describe.

The experiments

Field experiment. Receiving IAT feedback before grading narrowed the native–immigrant grade gap by 0.35 points (pooled ITT; SE 0.11). This is larger than the entire conditional gap the paper documents (about 0.10 points). It came from raising immigrants’ grades and lowering natives’ grades.

Did teachers with higher IAT scores respond more? The triple interaction is 0.214 (SE 0.302), which is null. In the field, the IAT predicts neither discrimination nor responsiveness to feedback. A reaction to being labeled biased explains the effect as well as correction of a bias does.

Online experiment. This was added for the AER version. Of 179 teachers who completed the survey, 146 are analyzed, and the attrition isn’t explained in the main text. Each teacher graded ten vignette tests with randomly assigned names.

• The average effect of personalized feedback versus a generic debiasing message is zero (0.017, SE 0.122).
• The headline result is a moderator effect: teachers with higher IAT scores responded more to their own feedback.

One result here does favor the IAT. In the active control group (about 74 teachers), higher IAT was associated with lower grades for immigrant-named tests (−0.426, SE 0.157). Three things should be weighed against it:

• It comes from a small sample, in an exploratory interaction model.
• The outcome is vignette grading, not real grades.
• The group had just received a debiasing message, and teachers with IAT = 0 graded immigrant names 0.42 points higher than native names.

It does not replicate in the field data with real grades and about 780 teachers.

The AEA registry IDs (AEARCTR-0003647 and -0013570) appear to postdate the respective data collections. If so, the moderator analyses were not preregistered. This is worth confirming against the registry.

Conclusion

The working paper’s main predictive-validity claim was a marginal (p < .10) association in math and a null in literature. In the published version, with subjects pooled and errors clustered by teacher, the IAT does not significantly predict discriminatory grading. The estimate is small enough to rule out even modest effects.

The publication process did not converge on this conclusion. Instead it added subgroup splits, an alternative bias measure, and a second, small online experiment. That gave the final paper a narrative of implicit bias that is “strongly suggestive” in the conclusion but not supported by the primary test.

Two of the paper’s own facts deserve more attention than they received. First, about 70% of teachers have “moderate to severe” IAT scores, yet grading discrimination is about 0.1 points. Second, the two IATs given to the same teachers on the same day correlate .28. Both are hard to reconcile with the view that the IAT measures a stable implicit bias that shapes behavior.

In real classroom grading of immigrant students, the IAT failed to predict discrimination.

---

van Zwet, E., Gelman, A., Greenland, S., Imbens, G., Schwab, S., & Goodman, S. N. (2024). A New Look at P Values for Randomized Clinical Trials. NEJM evidence, 3(1), EVIDoa2300003. https://doi.org/10.1056/EVIDoa2300003

Introduction

Randomized controlled trials occupy a privileged position in evidence-based medicine because randomization protects against many sources of confounding that complicate observational research. Yet influential critics have questioned whether even randomized evidence can be trusted at face value. Ioannidis’s Why Most Published Research Findings Are False argued that low power, bias, and low prior probabilities can make statistically significant findings more likely to be false than true, and his argument explicitly included clinical trials (Ioannidis, 2005). Andrew Gelman has expressed related skepticism, arguing that evidence-based medicine encounters serious problems when the evidence is weak and that even “clean randomized clinical trials can fail to replicate.”

However, Ioannidis and Gelman differ fundamentally about what makes a significant finding false. Ioannidis’s framework assumes that some tested effects are exactly zero. A significant result is then a false discovery when it rejects one of these true nil hypotheses. Gelman rejects this zero-versus-nonzero framing as a useful description of scientific research. In his discussion with O’Rourke, he argued that researchers are usually not studying effects that are exactly zero and questioned attempts to infer scientific truth by separating published results into exact-null and non-null components (Gelman & O’Rourke, 2014).

This disagreement became explicit in the debate over Jager and Leek’s attempt to estimate the false discovery rate in medical research. Using 5,322 reported $p$-values from five major medical journals, they estimated that approximately 14% of significant findings were false discoveries (Jager & Leek, 2014). Ioannidis strongly criticized their analysis and argued that the estimate was too reassuring (Ioannidis, 2014). Gelman and O’Rourke (2014) objected for a different reason: they questioned whether a science-wide false discovery rate based on dividing effects into exact zeros and nonzeros was a scientifically meaningful quantity at all.

Gelman and Carlin (2014) proposed focusing instead on Type S and Type M errors. Type M errors concern exaggeration of effect magnitude. Type S errors are more fundamental: a statistically significant result has the wrong sign—for example, a trial concludes that a treatment is beneficial when its true effect is harmful (Gelman & Carlin, 2014). A related quantity, the false-sign rate, has subsequently been developed in the multiple-testing literature (Stephens, 2017).

Jager and Leek’s estimated null component need not be interpreted literally as a population of effects that are exactly zero. Very small nonzero effects generate nearly uniform $p$-value distributions and are difficult to distinguish from exact nulls in a two-component mixture. If their estimated 14% false-discovery component instead represented effects very close to zero, approximately half of these significant findings would be expected to have the wrong sign. This suggests a false-sign rate of roughly 7% from this component, with only a comparatively small additional contribution expected from the stronger-effect component.

Thus, Jager and Leek’s results do not suggest that most significant medical findings either reject a true nil or point in the wrong direction. Nevertheless, their estimate was challenged on several grounds and had little influence on subsequent debates about the credibility of medical research.

Schimmack and Bartoš (2023) approached the problem differently. Rather than estimating the actual false discovery rate, we estimated an upper bound, which we called the false discovery risk. Sorić (1989) showed that the maximum false discovery rate is determined by the discovery rate—the proportion of all tests that are significant—without requiring an estimate of how many true effects are exactly zero. Using a new sample of clinical trials reported in medical journals, we estimated a false discovery risk of 13%, with a 95% confidence interval from approximately 8% to 21%.

This distinction is important because false-discovery methods do not require investigators to identify which hypotheses are truly null. The broader logic has long been used in large-scale multiple testing, particularly in genomics, where controlling the false discovery rate became an alternative to controlling the probability of any false positive (Benjamini & Hochberg, 1995; Storey, 2003). Z-curve extends this logic to literatures affected by publication selection: it estimates the discovery rate that would be expected without selection and uses this rate to obtain an upper bound on the FDR (Bartoš & Schimmack, 2022). Importantly, this approach does not require the assumption that exact nil effects actually exist. The purpose of estimating the false discovery risk is to examine how credible rejections of the nil hypothesis are.

A year after our study, van Zwet, Gelman, and colleagues analyzed 23,551 randomized clinical trials from Cochrane reviews (van Zwet et al., 2024). Their main concern was effect-size exaggeration, but they also estimated the probability that statistically significant trials had the wrong sign. Their model implied a false-sign rate of only about 2%. This remarkably low rate sits uneasily beside broad claims that statistically significant results from low-powered clinical trials are generally untrustworthy. It suggests that the principal problem identified by their model is not that significant clinical trials usually reach the wrong directional conclusion, but that their estimates of effect magnitude are noisy and selected upward.

Thus, three analyses of medical research appear to produce somewhat different pictures: Jager and Leek estimated an actual FDR of approximately 14%, Schimmack and Bartoš estimated a maximum FDR of approximately 13%, and van Zwet and colleagues estimated a false-sign rate of only about 2%. These quantities are not identical, but they address closely related questions about the credibility of statistically significant clinical-trial results. One important reason for their differences is the assumed distribution of true effects.

The present analysis examines this issue directly. Using the same Cochrane data, I fit several substantially different mixture models and examine which conclusions are robust to the choice of mixture and which depend on interpreting the fitted components as real populations of true effects.

The Credibility of Z-Curve

In a series of posts on Andrew Gelman’s blog, van Zwet criticized z-curve, the statistical method that we used to estimate the false discovery risk. His concerns included bootstrap confidence intervals in some settings and sensitivity of the expected discovery rate to misspecification of z-curve’s default discrete mixture. In particular, he showed examples in which a true noncentrality fell between z-curve’s fixed component locations and the expected discovery rate was biased.

These are legitimate concerns about model uncertainty. I responded by further developing z-curve and releasing zcurve3. One important extension is that zcurve3 no longer requires the traditional discrete mixture. Users can fit mixtures of normal distributions and vary the locations and variances of the components, making it possible to examine directly whether substantive conclusions depend on the particular representation of the latent distribution.

This extension is particularly useful here because van Zwet, Gelman, and colleagues modeled the Cochrane data with a mixture of normal distributions centered at zero. Zcurve3 makes it possible to fit the same Cochrane data with their zero-centered normal mixture, with a more flexible normal mixture in which both means and standard deviations are estimated, and with the traditional discrete z-curve model.

This provides a direct robustness test. If the principal z-curve estimands change substantially across these models, concerns about the discrete-component approximation are justified. If they remain stable despite substantial differences in the estimated mixture components, the estimands are more robust than the latent mixture itself. The same comparison can determine whether estimates of the false-sign rate, which depend directly on the inferred distribution of true effects, show the same robustness.

Reproducibility Code:
https://github.com/UlrichSchimmack/zcurve3_development_functions/blob/main/SecondLook.Cochrane.R

Zero-centered normal mixture

The first model specified three normal components with their means fixed at zero and their standard deviations freely estimated, closely reproducing the model used by van Zwet and colleagues. Zcurve3 fits absolute $z$-values and therefore represents these components as normal distributions truncated at zero. For zero-centered normal distributions, this is simply the folded representation of the same symmetric model and has essentially no substantive impact on the fit.

There are two additional differences. Zcurve3 can explicitly model selection for statistical significance, and confidence intervals were obtained with cluster bootstrap resampling to account for the nesting of individual study results within Cochrane reviews.

The z-curve plot shows that the mixture closely traces the distribution of significant results. It also predicts the nonsignificant distribution well. The similarity between the observed and expected discovery rates indicates little evidence of selection for significance in the Cochrane data. This finding is informative in its own right. Nonsignificant trials are not missing from these meta-analyses; they are considerably more common than significant trials. This does not rule out other sources of effect-size inflation, but strong publication selection against nonsignificant trials does not appear to characterize this dataset.

The estimated false discovery risk is approximately 20%, with the upper end of the 95% confidence interval at about 25%. Thus, even when z-curve is fitted with a latent distribution closely resembling the one preferred by van Zwet and colleagues, no more than approximately one quarter of significant findings could be exact-null false discoveries. This result is broadly consistent with our earlier analysis of $p$-values reported in medical-journal abstracts, despite the very different dataset and mixture specification.

Normal mixture with free means

The second model relaxed the assumption that all component means are fixed at zero. Fixing the means at zero served other purposes in van Zwet and colleagues’ application, including producing a symmetric reference distribution and symmetric shrinkage toward zero. However, the restriction could matter if the latent effect distribution were centered or concentrated away from zero.

In the Cochrane data, relaxing the restriction made little difference. The means of the two dominant components were estimated to be close to zero, and the principal z-curve estimates were virtually unchanged. Thus, the zero-mean restriction happens to be fairly benign for these data.

Default discrete z-curve

The third model fitted the default z-curve specification with seven discrete components. Once again, the principal results changed only slightly.

This illustrates an important property of z-curve. Its principal estimands are functions of the fitted distribution of test statistics rather than interpretations of individual mixture components. Different mixture models can assign very different weights and parameters to their latent components while producing nearly identical fitted densities. If they reproduce the relevant distribution of $z$-values equally well, they can therefore produce very similar estimates of the expected discovery rate, expected replication rate, and false discovery risk.

This result directly addresses one aspect of van Zwet’s criticism. A fixed discrete approximation can be biased in some data-generating scenarios, and this possibility should not be ignored. But zcurve3 makes the concern empirically testable. Researchers can fit alternative mixture specifications as a sensitivity analysis. In the Cochrane data, replacing the traditional discrete mixture with normal mixtures does not materially alter the principal z-curve conclusions.

Robust estimands, unstable components

The picture changes when the individual mixture components themselves are interpreted as latent populations.

In the two continuous normal-mixture models, the probability of an effect being exactly zero is zero by construction. The actual exact-null FDR is therefore zero under these models. This should not be confused with the false discovery risk, which remains around 20%. The latter asks how high the FDR could be without assuming that the chosen continuous latent model is literally true.

The zero-centered and free-mean normal mixtures imply false-sign rates of approximately 3.2%, reasonably close to van Zwet et al.’s reported estimate of about 2%.

The default discrete z-curve gives a different latent interpretation. About 4.5% of significant results are assigned to the component with a noncentrality parameter of zero. If this component is interpreted literally, the estimated actual FDR is therefore about 4.5%. If, instead, the zero component is regarded as a discrete approximation to a continuous collection of very small positive and negative effects, exact-zero FDR disappears and approximately half of the significant results in this component become sign errors. Under this interpretation, the false-sign rate is approximately 2.7%.

Thus, these relatively flexible models all imply a false-sign rate below about 5%. However, this apparent agreement should not be mistaken for identification of the latent distribution.

To illustrate the problem, I fitted another discrete mixture with components at noncentralities $0,2,4,$ and $6$. Removing the component at $z=1$ prevents the model from representing weak positive effects explicitly. Many of the low-power studies must therefore be assigned to the zero component instead.

The principal z-curve estimands changed only slightly. The latent interpretation changed dramatically. After accounting for significant observations above $z=6$, the zero component implies an actual FDR of approximately 23%, close to the maximum FDR permitted by the discovery rate. If the zero component is instead interpreted as a symmetric collection of effects extremely close to zero, approximately half of these findings have the wrong sign, producing an estimated false-sign rate of about 11.5%.

Nothing about the observed Cochrane data changed. Only the latent mixture specification changed.

This example illustrates why fitted mixture components should not automatically be interpreted as literal data-generating populations. Their main statistical advantage is precisely their flexibility: different mixtures can approximate the same observed density. Quantities that depend mainly on the fitted density can therefore be robust even when the decomposition into latent components is not. In contrast, quantities that require a literal interpretation of the components—including estimates of the actual proportion of exact zeros, false-sign rates, and some shrinkage quantities—can be substantially more model dependent.

This is not unique to z-curve. Stephens’s (2017) empirical-Bayes approach to false-sign rates, for example, obtains greater stability by imposing a substantive shape constraint: the latent effect distribution is assumed to be unimodal with its mode at zero. Such assumptions may be reasonable, but estimates obtained from them are conditional on those assumptions rather than determined by the observed data alone.

Conclusion

Mixture models remain relatively uncommon in meta-analysis and meta-science, and their use can invite a basic misunderstanding. The most serious mistake is to interpret the fitted components as if they were empirically identified populations of studies or true effects. Component locations, variances, and weights can be highly sensitive to model specification. Different mixtures can fit essentially the same observed distribution while implying substantially different latent decompositions. Good model fit alone therefore cannot establish that one particular decomposition is the true data-generating process.

Z-curve was designed to avoid relying on this interpretation. Its principal estimands—the expected discovery rate and expected replication rate—are global properties of the fitted distribution. The false discovery risk is subsequently obtained from the estimated discovery rate using Sorić’s bound. Consequently, substantially different mixture specifications can produce similar answers as long as they reproduce the relevant features of the observed distribution.

This distinction also clarifies the role of the nil hypothesis. It is not necessary to assume that some fixed proportion of scientific effects are literally zero in order to use false-discovery risk as a credibility criterion. If exact-zero effects do not exist, the actual exact-null FDR is zero. The Sorić bound remains useful because it asks the more conservative question: given the discovery rate, how large could the false discovery rate be? This is consistent with the broader logic of false-discovery methods already established in large-scale multiple testing.

The same point helps clarify discussions of “low power.” Van Zwet and colleagues define the signal-to-noise ratio as the true effect divided by its standard error and translate it into conventional power to reject $H_0:\theta=0$. This is mathematically legitimate, but if exact-zero effects are assumed never to occur, low power against zero cannot itself imply that significant findings are false. It primarily signals that effects are small relative to their sampling error, which creates imprecise estimates, magnitude exaggeration after selection, and some risk of sign errors. How much of this makes a study scientifically untrustworthy requires an explicit criterion.

The present Cochrane analysis provides one such criterion. Across substantially different mixture specifications, the estimated discovery and replication rates and the maximum false discovery rate are remarkably stable. In contrast, actual FDR and false-sign estimates can change substantially when the latent components are interpreted literally.

The lesson is therefore not that one mixture model is correct and another is wrong. Conclusions should be trusted to the extent that they survive reasonable changes in mixture specification. In the Cochrane data, the principal z-curve estimands pass this test. Literal interpretations of the latent mixture components do not.

References

Bartoš F, Schimmack U. Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology. 2022;6:2021.2720. doi:10.15626/MP.2021.2720.

Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Series B. 1995;57:289–300. doi:10.1111/j.2517-6161.1995.tb02031.x.

Gelman A, Carlin JB. Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspect Psychol Sci. 2014;9:641–651. doi:10.1177/1745691614551642.

Gelman A, O’Rourke K. Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics. 2014;15:18–23. doi:10.1093/biostatistics/kxt034.

Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124. doi:10.1371/journal.pmed.0020124.

Ioannidis JPA. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics. 2014;15:28–36. doi:10.1093/biostatistics/kxt036.

Jager LR, Leek JT. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics. 2014;15:1–12. doi:10.1093/biostatistics/kxt007.

Schimmack U, Bartoš F. Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values. PLoS ONE. 2023;18:e0290084. doi:10.1371/journal.pone.0290084.

Sorić B. Statistical “discoveries” and effect-size estimation. J Am Stat Assoc. 1989;84:608–610. doi:10.2307/2289950.

Stephens M. False discovery rates: A new deal. Biostatistics. 2017;18:275–294. doi:10.1093/biostatistics/kxw041.

Storey JD. The positive false discovery rate: A Bayesian interpretation and the q-value. Ann Stat. 2003;31:2013–2035. doi:10.1214/aos/1074290335.

Faridani, S. (2026). Testing for underpowered literatures. Journal of Econometrics, 257, 106312. https://doi.org/10.1016/j.jeconom.2026.106312

Abstract

Faridani (2026) introduces a nonparametric deconvolution method for estimating how much the proportion of statistically significant results would increase if studies used larger samples. The method is innovative because it avoids specifying a parametric distribution of true effects and can accommodate publication bias and point masses at zero. However, the estimand is an unconditional increase in significance rates, not statistical power conditional on a true alternative. As a result, a small estimated gain can reflect high power, many true or approximately true null effects, very small nonzero effects, or some combination of these. For research planning and cost-benefit decisions, methods that estimate the latent mixture of null and non-null effects and the power distribution among true effects may therefore provide more directly relevant information.

Review

Faridani develops an innovative method for addressing an important problem in meta-research: determining how much larger samples would have changed the statistical conclusions of a heterogeneous collection of studies. Rather than attempting to estimate the power of individual studies, the paper estimates the counterfactual increase in the proportion of statistically significant results if all sample sizes were multiplied by a common factor. The proposed estimator uses nonparametric deconvolution, accommodates publication bias, and allows substantial heterogeneity in true effects, including point masses at zero.

A major strength of the paper is its recognition that estimating the full distribution of true effects is a much harder inverse problem than estimating a particular functional of that distribution. Faridani therefore focuses on $\Delta_c$, the expected increase in the proportion of significant results when sample size is increased. This makes the problem statistically more tractable. The simulations suggest that the estimator performs reasonably well across a range of data-generating processes, and the empirical applications to economics and Many Labs provide useful demonstrations.

The interpretation of $\Delta_c$, however, deserves some qualification. Faridani motivates it as a measure of whether a literature is “undersized” and as information relevant to decisions about whether research funds should be spent on larger samples. This is a coherent estimand, but it is not the same as conventional statistical power, which is defined conditional on a false null hypothesis.

For a literature containing both true null and true alternative hypotheses,$$P(\text{sig}) = P(H_0)\alpha + P(H_1)E[\text{Power}\mid H_1].$$

Increasing sample size leaves the first component unchanged. Therefore,$$\Delta_c = P(H_1) \left\{ E[\text{Power}_{c}\mid H_1] – E[\text{Power}_{1}\mid H_1] \right\}.$$

A small value of $\Delta_c$ can therefore have several very different interpretations. Studies testing real effects may already have high power, in which case larger samples are indeed unnecessary. But $\Delta_c$ can also be small because many hypotheses are true or approximately true nulls, or because many true effects are too small for a moderate increase in sample size to help much.

A simple example illustrates the problem. Consider one literature in which every study has a moderate true effect but poor precision. Such a literature may have uniformly low conditional power and would clearly benefit from larger samples. Now compare it with a literature in which half the hypotheses are exactly null and the other half have large effects that are already detected reliably. The second literature may show an equally small or smaller increase in the number of significant results, but for a completely different reason. In the first case, low power is the problem. In the second, many non-significant results are correct indications of little or no effect.

This distinction matters for the funding interpretation. The scientific objective is not simply to maximize the number of $p<.05$ results. Ideally, a test should have a high probability of rejecting $H_0$ when a meaningful alternative is true while retaining the nominal Type I error rate when $H_0$ is true. A small increase in the unconditional rejection rate therefore does not by itself show that studies testing real effects have adequate power.

The main uncertainty lies among the non-significant results. If significant results already have high expected replicability, increasing their sample sizes has little practical value. The important question is whether non-significant results are mostly true or approximately true nulls, or false negatives arising from real effects studied with insufficient precision. The answer depends on the latent mixture of $H_0$ and $H_1$ and on the effect-size distribution within the $H_1$ component.

Faridani’s method deliberately avoids recovering this structure, which is part of what makes the estimand tractable. But this also limits its usefulness for planning future sample sizes. A deconvolution estimate of the average increase in significant results does not separately estimate the prevalence of null and non-null effects or the conditional power of studies testing real effects.

Mixture approaches such as z-curve aim to recover more of this latent structure. In particular, they can estimate the expected replicability of significant results and features of the latent power distribution. This permits a more informative separation between the prevalence of true effects and the power of studies conditional on those effects being present. Such information is more directly useful for cost-benefit analyses of future sample sizes than an average increase in significance rates that combines true or approximately true nulls with true alternatives.

A second, more technical point concerns the paper’s characterization of existing mixture methods. Faridani contrasts his nonparametric approach with methods such as Brunner and Schimmack (2020), which he describes as requiring a “specific shape” for the latent distribution. That description is somewhat too strong. A finite mixture on a prespecified grid does impose an approximation, but its purpose is precisely to approximate a wide range of latent distributions by estimating the mixture weights. The more informative distinction is therefore between different forms of regularization of the same inverse problem: finite-mixture approximation versus regularized nonparametric deconvolution.

These qualifications do not detract from the methodological contribution. Faridani provides a clever way to estimate a difficult counterfactual quantity without recovering the full latent distribution. But the resulting estimand answers a narrower question than the one most directly relevant to statistical power and research planning. It tells us, “How many more significant results would larger samples produce?” A more informative planning question is, “Which studies are underpowered because they are testing real effects, and how much would larger samples improve their ability to detect those effects?” Answering that question requires distinguishing the $H_0$ and $H_1$ components rather than averaging across them.

Robust Bayesian Meta-Analysis, or RoBMA, was developed in E. J. Wagenmakers’ research group as a method for evaluating meta-analytic evidence while allowing for the possibility that published effects are exaggerated by publication bias. An important feature of RoBMA is that it does not merely estimate an effect size; it also assigns evidence to models in which the average effect is exactly zero.

This approach builds on a form of Bayesian hypothesis testing that Wagenmakers and colleagues have long promoted as more balanced than conventional null-hypothesis significance testing. In standard significance testing, researchers can reject the nil hypothesis or fail to reject it, but a nonsignificant result does not itself constitute evidence that the effect is zero. Bayes factors, by contrast, can quantify relative evidence for either the nil or an alternative hypothesis.

This form of Bayesian hypothesis testing dates back to Harold Jeffreys’ Theory of Probability (1939), but it remained far outside the mainstream of psychological research for decades. It became highly visible in psychology after Bem (2011) reported a series of statistically significant experiments purporting to demonstrate extrasensory perception. Wagenmakers et al. (2011) reanalyzed Bem’s (2011) results with default Bayes factors and argued that the evidence for psi was weak or nonexistent despite the significant $p$-values. However, the Bayes Factors also failed to produce clear evidence that ESP does not exist.

Robust Bayesian Meta-Analysis, RoBMA for short, was developed to conduct Bayesian meta-analyses that take publication bias into account and can provide evidence for the nil hypothesis that the average true effect is zero. A Bayes-Factor that does not favor the broad alternative hypothesis of an effect is sometimes interpreted as “No evidence for an effect” (Maier et al., 2022). However, this conclusion is not warranted if the model also estimates heterogeneity around this average. Heterogeneity implies that some estimates reflect true effects. Moreover, the average effect size of positive results is implied to be greater than zero (Schimmack, 2026).

The main problem with RoBMA is that it s built on top of existing bias-correction models with strong assumptions, specifically step-function selection models (Vevea & Hedges, 1995; Hedges & Vevea, 1996). The most critical assumption of this model is that population effect sizes are normally distributed. When this assumption is violated because the dataset contains few negative results, the model assumes that selection bias removed negative results and produces a lower estimate closer to zero. In extreme cases, it can estimate that the mean is zero and find evidence for a false nil hypothesis about the mean.

When RoBMA was used with data from a meta-analysis of nudging interventions, the model adjusted the average effect of approximately $d=.43$, to d = .07. However, another bias correction methods that do not assume a normal distribution, produced an estimate of d= .44 (Schimmack, 2026b).

RoBMA Analysis of Government Nudging Studies

DellaVigna and Linos (2022) used results from large government nudge trials and compared them with results from broadly comparable academic studies. The difference was striking. In the academic literature, nudges changed behavior by about 8.7 percentage points on average. Thus, if 20% of people performed a behavior in the control condition, a typical effect of this magnitude would increase the rate to about 29%. In the government trials, however, the average effect was only about 1.4 percentage points. In the large government studies, this finding was still statistically significant, but the same effect size might be too weak in other studies to reject the nil hypothesis. Thus, the results of this study seem to be consistent with RoBMA’s bias-corrected estimate. However, an alternative explanation is that there government studies have smaller effect sizes because they are different from the nudging manipulations in academic studies. To explore these competing hypothesis, I analyzed the data with RoBMA and zcurve3, a model that does not assume a single normal distribution of population effect sizes.

The authors of RoBMA already analyzed these data, but they did not use RoBMA (Maier et al., 2024), although a reviewer suggested it. They argued that it would be inappropriate to use RoBMA with these data because the data do not fit a single normal distribution. This is an important caveat because it makes the distribution assumption salient and because the authors applied RoBMA to the academic meta-analysis without testing the distribution assumption.

Here I applied RoBMA to data that are known to violate the distribution assumption. The result was similar to the one for the academic meta-analysis: evidence for an overall mean effect was inconclusive and slightly favored the null model (inclusion BF = 0.46). The model-averaged estimate of the mean effect was $\mu = 0.016$, 95% credible interval [-0.413, 0.502]. In other words, “no evidence for an average effect in government nudging studies.” Again, this finding does not mean that nudging interventions do not work. Given large heterogeneity around the average, The analysis provided overwhelming evidence for heterogeneity in true effects (inclusion BF > 14,999), with an estimated heterogeneity parameter of $\tau = 3.34$, 95% credible interval [3.07, 3.63]. However, the results imply that nudging studies also often backfire, but that these negative results are missing, publication bias (inclusion BF > 14,999).

Figure 1 from Maier et al. (2024) shows the problem for the weight-function selection model underlying RoBMA. The distribution has a long positive tail that does not fit a single normal distribution (blue). A model that only fits the right side and attributes the low frequency of negative results to selection bias (red line). However, even this model does not fit the data well, but RoBMA has no other way to represent the data.

However, we can help RoBMA by removing extreme values from the data on both sides. However, because there are many more positive extreme values than negative extreme values, the exclusion of extreme values lowers the average of the observed data from 1.38 percentage points (pp) to 0.74 pp. In contrast, the median did not change much (0.50 vs. 0.42 pp).

Applying RoBMA to these data changed the estimate of the average effect size dramatically. The Bayes Factor now clearly rejected the nil hypothesis, BF > 14,999 (the maximum value in RoBMA). The estimated average was 0.58, only slightly lower than the simple average of 0.74 pp. This estimate also does not take the extreme positive results into account. The main point, however, is that RoBMA underestimated the average effect size when a long tail of positive results was included because it imagined a long tail of matching negative results based on the symmetry of the normal distribution. As Maier et al. (2024) already stated, this means RoBMA should not be used when data are likely to have a different distribution.

Maier et al. (2022) computed their RoBMA result ignoring the clustering of standard errors — as stated in their own figure caption. This is not a minor detail. The 448 effect sizes come from 212 studies, and treating dependent estimates as independent inflates the apparent precision that the bias correction then acts on. Without clustering, RoBMA returns a mean of essentially zero and evidence against an effect under every bias specification I tried — the full model-averaged ensemble, and a selection-model-only version with PET-PEESE removed (μ = 0.02, BF favoring the null). Once the clustering is modeled — using the multilevel extension of RoBMA (Bartoš, Maier & Wagenmakers, 2026) — the evidence reverses decisively (inclusion BF > 14,999) to a clearly positive mean, between d = 0.22 and d = 0.34 depending on the bias model. The published conclusion of “no evidence for nudging” is an artifact of ignoring the dependence structure of the data.

How Robust is RoBMA?

In statistics, robustness traditionally refers to methods that remain reliable when assumptions are violated or when data contain outliers or other atypical observations (e.g., Box, 1953; Huber, 1964; Tukey, 1960). Calling RoBMA “robust” may therefore give users the impression that it has these properties. However, robust in RoBMA refers primarily to averaging across a predefined set of meta-analytic models, not to robustness against misspecification of assumptions those models share. Bayesian model averaging can hedge against uncertainty about which model in the set is correct; it cannot protect against an assumption that every model in the set makes.

That distinction is important because RoBMA, like every statistical model, can produce misleading estimates when its assumptions are poorly suited to the data. In the present analyses, results depend strongly on assumptions about the distribution of population effect sizes and the treatment of extreme or highly heterogeneous effects. Other plausible models can produce substantially different conclusions.

The practical lesson is therefore that its results should not automatically be treated as the most robust or most credible estimate. When different models give different answers, the scientifically important task is to examine why they disagree and which assumptions are plausible for the data at hand.

Hu, B., Z. Xia, Q. Guo, C. Lu, S. Constantino, and X. Ju. 2025. “ Assessing Nudge Impact: A Comprehensive Second-Order Meta-Analysis.” Journal of Behavioral Decision Making 38, no. 5: e70053. https://doi.org/10.1002/bdm.70053.

Post-Publication Review

Hu et al. present a second-order meta-analysis of 14 meta-analyses covering 1,638 studies and nearly 30 million participants. They report an unadjusted mean effect of $d=.27$, while PET-PEESE reduces the estimate to approximately zero.

The main problem is that adding more and more studies does not solve the basic problem of extreme heterogeneity. The authors report $I^2=99.89\%$, indicating that the included meta-analyses differ enormously in their estimated effects. Under these conditions, the question “What is the effect of nudging?” is not well defined. Defaults, reminders, food placement, information labels, and many other interventions are not repeated estimates of one common treatment effect.

The publication-bias results do not resolve this problem. In fact, they disagree. The conventional random-effects estimate is $d=.27$. RoBMA gives $d=.29$, 95% CI [.15, .43], with $\tau=.26$. Trim-and-fill still gives $d=.27$, and using adjusted estimates from component meta-analyses gives $d=.28$. Only PET-PEESE produces the near-zero estimate of $d=.003$–.004.

It is therefore misleading to foreground the PET-PEESE result as if different methods converge on zero. RoBMA already averages over PET-PEESE, selection, and no-bias models, yet its model-averaged estimate remains positive. Moreover, PET-PEESE relies on the relation between effect size and standard error, which can be confounded by genuine heterogeneity. In a literature where small and large studies differ in intervention type, setting, population, and outcome, that relation need not reflect publication bias alone. A separate weight-function estimate would have been informative, but none is reported.

More fundamentally, even a perfectly estimated grand mean would tell us little. The authors themselves show that effect sizes differ substantially across categories and domains; for example, decision-structure nudges have a descriptive estimate of $d=.40$, environmental nudges $d=.45$, and food nudges $d=.33$. Their moderator tests are underpowered and do not explain the heterogeneity, but failure to explain heterogeneity does not make the grand mean meaningful.

The article therefore does not tell us much that is new about nudging. We already knew that published effects are heterogeneous and that publication bias is a concern. Pooling ever larger numbers of heterogeneous studies does not answer the scientifically important questions: Which nudges work? How large are their effects? Under what conditions do they work?

The conclusion that the “true effect of nudging” is zero has the appearance of precision, but it is closer to answering an ill-defined question. The problem is not whether the grand mean is $d=.27$ or $d=.004$. The problem is that there is no single effect of nudging to estimate.