A Critical Review of the Cybernetic Theory of Meta-Traits

To understand psychological science, it is necessary to understand how academia works. Research at universities is organized around paradigms (Kuhn, 1962). A paradigm consists of shared ideas, assumptions, methods, and practices—something like a scientific culture.

Psychology is organized into broader research paradigms that roughly correspond to scientific disciplines. Developmental psychologists study children and developmental changes. Experimental cognitive psychologists study mental processes, often using computer-based tasks and reaction times. Personality psychologists typically collect data from moderately large samples (often N = 100 or more), measure individual differences on multiple variables, and examine the correlations among these measures. This approach is often described as multivariate or correlational, in contrast to experimental research that manipulates independent variables to study their effects on specific outcomes.

Within the personality paradigm, we can distinguish smaller paradigms, or mini-paradigms, organized around specific theories, constructs, or measurement instruments.

For example, the life-satisfaction paradigm, strongly associated with Ed Diener’s research, focuses on subjective evaluations of people’s lives rather than an objectively defined concept of happiness. Much of this research also relies on the Satisfaction With Life Scale developed by Diener and colleagues.

Other paradigms are organized around specific models and instruments for measuring personality. The Big Five paradigm can itself be divided into smaller research traditions that use different instruments to operationalize the five broad personality dimensions. Competing paradigms, such as HEXACO, propose alternative structures and corresponding measurement instruments.

One Big Five mini-paradigm is the Cybernetic Meta-Trait Paradigm, developed by Colin DeYoung in collaboration with Jordan Peterson and colleagues beginning in 2002.

The aim of this blog post is to examine what we have learned from more than 20 years of research in this paradigm. To do so, I conducted a literature review with the assistance of my AI collaborator, which can locate, read, and summarize scientific publications much faster than I can. The objective is not simply to count publications or summarize findings, but to evaluate what this research program has contributed to our scientific understanding of personality.

What Is the Cybernetic Metatrait Paradigm?

The empirical starting point was Digman’s (1997) observation that the Big Five personality dimensions are often systematically correlated.

Digman proposed that these correlations might reflect two broader personality dispositions, which he called Alpha and Beta. These names were descriptive placeholders rather than explanations of the psychological processes responsible for the correlations.

DeYoung, Peterson, and Higgins (2002) proposed the more theoretically meaningful labels Stability and Plasticity.

Stability refers to the common variance among Agreeableness, Conscientiousness, and Emotional Stability (low Neuroticism). Plasticity refers to the common variance between Extraversion and Openness to Experience.

However, giving statistical factors psychologically meaningful names does not establish what they are. Correlations among self-reported personality characteristics cannot, by themselves, reveal the psychological mechanisms responsible for those correlations.

DeYoung subsequently developed a more comprehensive theory, presented in Cybernetic Big Five Theory (2015). This theory conceptualizes personality as a system of goal-directed regulation and adaptation. Stability reflects the maintenance of goals and organized functioning in the face of disruption, whereas Plasticity reflects exploration, adaptation, and engagement with new possibilities.

The theory thus moves beyond describing correlations among personality measures toward proposing psychological functions that might explain them.

But proposing a theory and finding empirical observations consistent with it does not establish that the theory is correct. This is particularly important in correlational research, where different causal models can generate similar patterns of observed associations. Convincing evidence for a theory therefore requires also to present evidence against competing theories. However, this work is rarely conducted by researchers within a paradigm because it acknowledges competing paradigms and the possibility that the assumptions of ones own paradigm might be false.

The central question of this review is whether research within the cybernetic meta-trait paradigm has produced findings that answer broader questions in personality research outside the paradigm.

Do the Metatraits Even Exist?

Much of the research conducted over the past two decades has focused on a more fundamental question: Do Stability and Plasticity actually exist? Are the Big Five dimensions genuinely correlated because of broader personality dispositions, or do their observed correlations arise from the way personality is measured?

Ashton and Lee (2008) challenged the interpretation of higher-order factors by emphasizing the effects of measurement and the heterogeneity of the traits represented by broad personality scales. Most research on metatraits has relied on Big Five sum scores. However, these scores combine variance associated with broad personality dimensions with variance associated with narrower characteristics that may overlap across domains.

For example, an Extraversion scale emphasizing Excitement Seeking may correlate with an Openness scale emphasizing Novelty Seeking. Both measure aspects of an adventurous disposition. In contrast, an Extraversion scale emphasizing sociability and an Openness scale emphasizing enjoyment of intellectually demanding activities, such as solving puzzles, may show much weaker correlations. Consequently, correlations among Big Five sum scores do not necessarily imply the existence of higher-order personality dispositions. They may reflect the particular content of the instruments used to measure the Big Five.

An important empirical challenge is therefore to establish that the proposed higher-order structure is not an artifact of overlapping item content, secondary factor loadings, or evaluative response biases. Recently, DeYoung and colleagues (2026) published another article arguing that Stability and Plasticity are genuine higher-order personality traits. I have critically examined the methodological problems underlying this claim in several previous blog posts (Schimmack, 2019, 2020, 2022, 2026). The most recent analysis, Lost in Factor Space: The Search for Personality Meta-Traits (Schimmack, 2026), specifically challenges the use of estimated Big Five factor scores to test hypotheses about correlations among latent personality traits.

The present review addresses a different question. For the sake of argument, I will assume that Stability and Plasticity are genuine personality dispositions. What have more than two decades of research taught us about their psychological nature, and what contributions have these constructs made to broader theories of personality?

For the sake of argument, I will assume that Stability and Plasticity are genuine personality dispositions. I will also accept the proposed hierarchical organization in which these broad dispositions influence the Big Five and, through them, narrower personality traits such as helpfulness, courage, argumentativeness, and neatness.

The question is what follows from accepting these assumptions.

What have 20 years of research taught us about the psychological nature of Stability and Plasticity? What role do these metatraits play in explaining personality and behavior?

More specifically, has the Cybernetic Metatrait Paradigm produced findings that are important for personality psychology generally, or has it mainly generated research of interest to scientists already committed to the metatrait framework?

A productive scientific paradigm should do more than demonstrate the existence of the constructs around which it is organized. It should clarify their nature, identify the mechanisms responsible for them, and explain phenomena that competing theories cannot explain equally well.

What Do Metatraits Contribute to the Prediction of Behavior and Life Outcomes?

Personality psychology has three major aims: to identify and classify personality traits, to investigate their causes, and to understand how they influence behavior, feelings, and thoughts.

Personality psychology has arguably made the greatest progress toward the third aim. Personality measures predict a wide range of behaviors and consequential life outcomes, sometimes over remarkably long periods. For example, personality characteristics assessed around the time of marriage can predict the likelihood of divorce years later.

These findings demonstrate that relatively stable individual differences contain information about people’s future lives. The challenge is to understand which dispositions are responsible for these predictions and how they influence particular outcomes.

This is where the cybernetic metatraits face a fundamental problem.

Metatraits are, by definition, more remote from specific behaviors and outcomes than the narrower personality traits they are supposed to influence. The theory proposes that Stability and Plasticity contribute to individual differences in the Big Five, which in turn contribute to more specific personality dispositions.

Consequently, under a hierarchical common-cause model, the influences of metatraits on specific outcomes should be reflected in the narrower dispositions through which they operate.

An Example: Personality and Life Satisfaction

One of the most robust findings in life-satisfaction research is that Neuroticism is among the strongest Big Five predictors of life satisfaction. People who are prone to negative emotions, worry, and dissatisfaction tend to evaluate their lives less favorably.

This is not a particularly surprising finding. What makes it important is the stability of these individual differences and their predictive associations over long periods.

However, Neuroticism is a broad personality dimension that includes several distinguishable characteristics, such as anxiety, irritability, emotional vulnerability, and depressiveness.

These narrower traits need not have identical relationships with life satisfaction.

For example, irritability may create interpersonal problems without necessarily having the same relationship with a person’s own life satisfaction as depressiveness. Anxiety may produce unpleasant experiences, but it can also motivate vigilance and avoidance of threatening situations.

The Depression facet of Neuroticism is the strongest predictor of low life-satisfaction and it is stronger than the broad Neuroticism disposition (Anglim et al., 2020; Schimmack et al., 2004). This finding shows that moving down the hierarchy towards specific traits can increase prediction and clarify the meaning of findings. So what can the meta-trait of Stability add to this finding?

Stability is defined as a higher-order disposition contributing to individual differences in Emotional Stability, Agreeableness, and Conscientiousness. For the sake of argument, assume that Stability really exists as a common psychological cause of these three personality dimensions. How would this improve our prediction of life satisfaction? It doesn’t.

Suppose depressiveness is the strongest specific predictor of life satisfaction. If Stability influences depressiveness through Neuroticism, then some portion of depressiveness reflects the influence of Stability. But this influence is already represented in the depressiveness measure. Other components of Stability could predict life satisfaction through additional pathways. For example, Conscientiousness might contribute through goal achievement, and Agreeableness through interpersonal relationships. Yet these pathways can be investigated directly by measuring the relevant narrower traits. A broad Stability score does not explain which mechanisms matter. Indeed, it risks obscuring the different ways in which personality influences life satisfaction.

A Mathematical Problem for the Predictive Utility of Metatraits

The problem is not merely that metatraits are more abstract than the Big Five. It follows from a basic property of multiple regression. Adding predictors cannot reduce the proportion of variance explained by an ordinary least-squares regression model in the same sample.

Consider Plasticity, which is defined by the shared variance between Extraversion and Openness. If we measure Extraversion and Openness separately, we can use both variables to predict any outcome. Alternatively, we can combine them into a single Plasticity score.

But the second approach cannot explain more variance than a multiple regression using both Extraversion and Openness, provided Plasticity is computed as a linear combination of those two scores. The multiple regression is free to assign different weights to Extraversion and Openness. It can therefore reproduce the prediction obtained from the Plasticity composite or improve upon it.

More generally, a multiple regression using all five Big Five scores must explain at least as much variance as one using only Stability and Plasticity scores derived from those same variables.

This is not an empirical hypothesis about the relationships between personality and life outcomes. It is a mathematical property of regression. Thus, metatraits have no inherent advantage in predicting outcomes once their constituent traits have been measured.

Why the Big Five Are Useful but Metatraits May Not Be

There is an important reason why broad personality measures such as the Big Five have become so successful. Each Big Five domain represents a large number of more specific personality characteristics.

A comprehensive assessment of personality might require hundreds of items covering dozens of facets. However, administering such extensive measures is often impractical. Big Five instruments provide an economical alternative.

For example, Neuroticism measures tend to capture variance associated with anxiety, stress sensitivity, emotional vulnerability, irritability, and depressiveness. A brief Neuroticism scale may therefore predict a variety of outcomes associated with these characteristics, even when it does not measure the most relevant specific trait very precisely.

Consider life satisfaction again. A short Neuroticism scale containing anxiety-related items may still predict life satisfaction because anxiety and depressiveness share substantial variance. The measure indirectly captures some of the depressive disposition that is particularly relevant to the outcome.

The practical advantage of the Big Five is that a small number of measures can cover a broad range of personality characteristics and predict many different outcomes. However, this advantage is much less obvious for the metatraits.

Plasticity is defined as the shared variance between Extraversion and Openness. Yet there is no independently established measure of Plasticity that does not rely on indicators of its constituent personality dimensions. Consequently, researchers typically estimate Plasticity from measures of Extraversion and Openness. Once these dimensions have been measured, however, averaging the two measures destroys important information about Extraversion and Openness.

Take life-satisfaction as an example, again. Extraversion is a positive predictor of life-satisfaction, but Openness is not (Anglim et al., 2020). These findings are informative because they suggest that characteristics associated with Extraversion, especially positive emotionality, are more relevant to life satisfaction than many of the cognitive and experiential tendencies associated with Openness.

Now suppose we average Extraversion and Openness to create a Plasticity score. We have combined a useful predictor with another variable that contributes comparatively little. Under these circumstances, averaging the two typically dilutes the predictive association and obscures the difference between the domains.

Rather than improving our understanding of life satisfaction, the Plasticity composite discards information that is scientifically relevant to explaining it.

What about Understanding Personality Traits?

The limited usefulness of metatraits for predicting behavior and life outcomes leaves understanding the structure of personality as the central objective of the metatrait paradigm.

Why are some personality traits correlated with each other? What psychological mechanisms produce these correlations? And why do some traits share variance while others are largely independent?

These questions are at the heart of Cybernetic Big Five Theory. DeYoung proposes that Stability and Plasticity are not merely statistical summaries of correlations among the Big Five, but broader psychological dispositions that help explain why these correlations exist.

The important question is what answers the meta-trait paradigm has provided to these questions. The theoretical assumptions are spelled out in DeYoung’s Cybernetic Big Five Theory (2015).

Cybernetics is the study of goal-directed systems that regulate their behavior through feedback. A thermostat is a simple example. It monitors the temperature, compares it with a desired state, and adjusts its actions to maintain that state.

Human behavior is much more complicated, but the general principle is similar. People pursue goals, monitor their progress, respond to obstacles, and adjust their behavior when circumstances change.

DeYoung argues that individual differences in personality reflect differences in the functioning of these regulatory systems. Within this framework, the two metatraits represent two fundamental requirements of adaptive behavior.

Stability is the capacity to maintain goals and organized functioning despite disruptions. It supposedly explains why people who are emotionally stable also tend to be conscientious and agreeable.

Plasticity is the capacity to explore new possibilities and adapt to changing circumstances. It supposedly explains why people who are extraverted also tend to be open to new experiences and ideas.

The theory is appealing because successful adaptation requires both functions. An organism that cannot maintain its goals will have difficulty achieving them. An organism that cannot explore new possibilities will have difficulty adapting when existing strategies no longer work.

But this general observation does not explain individual differences in personality. Everybody needs some ability to maintain goals and explore alternatives. The question is whether people differ in two general psychological capacities that produce correlations among the Big Five.

More specifically, what makes emotionally stable people more conscientious and agreeable? And why should people who enjoy social interactions also enjoy intellectual exploration?

Can We Identify the Biological Causes of Metatraits?

It is difficult to study the causal mechanisms underlying personality traits. Personality traits are relatively stable individual differences, and their biological causes are likely to involve complex interactions among genes, brain systems, and environmental influences. The challenge becomes even greater when the aim is to explain abstract dispositions such as Stability and Plasticity, rather than narrower characteristics such as anxiety, sociability, or impulsivity.

Nevertheless, Cybernetic Big Five Theory makes some relatively straightforward predictions that can be tested empirically.

The theory proposes that dopamine contributes to Plasticity, whereas serotonin contributes to Stability. One way to test these hypotheses is to examine whether drugs that alter these neurotransmitter systems produce changes in personality traits.

For example, if serotonin is a common biological cause of Stability, drugs that alter serotonergic functioning should produce coordinated changes in the three constituent Big Five dimensions: Neuroticism, Agreeableness, and Conscientiousness. Similarly, if dopamine is a common cause of Plasticity, drugs that alter dopaminergic functioning should affect both Extraversion and Openness.

Of course, neurotransmitter systems are complicated. Dopamine and serotonin act through different receptors and pathways, and pharmacological interventions are not perfectly selective. Moreover, acute changes in behavior are not necessarily equivalent to enduring changes in personality.

However, these complications do not make pharmacological studies irrelevant. They make it necessary to specify which biological mechanisms are hypothesized to influence the meta-traits and what pattern of personality changes should follow from manipulating them.

What Have Pharmacological Studies Found?

There is evidence that drugs affecting serotonergic functioning can change personality scores.

For example, Knutson et al. (1998) administered the selective serotonin reuptake inhibitor (SSRI) paroxetine or a placebo to healthy volunteers for four weeks. The treatment reduced negative affect and hostility and increased affiliative behavior.

Tang et al. (2009) conducted a placebo-controlled study of paroxetine in patients with major depression. They found substantial reductions in Neuroticism and increases in Extraversion. These effects remained significant after statistically controlling for improvements in depressive symptoms.

These findings are important because they demonstrate that pharmacological interventions can produce changes in personality measures.

However, the observed pattern does not correspond to the predictions of a simple serotonin–Stability theory.

Lower Neuroticism is consistent with an increase in Stability. But the studies do not establish corresponding increases in Agreeableness and Conscientiousness. Moreover, paroxetine increased Extraversion, which belongs to Plasticity rather than Stability.

Thus, the existing findings suggest that serotonergic interventions influence particular personality characteristics rather than demonstrating coordinated changes in the traits defining Stability.

The evidence for dopamine and Plasticity is even more limited.

Researchers have manipulated dopaminergic functioning using drugs such as haloperidol and sulpiride. However, the studies reviewed here primarily examined whether personality moderated the effects of these drugs on cognitive or behavioral tasks.

Such findings do not demonstrate that the drugs change personality traits.

The relevant experiment would be to administer a drug that alters dopaminergic functioning and examine whether Extraversion and Openness change together, as predicted by a common Plasticity mechanism.

I did not identify a controlled pharmacological study that provides this test.

Why Has There Been So Little Direct Experimental Research?

This raises a question about the scientific progress of the metatrait paradigm.

If dopamine and serotonin are central to the proposed explanations of Plasticity and Stability, why have researchers not conducted more experiments directly testing their effects on the constituent personality traits?

The metatrait paradigm has existed for more than 20 years. During this period, researchers have repeatedly examined whether Stability and Plasticity can be identified in personality data and whether metatrait scores correlate with various psychological outcomes.

Yet the central biological hypotheses have received comparatively little direct empirical investigation. Of course, pharmacological studies are expensive, ethically constrained, and difficult to interpret. Demonstrating changes in enduring personality dispositions may also require longer treatment periods and repeated assessments.

Nevertheless, the relevant research designs are feasible. Studies of antidepressant medications have already demonstrated that personality scores can change following pharmacological interventions. A research program that proposes biological causes of personality differences should ultimately investigate whether manipulating these causes produces the predicted changes.

What Has the Meta-Trait Paradigm Explained?

In conclusion, the cybernetic metatrait paradigm has produced few findings that substantially advance our understanding of personality beyond the paradigm itself.

Without independent measures of Stability and Plasticity, metatraits cannot improve the prediction of behaviors and life outcomes beyond the lower-order traits from which they are constructed. The Big Five, and especially their narrower facets, provide more detailed information about the personality characteristics that matter for particular outcomes.

The metatraits’ potential contribution is therefore primarily explanatory. If Stability and Plasticity are genuine higher-order dispositions, they might explain why certain personality traits are correlated.

However, after more than 20 years of research, the proposed biological mechanisms remain poorly understood. Theories linking Stability to serotonin and Plasticity to dopamine have generated relatively little direct experimental evidence. Moreover, the available pharmacological findings do not provide the coordinated changes in personality traits that would support a straightforward interpretation of these biological mechanisms.

This raises a broader question about the importance of the phenomenon that originally motivated the paradigm.

Digman (1997) observed surprisingly strong correlations among Big Five measures, challenging the assumption that the five major personality dimensions are largely independent.

Subsequent research demonstrated that some of these correlations reflect measurement artifacts, including evaluative biases and overlapping item content. When these influences are taken into account, correlations among the substantive Big Five dimensions are often substantially weaker.

This leaves a more limited theoretical question: Do the remaining correlations reflect systematic influences of broader personality dispositions, or do they arise from several distinct mechanisms that produce modest associations among particular traits?

That is a legitimate scientific question. However, its importance for personality psychology should not be exaggerated.

Most individual differences in personality are not captured by the two metatraits. They are found at lower levels of the personality hierarchy, including the Big Five and the many narrower dispositions that contribute to them.

These narrower traits are more closely related to the behaviors, emotions, and thoughts that personality psychologists seek to understand. They can also be connected to more specific psychological and biological mechanisms.

Consequently, research programs that focus on these lower levels of the personality hierarchy appear more promising for advancing both the prediction and explanation of personality.

The central limitation of the meta-trait paradigm is not that Stability and Plasticity may fail to exist. It is that their existence may not be particularly important.

References

Anglim, J., Horwood, S., Smillie, L. D., Marrero, R. J., & Wood, J. K. (2020). Predicting psychological and subjective well-being from personality: A meta-analysis. Psychological Bulletin, 146(4), 279–323. https://doi.org/10.1037/bul0000226

DeYoung, C. G. (2013). The neuromodulator of exploration: A unifying theory of the role of dopamine in personality. Frontiers in Human Neuroscience, 7, Article 762. https://doi.org/10.3389/fnhum.2013.00762

DeYoung, C. G. (2015). Cybernetic Big Five Theory. Journal of Research in Personality, 56, 33–58. https://doi.org/10.1016/j.jrp.2014.07.004

Schimmack, U. (2019a, August 13). When personality psychologists are high. Replicability-Index. https://replicationindex.com/2019/08/13/when-personality-psychologists-are-high/

Schimmack, U. (2019b, September 7). Testing hierarchical models of personality with confirmatory factor analysis. Replicability-Index. https://replicationindex.com/2019/09/07/testing-hierarchical-models-of-personality-with-confirmatory-factor-analysis/

Schimmack, U. (2019c, September 13). Confirmation bias is everywhere: Serotonin and the meta-trait of stability. Replicability-Index. https://replicationindex.com/2019/09/13/stability-serotonin/

Schimmack, U. (2020, October 25). No evidence for a higher-oder plasticity factor. Replicability-Index. https://replicationindex.com/2020/10/25/no-evidence-for-a-higher-oder-plasticity-factor/

Schimmack, U. (2022, January 25). More madness than genius: Meta-traits of personality. Replicability-Index. https://replicationindex.com/2022/01/25/more-madness-than-genius-meta-traits-of-personality/

Schimmack, U. (2026, October 31). Lost in factor space: The search for personality meta-traits. Replicability-Index. https://replicationindex.com/2026/10/31/lost-in-factor-space-the-search-for-personality-meta-traits/

Schimmack, U., Oishi, S., Furr, R. M., & Funder, D. C. (2004). Personality and life satisfaction: A facet-level analysis. Personality and Social Psychology Bulletin, 30(8), 1062–1075. https://doi.org/10.1177/0146167204264292

I first learned structural equation modeling with Ralf Schwarzer at the Free University Berlin. Alexander Grob taught me how to achieve good model fit when I was a post-doctoral student at the University of Illinois. Since then I have learned by experience and with a few tips from Linda Muthen. In the beginning, I would often encounter convergence problems. That is, the software (usually MPLUS) terminated without results. Over time, I learned how to diagnose these problems and avoid them. However, knowledge about these solutions is not widely shared and I see published articles that report conversion failure as a result. However, convergence failures are not a result. They reveal a problem in testing a model that may occur for a simple reason (e.g., not specifying a negative starting value for a negative loading).

I hope that AI will make it easier for people to find problems in their code (or just ask AI to write the code), but here are some resources that I (well AI) found that may help to avoid these problems.

———————————————————————————————

Yes. There are several useful resources in the SEM community, including dedicated troubleshooting guides, tutorials with executable code, and methodological papers on model identification. Some align closely with what you are proposing for OPeRA.

The most relevant resources are these:

1. Tutorials specifically addressing convergence and identification

Where Did I Go Wrong with My Model? Ten Tips for Getting Results in SEM

Structural Equation Modeling: A Multidisciplinary Journal (2023).

A practical troubleshooting article covering identification problems, poorly fitting measurement models, convergence warnings, and stepwise model development. Probably the best general starting point. Get Access

The Empirical Check for Identification

From A lavaan Compendium for Structural Equation Modeling in Educational Research.

Particularly relevant to your concern about whether adding secondary loadings or correlated residuals makes a model unidentified. It demonstrates how to generate data from the specified model and investigate whether its parameters can be recovered. tdjorgensen.github.io

Mplus User’s Guide — Chapter 14: General Convergence Problems

Official Mplus guidance on diagnosing optimization failures, examining TECH5/TECH8 output, changing starting values, rescaling variables, and estimating parts of a model separately before assembling the full specification. This is remarkably close to your proposed diagnostic approach. statmodel.com

Video lessons on SEM convergence problems

A short instructional series shared by an SEM researcher in the lavaan discussion community. The examples use Stata, but the underlying diagnostics apply to other SEM programs. groups.google.com

2. Especially relevant to your personality models

Two additional tutorials address the kinds of complex factor structures you have been working with.

Marsh & Alamer (2024): When and How to Use Set-Exploratory Structural Equation Modelling

This open-access paper compares CFA with set-ESEM, which allows selected cross-loadings rather than requiring every secondary loading to be zero. It includes real-data examples and lavaan code.

The authors show that restrictive CFA specifications can distort factor correlations and structural parameters. This is directly relevant to your concern about the inflated HH–Agreeableness correlation in Blötner’s model. Wiley Online Library

Bifactor ESEM Tutorial

This provides executable specifications for general and specific factors. It is useful for understanding why bifactor models can become difficult to identify and what constraints different parameterizations impose. bpb-us-e1.wpmucdn.com

3. What seems to be missing

The existing resources tend to concentrate on either statistical identification or technical troubleshooting.

What I find less developed is the kind of case-based model-development tutorial you have in mind:

Start with a theoretically interesting model that does not converge. Fit simpler nested models, identify the specific component responsible for failure, investigate whether the problem is numerical or structural, impose defensible constraints, and then assess whether the resulting model provides a substantively interpretable account of the data.

Such a tutorial would also distinguish three outcomes that researchers sometimes conflate: convergence, acceptable fit, and theoretical validity.

That would be especially useful for personality researchers working with many correlated facets, secondary loadings, hierarchical structures, and method factors.

4. An opportunity for OPeRA

I would not present OPeRA as introducing convergence diagnostics to SEM. The expertise and instructional materials clearly exist.

Instead, OPeRA could bring established SEM expertise into personality-structure research, using actual personality datasets to illustrate problems that arise in complex measurement models.

It could also invite methodological specialists to contribute reproducible tutorials, with Mplus and lavaan scripts and example data.

The Jorgensen identification chapter and Marsh and Alamer’s set-ESEM paper would be particularly good foundations.

In The Structure of Scientific Revolutions, Thomas Kuhn (1962) described how scientific communities develop around a paradigm. A paradigm involves more than a theory. It includes assumptions about what exists, which questions are worth asking, which methods are appropriate, and what constitutes acceptable evidence. These shared assumptions make cumulative research possible. However, they can also prevent researchers from recognizing the limitations of established practices.

Paradigms are like cultures, and like cultures, they can contain subcultures. For example, academic psychologists generally favor quantitative methods to provide empirical evidence for theoretical claims. Within academic psychology, experimental psychologists emphasize the manipulation of independent variables to test causal hypotheses. Personality psychologists, in contrast, rely primarily on correlational evidence because personality traits are difficult or impossible to manipulate directly.

This does not make personality psychology inherently less scientific. Astronomy also relies extensively on observational data, yet few would question its status as a science.

However, there is an important difference between astronomy and mainstream personality psychology.

Astronomers distinguish between the objects they study and the instruments used to observe them. A star is not defined by the measurements obtained with a particular telescope. Different instruments provide different information about the same object, and astronomers develop theories that explain these observations, taking the properties of the instruments into account.

Personality psychologists are less consistent in maintaining this distinction. Personality inventories often serve simultaneously as measurement instruments and definitions of the constructs they supposedly measure.

Consider Extraversion. Personality psychologists have proposed several theories about its psychological nature, emphasizing sociability, positive affect, reward sensitivity, or approach motivation. However, proposing a theory is not the same as providing empirical support for it. The existence of competing theories does not tell us which, if any, identifies the psychological disposition responsible for individual differences in Extraversion.

Meanwhile, researchers continue to use questionnaire sum scores labeled Extraversion as if the theoretical construct were already established. Different inventories combine different behaviors and characteristics, producing positively correlated but nonidentical scores. Yet there is no generally accepted explanatory model that specifies precisely what Extraversion is, demonstrates how its underlying disposition produces the observed responses, and establishes which inventory provides the most valid measurement.

Extraversion is therefore an established label for a family of correlated questionnaire scores, but not an equally well-established explanatory construct.

Another way to understand this problem is to compare personality taxonomies with geographical maps.

Maps represent real properties of Earth, including the distribution of land and water, elevation, and the location of geographical features. We can evaluate their accuracy by comparing them with observations of the physical world.

However, maps also use coordinate systems whose reference points are conventional rather than natural. Longitude provides a familiar example.

Greenwich became the internationally accepted reference for longitude in 1884. Before that, different countries used different prime meridians. The choice of Greenwich did not reveal a fundamental property of Earth. It established a common reference system that made navigation and geographical communication easier.

Changing the prime meridian changes the longitude assigned to a location. It does not change the location itself.

Personality inventories are similar to coordinate systems. They organize individual differences within a multidimensional personality space. The Big Five, the Big Five Aspect Scales, and HEXACO provide different ways of representing that space.

The existence of parallel and competing paradigms has advantages and disadvantages. The advantage is that competing paradigms can lead to empirical tests of competing predictions and lead to new discoveries. The disadvantage is that it is difficult to integrate research findings from different paradigms. For descriptive paradigms, the later problem can be resolved by translating findings from one descriptive system into another. For example, we can translate 15.00 Berlin into the local time in Sydney just like we can convert inches into centimeters or Fahrenheit into Celsius.

In personality space, we can use the correlations among dimensions from different inventories to translate findings from one system into another. A recent article by Ashton and Lee, the inventors of the HEXACO paradigm, provides an example. The authors focused on the description of pro- and anti-social traits with the Big Five Aspect Scales (BFAS) and their own HEXACO dimensions.

The BFAS uses two dimensions, Compassion (COM) and Politeness (POL), to represent individual differences in prosocial and antisocial characteristics. Although these names reflect the content of the scales, they do not establish that the dimensions measure compassion or politeness as independently defined psychological constructs. To avoid confusing labels with theoretical constructs, I will refer to these dimensions as BFAS-COM and BFAS-POL.

HEXACO also uses two dimensions to organize much of this domain: Agreeableness and Honesty-Humility. Importantly, HEXACO Agreeableness differs from Big Five Agreeableness, and Honesty-Humility refers to a particular combination of characteristics represented by its questionnaire items. I will therefore refer to these dimensions as HEXACO-A and HEXACO-HH.

Both systems provide two-dimensional representations of interpersonal individual differences. Specific characteristics, such as kindness, sympathy, helpfulness, forgiveness, or fairness, can be located within either system based on their correlations with the respective dimensions.

We can examine how closely these two systems correspond by calculating correlations among their four sum scores.

The correlations show that the two coordinate systems are related but not aligned. HEXACO-A correlates .29 with BFAS-COM and .50 with BFAS-POL. Similarly, HEXACO-HH correlates .33 with BFAS-COM and .48 with BFAS-POL. Thus, both HEXACO dimensions are more strongly associated with BFAS-POL than with BFAS-COM.

More importantly, these are not simply two different orientations of the same two-dimensional space. If they were, we could translate scores from one system into the other without losing information. Instead, each system captures information that is not represented by the other.

Some of this difference may reflect measurement error, but it may also reflect the fact that interpersonal personality characteristics cannot be adequately represented in just two dimensions.

The important conclusion is that BFAS and HEXACO provide different descriptive representations of interpersonal personality characteristics. Neither system can be assumed to represent the fundamental psychological structure simply because its dimensions were obtained through factor analysis.

The Confusion of Objects and Instruments in the HEXACO Paradigm

The main problem with personality paradigms organized around specific inventories is that researchers may come to treat their preferred descriptive framework as a theoretically privileged representation of personality.

Lee and Ashton (2026) provide an example. They examined the relationships between BFAS-POL and the HEXACO dimensions and concluded that BFAS Politeness is best understood as roughly an equal blend of HEXACO Agreeableness and Honesty-Humility.

This interpretation goes beyond demonstrating correlations among questionnaire scores. It suggests that the HEXACO dimensions provide the appropriate framework for understanding BFAS Politeness.

But why should we privilege the HEXACO coordinate system?

We would not claim that a location northwest of Greenwich is best understood as a blend of North and West. North and West are coordinates, not fundamental ingredients of a geographical location. A different coordinate system could describe the same location differently.

Similarly, the observation that BFAS-POL correlates with both HEXACO-A and HEXACO-HH does not establish that these HEXACO dimensions are the fundamental psychological dispositions underlying BFAS-POL.

Lee and Ashton go beyond describing correlations. They argue that personality measures must distinguish the six HEXACO dimensions for personality phenomena to be properly understood (p. 7). This is a theoretical claim about the privileged status of the HEXACO framework.

But the analyses do not establish that these six dimensions represent fundamental personality dispositions rather than one useful organization of correlated characteristics.

Observations obtained with one instrument do not explain observations obtained with another. Measurements from one telescope do not explain measurements from a second telescope. Both are observations that require an explanatory theory of the objects and processes producing them.

The same principle applies to personality questionnaires. Correlations among sum scores describe relationships among observations. They do not identify the psychological dispositions responsible for those relationships.

The methods needed to investigate these questions already exist. Structural equation modeling makes it possible to specify latent personality dispositions, distinguish them from their observed indicators, and test whether proposed measurement and structural models adequately reproduce the observed data.

I applied this approach to the same data used by Lee and Ashton, specifying a simple model of the relationships among the 40 items used to construct BFAS-COM, BFAS-POL, HEXACO-A, and HEXACO-HH. The model reproduced the broad pattern of sum-score correlations but failed to adequately reproduce the complete item covariance structure.

The purpose here is not to present an alternative model or claim that a particular personality theory is correct. The point is that an explanatory interpretation should be formulated as a testable model rather than inferred from correlations among questionnaire scores.

There is also a broader problem with the competing descriptive frameworks.

Neither BFAS nor the two HEXACO dimensions provide a comprehensive representation of prosocial and antisocial personality characteristics. The HEXACO dimensions capture much more of the information represented by BFAS-POL than by BFAS-COM. Conversely, HEXACO-HH represents characteristics that are only partly captured by BFAS-POL.

These differences suggest that interpersonal personality characteristics may require more than two dimensions for an adequate representation. Measurement error and differences in item content may also contribute to the imperfect correspondence.

The fundamental question is not whether BFAS or HEXACO provides the better map of personality space. It is whether we can develop and empirically test theories of the psychological dispositions that these maps attempt to represent.

Until we distinguish theoretical constructs from the instruments used to measure them, personality psychology risks confusing its coordinate systems with the underlying structure of personality.

Shaky Scientific (Epistemiological) Foundations of Personality Psychology

Personality psychology lacks solid scientific foundations. Behaviorism was impoverished, but at least it was based on clear rules. Only the effects of situational manipulations on observable behavior were admissible empirical observations, and theories were not allowed to invoke metaphysical constructs like thoughts and feelings to explain behavior.

Modern empirical psychology has moved on from behaviorism, but behaviorism still influences the thinking of psychologists without their awareness. Personality psychology survived behaviorism, but it kept one of its rules: causal questions can only be answered with experiments. This is a problem for a research area that cannot run experiments with random assignment to study causality.

Are taller people more confident because of their height? We cannot simply assign people to a tall or short condition, alter their height, and see what happens. Behaviorism simply decided that such questions are not psychological questions. Nowadays, hundreds of articles report correlations of personality traits with each other or other variables, but personality psychologists have not revised the way they think about causality.

For the most part, they have accepted that observational research cannot test causal theories and frame their research in terms of prediction: can my personality item sum score predict something of interest? For example, does a Conscientiousness scale predict GPA? The answer is yes, the correlation is about r = .2, and no definition of conscientiousness or causal theory is needed to explain the correlation.

However, personality psychologists sometimes make causal assumptions, even if these assumptions are not spelled out explicitly to avoid criticism of causal claims based on correlational research.

The most common causal assumption in personality research is that responses to similar items correlate because they measure a common trait. For example, responses to the items “I am talkative” and “I am outgoing” are correlated, and this correlation is used as evidence that both items measure a common trait, a personality disposition. This justifies averaging the responses to create a personality item sum score and calling it sociability. Importantly, the name of a personality item sum score is arbitrary. It could also be called T346, or “that thing that is reflected in the shared variance between responses to items like talkative and outgoing.”

No theoretical definition of a construct and no theory of measurement is needed. Sociability is whatever a specific sociability scale measures. Here lies the contradiction: a causal assumption, a common trait, is used to justify summing the items, and then an operational definition is used to avoid ever testing that assumption.

The problem with this approach to measurement becomes more obvious when we go beyond correlations among items that are synonymous or semantically similar. For example, take the items “I am often in a good mood” and “I am talkative.” Responses to these items are also positively correlated, but the items have distinct meanings. While it may be easier to imagine that people in a good mood talk a lot, it is also easy to imagine people who are in a good mood while absorbed in a solitary activity (flow) without engaging in any social interaction.

Research on the relationships among specific traits like sociability, good humor, anxiety, and anger is nearly 100 years old and took off when Thurstone (1934) developed multiple-factor analysis to study these relationships systematically. In the 1980s, the Big Five model emerged, and it has dominated personality psychology for nearly half a century. However, a clear definition of the Big Five is still elusive.

The reason is that personality psychologists rarely study the Big Five factors. Instead, they create more and more Big Five scales that average a set of items and are given Big Five names. The problem is that these sum scores depend on the items that are summed, and different sum scores produce different results. Thus, Extraversion is not a clearly defined construct. It is a label for scales that tend to correlate moderate to highly with each other because they contain overlapping sets of related items.

These sum scores cannot be used to study the Big Five factors. For example, we can create an Extraversion sum score by averaging the items “I am often in a good mood” and “I talk a lot.” Now we can show that this sum score correlates with a Good Humor scale that contains the item “I am often in a good mood” and with a Sociability scale that contains the item “I talk a lot.” However, these correlations are no longer empirical observations of something interesting. Scales that share items will be correlated, just as 6 and 9 are both divisible by 3 because both are products of 3 and another number (3 × 2 = 6, 3 × 3 = 9). The shared items guarantee the correlation, just as the shared factor of 3 guarantees divisibility.

Unfortunately, many personality psychologists seem unable to realize that these correlations do not tell us anything theoretically interesting or are simply not interested in scientific theories of personality. To examine why Good Humor and Sociability are correlated we need distinct measures of each construct not sum scores that average across items from both measures.

But when faced with a correlation between two observed measures, the alarm bells go off because we are now asking a causal question without experimental evidence. As this is considered improper, the question is never asked.

The simple solution to this problem is to shed the experimental dogma and treat personality psychology like any other science that cannot manipulate the objects it studies. Take astronomy as an example. We resolved the question whether Earth revolves around the sun or vice versa without randomly assigning planets and stars to different experimental conditions. Welcome to personality science, where we test causal theories without random assignment and without directly observable phenomena.

In personality science, Big Five theory makes clear predictions. It assumes that correlations among specific traits are influenced by five broader dispositions. People who are more often in a good mood are also more likely to be talkative because both dispositions reflect a broader disposition called Extraversion. Importantly, Extraversion is just a label, and it is confusing to use the everyday meaning of the term as if we knew what this broader disposition is. A better name might be Factor I, following the numbering of the factors in the lexical tradition (Norman, 1963; Goldberg, 1990), which evokes no associations with bubbly conversation partners.

The nature of Factor I is unknown, but we infer that it exists from the observation that Good Humor and Sociability are correlated. This makes Factor I a causal theory, and the theory could be wrong. Maybe there is another reason why the two traits are correlated. Personality psychologists avoid testing this theory by using sum scores and giving them labels. A sum score is simply a fact about how people answered a set of questions; no theory is needed to call it Extraversion. To keep the two apart, we can refer to factors by number (Factor I) and to scale scores by their labels, together with the instrument used: BFI Extraversion, BFI-2 Extraversion, NEO-FFI Extraversion, and so on. A scale can be used without examining whether it is related to Factor I, or without knowing what Factor I is. How well BFI Extraversion measures Factor I is an empirical question, and the answer may differ for each instrument.

From Exploratory Factor Analysis to Confirmatory Factor Analysis

Thurstone’s original method was exploratory factor analysis (EFA). It did not impose constraints on the relationships between specific traits and broader factors. Researchers chose the number of factors, and the method produced the best possible loadings (the relationships between observed variables and factors) given the correlations among the measures. The model usually did not reproduce all of the correlations, but fit to the data was not a major concern. Often the goal was not to test a theory, but simply to reduce a large number of items to a few sum scores.

Since then, the measurement methods have been developed further in a field called psychometrics. In psychometric measurement models, correlations among items are also explained by unobserved factors (often called latent variables), but these models differ from EFA in two important ways. First, researchers can specify a predicted pattern of loadings and thereby test a theory about why items are correlated; this approach is called confirmatory factor analysis (CFA; Jöreskog, 1969). Second, a good theory should explain the full pattern of correlations, so model fit matters.

Personality psychologists tried these methods to find evidence for Big Five theory, but their models did not fit the data (McCrae et al., 1996). Instead of asking whether their theory might be at least partially wrong, they concluded that confirmatory factor analysis is not appropriate for personality questionnaires. To this day, many of them prefer exploratory methods that never put a specific structural hypothesis to the test.

This is akin to astronomers refusing to use more advanced telescopes and working with the one invented by Galileo. Not surprisingly, little progress has been made. The structural model established in the 1980s has barely been tested, let alone improved.

Using CFA with theoretically constrained loading patterns, it is actually possible to find evidence for the Big Five factors. They do not explain all of the relationships among specific traits, which explains earlier failures, but many traits are related to one or more of the Big Five factors.

This makes it possible to create measurement models of the Big Five without knowing what they are. The shared variance among a set of specific traits is attributed to the causal influence of a broader trait. This does not mean that the Big Five really exist; until more evidence emerges, we can study them as “hypothetical constructs.” For example, we can examine whether Factor I predicts outcomes like life satisfaction or whether Factor III predicts GPA.

Many psychologists, especially older ones socialized during the era of behaviorism, are reluctant to view factors as at least hypothetically real entities (Revelle, 1983, 2016) . However, astronomy shows how a science can make progress by studying hypothetical constructs. Black holes are a useful reminder that some things are real even though we can never see them. Nothing escapes a black hole, not even light, so no telescope will ever show us one. Astronomers know that black holes exist only through their effects on things we can observe, such as stars orbiting an invisible mass or light bending around empty space. No astronomer would say that black holes are unscientific because they are unobservable. Yet this is exactly the rule behaviorism imposed on psychology: if we cannot observe it, it does not belong in science. Personality traits are like black holes. We cannot see them, but we can infer them from their effects on what people say and do. The scientific question is not whether something is observable, but whether the indirect evidence is strong enough to justify the inference. For the Big Five, that evidence is modest, but the reason is that nobody has even tried to find evidence for or against them. If we never test a theory, it’s validit remains unknown. And we cannot test it by correlating more and more personality item sum scores with each other. At some point, we have to study variables that are not directly observable like the Big Five factors with measurement models that rely on the indirect evidence of correlations among specific traits that are easier to measure.

Do Meta-Traits Produce Correlations Among the Big Five?

Some personality psychologists have proposed that there is a set of even broader personality dispositions than the Big Five. This theory was originally proposed by Digman (1997) who found consistent patterns in correlations among Big Five scales. For example, Extraversion scales tended to correlate with Openness scales. These days, the strongest proponent of this theory is Colin DeYoung, who renamed the factors Plasticity and Stability.

The main problem for this theory is that it rests on correlations among Big Five scales, not Big Five factors. Even if BFI-Extraversion and BFI-Openness scales correlate positively, it does not provide evidence that the corresponding factors, Factor I and Factor IV, are correlated with each other. The reason is that scale scores also contain information about the specific traits that were used to create the sum scores. To study the relationship among Big Five factors, we would first need to create a measurement model of the Big Five factors and then examine whether a model with Meta-Traits fits the data and whether it fits the data better than alternative models.

Even the latest article claiming support for the metatraits (DeYoung et al., 2026) did not provide this crucial test. The authors used exploratory factor analysis to compute five factor scores and then treated these scores as measures of the Big Five. But factor scores are not factors. As Revelle (2024) put it, “factor score estimates are not latent variables, they are merely weighted sum scores.” So, it can be both true that Extraversion and Openness sum scores created by DeYoung are positively correlated, but Factor I and Factor V are not.

The fact that 30 years after Digman, an article that uses sum scores to test theories about the correlation among factors underscores my observation that personality psychology as a field is stuck in behavioristic thinking, lacks clearly defined constructs, and lacks measurement models for Big Five scales. My invitation to collaborate on a proper test of Meta-Trait theory with proper measurement models was rejected by DeYoung. Others who are more interested in their own sum scores (inventors of the BFI-2 or HEXACO inventory) similarly showed no interested in testing personality theories.

To turn personality psychology into personality science does require more than just a change of journal names like “Personality Science” or “Social and Personality Science.” It requires a psychometric revolution that takes measurement and model fit seriously. I had little hope that such a revolution would happen during my life time, but AI is revolutionizing sciences as I am typing. So there is some hope that it may also force personality psychologists to become a science with explanatory theories.

Scientific revolutions usually happen when they give new researchers something new to do. I think applying psychometrics to the measurement of personality is timely (or should I say long overdue) and an opportunity for young scientists ready to make a mark. The availability of free software and the new norm to share data provides a unique opportunity for graduate students to do original research and publish articles in journals that are not controlled by guardians of the sum score paradigm.

References

DeYoung, C. G. (2006). Higher-order factors of the Big Five in a multi-informant sample. Journal of Personality and Social Psychology, 91, 1138–1151.

Digman, J. M. (1997). Higher-order factors of the Big Five. Journal of Personality and Social Psychology, 73, 1246–1256.

Goldberg, L. R. (1990). An alternative “description of personality”: The Big-Five factor structure. Journal of Personality and Social Psychology, 59, 1216–1229.

McCrae, R. R., Zonderman, A. B., Costa, P. T., Jr., Bond, M. H., & Paunonen, S. V. (1996). Evaluating replicability of factors in the Revised NEO Personality Inventory: Confirmatory factor analysis versus Procrustes rotation. Journal of Personality and Social Psychology, 70, 552–566.

Norman, W. T. (1963). Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. Journal of Abnormal and Social Psychology, 66, 574–583.

Revelle, W. (1983). Factors are fictions, and other comments on individuality theory. Journal of Personality, 51, 707–714.

Revelle, W. (2024). The seductive beauty of latent variable models: Or why I don’t believe in the Easter Bunny. Personality and Individual Differences, 221, 112552. https://doi.org/10.1016/j.paid.2024.112552

Thurstone, L. L. (1934). The vectors of mind. Psychological Review, 41, 1–32.

An open invitation to personality researchers

Personality psychology has produced numerous taxonomies of personality traits, but we still lack a widely accepted hierarchical measurement model that specifies how narrower traits relate to broader personality dimensions.

Although the Big Five are widely accepted as a descriptive framework, they lack a consensual measurement model. Moreover, other models like the HEXACO model exist without any attempt to create a unified model. This means that personality psychology still lacks a scientific taxonomy of personality traits. Instead, different researchers use different inventories, item selections, and statistical models, often producing different conclusions about personality structure.

How can we establish a common empirical foundation?

Like all other sciences, we have to start with a broad – not necessarily representative or exhaustive – list of objects that we want to classify. For personality psychologists these objects are specific personality traits. The key information for a classification system is how specific traits are related to each other. A key problem is that the standard method used by personality psychologists (EFA, PCA) is limited and produces new structures with every new set of traits. The solution to this problem is to use hierarchical structural equation modeling (or Hierarchical Factor Analysis, HFA) that imposes a theoretical structure on the data that does not change with the item set and correlations, but tests whether the data are consistent with the theoretical model.

I am inviting personality researchers, psychometricians, graduate students, and anyone interested in personality structure to participate in an open, collaborative many-analyst project.

This is a good time for such a project because data are openly available, software is free, and learning HFA is easier than ever with the help of AI. AI can even fit models for you. So, even personality scientists who are not familiar with structural equation models can formulate models and test them.

Participation does not require funding, new data collection, or specialized software. All you need is a computer. Call it theoretical or computational personality science.

The starting point for this project are data from the FacetMAP dataset (Irwing et al., 2024) that contains correlations for 77 facets, each measured with six items.

UlrichSchimmack/Personality-Trait-Taxonomy-PTT-: Developing Hierarchical SEM Models of Personality Traits

I created a GitHub repository that contains the data, syntax files for MPLUS and lavaan, and a model with five independent factors (the classic Big Five model). The model has a well fitting measurement model for the facets. Thus, researchers can focus on alternative models of the structural model that relates the facets to each other. However, modification to the measurement model are also possible. Starting with this model can save a lot of time and prevent convergence problems, but experienced users are free to start from scratch.

The immediate goal is to develop and evaluate measurement models that can help us answer several fundamental questions:

• Which facets can be measured reliably and distinctly?
• How are narrow personality traits related to one another?
• Can these relationships be explained by broader personality factors?
• How much of the shared variance reflects substantive personality characteristics versus method variance?
• Which structural conclusions remain robust across alternative, theoretically defensible measurement models?

Researchers are invited to contribute their own analytical approaches, challenge existing specifications, and propose alternative models.

Rather than trying to establish which researcher or existing taxonomy is correct, we want to identify results that withstand independent scrutiny and develop a measurement model that researchers with different theoretical perspectives can agree on.

All analyses, code, and findings can be shared openly, including results that challenge preferred theoretical models.

The FacetMAP dataset is only the beginning. Once we have established promising models, we can test whether the same structure emerges in other datasets and item pools, including large collections of personality-descriptive adjectives.

The ultimate goal is not another personality inventory. It is a cumulative, empirically grounded understanding of the hierarchical structure of personality traits.

Researchers interested in contributing are invited to explore the GitHub repository, run the existing models, and propose new analyses.

Ulrich Schimmack
University of Toronto Mississauga

Dozens of studies have correlated students’ personality with their GPA (Grade Point Average). The reason is simple. Students are easily accessible, and GPA is an easily measured objective outcome variable. The main finding of these studies is that personality predicts GPA. This is not very surprising to the average reader or students themselves, but in psychology, it was once common to dismiss personality as an explanation for behavior and behavioral consequences. Personality matters!

The more interesting question is which personality traits predict GPA and how much of the variance in GPA they explain. Study after study and meta-analyses of these studies show that Conscientiousness (being orderly, planning ahead, persistence, and low impulsivity and procrastination) is the strongest personality predictor of GPA (O’Connor & Paunonen, 2007; Poropat, 2009). Effort matters, and some students are more able and willing to work hard for a higher GPA. Nobody claimed that psychology is rocket science.

However, it may be surprising how weak the relationship between Conscientiousness and GPA really is. The typical meta-analytic estimate is a correlation of approximately r = .24 (O’Connor & Paunonen, 2007; Poropat, 2009). This means that Conscientiousness explains only about 6% of the variance in GPA.

To make this correlation more meaningful, we can use it to compute the probability that a student who is above average in Conscientiousness also has an above-average GPA. Assuming a bivariate normal distribution, the probability is only about 58%, compared with 50% if Conscientiousness and GPA were unrelated.

This blog post uses data from a published journal article (McCredie & Kurtz, 2020) to examine three additional questions. First, do studies that rely on self-ratings underestimate the true relationship between Conscientiousness and GPA because self-ratings are imperfect measures of personality? Second, are informants, such as parents and peers, better judges of students’ Conscientiousness than students themselves? Finally, do evaluative biases in self-ratings reflect a real personality characteristic that predicts higher GPA, or are they merely rating biases?

The use of multiple raters also makes it possible to examine another interesting question: Does having a positive view of oneself predict higher GPA?

This possibility was suggested by Biderman et al. (2019), who examined evaluative factors in Big Five and HEXACO personality questionnaires. They found that a general evaluative factor predicted GPA even after controlling for the individual personality domains.

[Insert image of Biderman et al. (2019)]

However, their analyses relied exclusively on self-ratings. This makes it difficult to determine whether the evaluative factor represents a substantive personality characteristic or whether it captures valid personality variance that is not adequately separated from evaluative bias.

A multi-informant design provides a stronger test of this question.

Results: How Valid Are Self- and Informant Ratings?

I used structural equation modeling to reanalyze the open data. The model distinguished common personality factors from evaluative factors that were specific to each rater. Here, I summarize the key findings.

All three raters contributed to the measurement of Conscientiousness. The standardized loadings on the common Conscientiousness factor were:

• Self-ratings: .79
• Parent ratings: .75
• Peer ratings: .64

These coefficients indicate how strongly each rating reflects the common Conscientiousness factor. Their squares suggest that the common factor accounted for approximately 62% of the variance in self-ratings, 56% in parent ratings, and 41% in peer ratings.

Thus, self-ratings were at least as strongly related to the common factor as parent ratings, whereas peer ratings showed somewhat weaker agreement.

These findings are broadly consistent with the literature on self–other agreement in personality judgments (Connelly & Ones, 2010). There is no compelling evidence that informant ratings are consistently and substantially more valid than self-ratings. Both methods provide useful information, and their relative validity may depend on the trait, the informant, and the circumstances.

How Strongly Does Conscientiousness Predict GPA?

The common Conscientiousness factor predicted GPA with a correlation of approximately r = .31 (SE = .09).

This estimate is somewhat higher than the typical meta-analytic correlation of r = .24 based on observed personality scores. The difference is consistent with the expectation that measurement error attenuates correlations between observed personality scores and external outcomes.

However, the difference is not dramatic. Even a correlation of .31 explains less than 10% of the variance in GPA.

Thus, correcting for measurement error strengthens the relationship between Conscientiousness and academic performance, but it does not fundamentally change the conclusion: Conscientiousness matters, but most of the variance in GPA remains unexplained.

Do Positive Self-Evaluations Predict GPA?

One of the advantages of the multi-informant model is that it can distinguish common personality variance from rater-specific evaluative variance.

Personality ratings are influenced by the desirability of traits. People generally prefer to describe themselves and others as conscientious, agreeable, extraverted, emotionally stable, and open to experience. As a result, ratings of different personality traits may be positively correlated simply because a rater has a generally favorable impression of the person being rated.

This phenomenon is commonly called a halo effect.

The model identified clear evaluative factors in all three rating sources: self-ratings, parent ratings, and peer ratings.

If these evaluative factors represented a substantive personality characteristic, we might expect them to show positive agreement across raters. For example, people who evaluate themselves especially favorably might also receive especially favorable evaluations from their parents and peers.

However, the correlations among the three evaluative factors ranged from −.45 to .09, and none was statistically significant.

There was no consistent positive agreement across raters. This pattern is consistent with the interpretation that the evaluative factors primarily reflect rater-specific tendencies rather than a common personality characteristic.

The second finding was equally interesting. The evaluative factors did not predict GPA.

In particular, the evaluative factor in self-ratings was not a significant predictor of GPA (β = −.12, SE = .14).

The sample size limits strong conclusions about small effects. Nevertheless, the results provide no evidence that students who evaluate themselves more favorably across personality traits achieve higher grades after controlling for their substantive personality characteristics.

This finding is relevant to Biderman et al. (2019), who reported positive relationships between evaluative factors and GPA in three large samples. Their standardized regression coefficients ranged from approximately .15 to .21.

The present results do not establish that those findings were statistical artifacts. However, they illustrate an important identification problem in single-rater studies. A general evaluative factor extracted from self-report items may contain substantive personality variance as well as evaluative bias. Predicting an external outcome does not, by itself, establish which component is responsible for the relationship.

The multi-informant model provides a way to separate these sources of variance. In the present data, the common Conscientiousness factor predicted GPA, whereas the self-specific evaluative factor did not.

What About the Other Big Five Traits?

Conscientiousness was not the only personality trait associated with GPA in this sample.

Neuroticism was a negative predictor (β = −.22), suggesting that students who were more emotionally unstable tended to have lower grades. Openness was a positive predictor (β = .15).

Extraversion (β = −.05) and Agreeableness (β = −.01) were practically unrelated to GPA.

These findings should not be interpreted as establishing a universal pattern. The effects of personality traits other than Conscientiousness tend to be smaller and less consistent across studies (O’Connor & Paunonen, 2007; Poropat, 2009). The present sample is too small to draw strong conclusions about these additional predictors.

Why Did Peer Ratings Predict GPA?

Perhaps the most interesting finding was that the unique variance in peer ratings of Conscientiousness was related to GPA.

At first glance, this might suggest that peers are better judges of Conscientiousness than students themselves or their parents. However, this interpretation is difficult to reconcile with the lower loading of peer ratings on the common Conscientiousness factor.

If peer ratings were generally more valid indicators of Conscientiousness, we would expect them to show stronger agreement with the other measures, assuming that the common factor captures the valid trait variance.

An alternative explanation is that peers rely heavily on academic performance when evaluating their friends’ Conscientiousness.

For example, students who receive high grades may be perceived as conscientious even if they are disorderly, impulsive, or unreliable in other areas of life. Conscientiousness is a broad personality trait, and students may be highly conscientious about their academic work without being equally conscientious in other domains.

This would make peer ratings useful predictors of GPA without necessarily making them better measures of general Conscientiousness.

Another possibility is that peers observe academically relevant behaviors that are less visible to parents and are not adequately captured by students’ self-ratings.

These explanations are not mutually exclusive. Facet-level data and more detailed information about what peers actually observe would be needed to distinguish them.

The broader lesson is that unique predictive validity does not necessarily imply superior construct validity. A measure can predict an outcome because it captures criterion-specific information rather than because it is a better measure of the intended broad personality trait.

Conclusions

Three results stand out.

1. Self-ratings and ratings by a single knowledgeable informant tend to be about equally valid. Some studies report differences in validity, but there is no consistent evidence that informant ratings are substantially superior to self-ratings. The present results are consistent with this broader conclusion.
2. Personality matters for real-world outcomes, but the effects are not huge. Conscientiousness is a reliable predictor of academic performance, and correcting for measurement error produces a somewhat stronger relationship. Nevertheless, most of the variance in GPA remains unexplained. Personality matters, but it does not determine destiny.
3. Evaluative factors in personality ratings appear to reflect primarily rater-specific bias. The lack of consistent positive agreement among evaluative factors across raters and their failure to predict GPA suggest that much of this variance reflects evaluative tendencies rather than a substantive personality characteristic. Even the evaluative component of self-ratings, which may reflect positive self-evaluation, provided no evidence of incremental validity for GPA.

As always, results from a single study are never conclusive. The sample is relatively small, and some estimates have substantial uncertainty. Nevertheless, the pattern of findings is broadly consistent with the existing literature on self–other agreement and the validity of personality ratings.

References

Biderman, M. D., McAbee, S. T., Hendy, N. T., & Chen, Z. J. (2019). Validity of evaluative factors from Big Five and HEXACO questionnaires. Journal of Research in Personality, 80, 84–96. https://doi.org/10.1016/j.jrp.2019.04.010

Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122. https://doi.org/10.1037/a0021212

O’Connor, M. C., & Paunonen, S. V. (2007). Big Five personality predictors of post-secondary academic performance. Personality and Individual Differences, 43(5), 971–990. https://doi.org/10.1016/j.paid.2007.03.017

Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996

Peer review is supposed to uncover errors in manuscripts. Sometimes it does. Sometimes it provides a rather nice demonstration of the errors humans make while trying to find errors in other people’s work.

I recently submitted a manuscript on publication-bias detection in meta-analysis. The central empirical result was simple: tests of publication bias often have low power when effects are heterogeneous and model assumptions are violated. Consequently, a nonsignificant test for publication bias should not be interpreted as evidence that publication bias is absent. The paper therefore argued for estimating how much publication bias remains compatible with the data, rather than relying on binary significance tests.

The paper was rejected. In his editorial letter, Associate Editor Stephen West wrote that there were “several statistical misstatements throughout the paper which need to be corrected.”

That sounds serious. So I went back through the reviews to identify these errors.

The exercise turned out to be more interesting than expected.

Error 1: What is statistical power?

The manuscript stated:

“Classical power is conditional on the null hypothesis being false and on a specified population effect size being the true effect size.”

Reviewer 2 responded:

“This is false. See a basic introductory statistics textbook such as Testing Statistical Hypotheses by Lehmann and Romano.”

This criticism is particularly interesting because it directly contradicts another criticism of my work that I tried to estimate “average power” without conditioning on a true effect size.

In other words, I was first criticized because power is conditional, I changed the terminology accordingly, and now I am now criticized because defining power as conditional is supposedly false.

Welcome to psycho-science.

Part of the confusion is historical. Lehmann’s Testing Statistical Hypotheses, first published in 1959, uses the classical mathematical-statistics concept of a power function, which gives the probability of rejecting the null as a function of the population parameter and can therefore be evaluated both under the null and under alternatives. That terminology was retained in later editions. In contemporary psychological research, however, statistical power is ordinarily used more narrowly for the probability of rejecting the null conditional on a specified alternative effect size—a usage central to prospective sample-size planning (e.g., Pek et al., 2024).

Maria Soto and I recently published a power primer specifically to disentangle these different usages of power (Soto & Schimmack, 2026). So the problem here is not that I fail to understand conditional probabilities. The problem is that different statistical traditions use the same word differently. It is particularly unhelpful for a reviewer to take one legitimate historical usage and declare another established usage simply “false” and for a lazy editor to repeat this claim without being aware that the term power has no clear definition in the psychological literature.

Error 2: You cannot manipulate the number of significant studies or the FDR

Reviewer 1 objected to two factors in the simulation:

“I don’t see that you can directly vary the number of significant studies or the FDR.”

Of course you can.

A simulation does not have to start with a fixed number of attempted studies and then passively accept whatever number of significant findings happens to emerge. One can condition the simulation on the evidence of interest.

If I want 300 significant results with an FDR of .20, I can generate 60 significant results under the null and 240 significant results under alternatives. This is no more mysterious than fixing the sample size, heterogeneity, or mixture proportions in any other simulation.

Whether this is a useful simulation design is a legitimate question.

Whether it is possible is not.

The reviewer additionally wrote that they did not see the relevance of FDR to meta-analysis. That is a substantive theoretical position. It effectively assumes that a heterogeneous literature should be represented as a continuous distribution of nonzero effects rather than as a mixture that can include true null effects.

But one purpose of the simulation was precisely to avoid making that assumption. The replication crisis has made it entirely plausible that some proportion of published significant findings are false positives. If that possibility is excluded by design, then publication-bias methods are being evaluated only in a world in which every study investigates a real nonzero effect. That is a very strong assumption to build into a simulation of publication bias.

Error 3: Zero effects make high power impossible

The manuscript reported:

“No method approached the conventional .80 benchmark when averaged across the full design.”

Reviewer 1 responded:

“But the full design includes zero effect sizes, doesn’t it? … So no test would have .80 power under those conditions.”

Here the reviewer has confused two entirely different meanings of power.

The quantity being evaluated was the power of a publication-bias test: if a simulated data set is biased, what is the probability that the test correctly rejects the null hypothesis of no publication bias?

It was not the power of the original studies to reject a substantive null hypothesis.

A literature in which every population effect is zero can nevertheless provide overwhelming evidence of publication bias. Suppose only 5% of results would normally be significant, but nearly all nonsignificant findings disappear. With enough observations, a publication-bias test could have very high power to detect that selection.

Thus, including studies with true null effects does not make .80 power impossible. If anything, strong selection is easier to detect when the underlying discovery rate is low, because publication bias produces a large distortion in the observed proportion and distribution of significant results. By contrast, when studies already have high power and most results are significant before selection, removing nonsignificant studies produces a much smaller observable distortion and publication bias becomes harder to detect.

Error 4: 5,184 simulation conditions is a miscalculation

Reviewer 2 wrote:

“The paper reports 5,184 conditions. I count 3 number of studies × 4 effect sizes × 3 heterogeneity values × 3 sample sizes × 4 FDRs × 3 publication bias settings which gets me to 1,728 so off by a factor of three. What am I missing?”

What was missing was one of the simulation factors: the three distributions of population effects.

The design was 3 x 4 x 4 x 3 x 4 x 3 x 3 = 5,184.

There were four, not three, levels of heterogeneity, and three effect-distribution conditions.

There is another small problem. Even the multiplication printed in the review,
3 x 4 x 3 x 3 x 4 x 3 equals 1,296, not 1,728.

So this particular attempt to identify an arithmetic error contains an arithmetic error.

Error 5: A beta distribution cannot have mean zero

Error 5: A beta distribution cannot have mean zero

Here the reviewers found a real problem.

The manuscript described a Beta(1,10) distribution as having mean 0 and standard deviation .4. A raw Beta(1,10) distribution obviously does not have those moments.

The problem, however, was in the description of the simulation, not in the simulation itself. The beta distribution was used to generate the shape of the effect-size distribution and was then transformed to have the intended mean and standard deviation. The manuscript failed to describe that transformation clearly.

In the era of open science, this distinction is unusually easy to resolve. The simulation code was available, so a reviewer who suspected that the reported moments were impossible could have checked whether the error was in the prose or in the actual data-generating process. The code shows that the simulation itself implemented the intended transformed distribution.

So this was a genuine reporting error that peer review correctly detected. But it was not evidence that the simulation results were based on an incorrectly specified Beta(1,10) distribution.

Error 6: The mysterious noncentrality parameter

Another reviewer objected to the statement that a simulated z statistic had a noncentrality parameter of 2:

“There is no noncentrality parameter for a normal distribution. Presumably the mean was 2?”

This is mostly terminology. Under an alternative hypothesis, a normal distribution is centered over a value other than zero. If all studies have 50% power with a significance criterion value of 1.96, the distribution is centered at 1.96 (~ 2).

In power calculations this quantity is routinely described as the noncentrality parameter, although unlike the t, F, and chi-square families there is no separately named “noncentral normal distribution.”

Calling it the mean of the alternative z distribution would avoid the terminological distraction. Call it what you want, but all sampling distributions have a median and when the effect size is not zero, the median is not zero. Whatever you call it, everybody can understand the meaning here and maybe propose a wording change rather than claiming that I invented a statistical entity that does not exist.

Again, this is very different from discovering that the simulation or its conclusions are wrong.

What was actually wrong?

After going through the reviews, I can identify one clear reporting problem: the description of the transformed beta distribution. There are also several places where the manuscript could have been clearer—the z-curve version numbers were particularly unhelpful, and some terminology could be tightened.

Those are useful comments.

Other alleged errors fare less well. The number of simulation conditions was correct. It is perfectly possible to condition a simulation on the number and composition of significant studies. Including zero population effects does not make it impossible for a publication-bias test to have high power. And the definition of power that one reviewer declared “false” is closely related to the very definition that other critics previously insisted I adopt.

None of this proves that the manuscript deserved acceptance. The paper was too long. Its argument could have been more focused. Including a conventional multi-step selection model would strengthen the comparison. The presentation of the various z-curve versions invited confusion.

These are legitimate criticisms.But those are different claims from saying that the paper contained “several statistical misstatements.”

ut those are different claims from saying that the paper contained “several statistical misstatements.”

There is also a simple reason why the manuscript contains no major statistical errors. Before submitting it, I used AI—both ChatGPT and Claude—to challenge my arguments, check calculations, and look for weaknesses. AI did not write the ideas for me, but it was extremely useful as an adversarial reader.

After receiving the reviews, I conducted a small experiment. I uploaded the anonymous manuscript to Claude and presented the reviews as if they were drafts of reviews that I had written and wanted evaluated. Claude was not particularly impressed. (It eventually figured out what was going on because it located the preprint with my name on it.)

Reviewer 1

“Your review raises several valid concerns, but about a third of the specific comments contain an error or overreach that the author could easily rebut. It also misses some issues that are stronger than the ones it leads with. My suggestions are below, in order of importance.”

Reviewer 2

“Reviewer 2 raises the most damaging points in either review: whether z-curve has been properly vetted, and the coverage claim. Several of the supporting comments, though, are factually off or point at the wrong target, and the author could rebut them easily.”

Conclusion

Peer review is commonly described as quality control, but anonymous peer review does not guarantee quality control. It is a human judgment process, and human judges make mistakes. Reviewers misunderstand arguments, overlook information, perform arithmetic incorrectly, import their own theoretical commitments, and sometimes mistake disagreements about terminology or modeling assumptions for factual errors.

Editors then have to evaluate those reviews. If they do not independently check the reviewers’ claims, reviewer errors can become editorial “facts.” In this case, an editorial decision referred to “several statistical misstatements,” even though several of the alleged mistakes were themselves mistakes in the reviews.

This is where AI changes the scientific publishing system. An AI system can read a manuscript, inspect a review, check arithmetic, distinguish a factual error from a modeling disagreement, and compare a criticism with the actual contents of the manuscript in seconds. It will also make mistakes, of course. But unlike traditional peer review, those judgments can be generated repeatedly, compared across systems, and audited immediately.

The interesting question is therefore no longer whether AI should be allowed into peer review. It is already better than pretending that two anonymous humans constitute a reliable error-detection system.

Journals may increasingly use AI to assist editors and reviewers. Authors will certainly use it to evaluate reviews. And eventually the importance of journal gatekeeping itself may decline. Scientific papers increasingly function as inputs into larger information systems—search engines, databases, systematic reviews, and now AI models—that integrate evidence across sources rather than treating publication in a particular journal as the final certificate of truth.

Peer review will probably survive. But its monopoly on deciding what counts as credible scientific information may not.

AI may eventually kill us, but I hope it will kill bad peer reviews first.

Editorial Letter and Reviews

Alesina, A., Carlana, M., La Ferrara, E., & Pinotti, P. (2018). Revealing stereotypes: Evidence from immigrants in schools (NBER Working Paper No. 25333). National Bureau of Economic Research. https://doi.org/10.3386/w25333

Abstracts are short and cannot tell the whole story. A previous working paper and data in this article show that the IAT was also used to predict grading and that there was no evidence that IAT scores predicted lower grades for immigrant students.

The study

Alesina, Carlana, La Ferrara and Pinotti asked whether Italian middle-school teachers’ implicit stereotypes about immigrants, measured with an immigrant/Italian-name IAT, show up in how they grade immigrant students. They also asked whether telling teachers their IAT scores changes their grading.

The design has real strengths:

• 1,384 teachers in 102 schools took the IAT.
• Grades came from administrative records, so teachers didn’t know their grading was being observed.
• Blind-graded national tests (INVALSI) provide a performance benchmark.
• A randomized field experiment varied when teachers received their IAT feedback.

The paper circulated as NBER Working Paper 25333 (December 2018). It was published in the American Economic Review in July 2024. Between the two versions, the evidence on the central predictive-validity question changed little, but the presentation changed a lot.

The predictive-validity test

The key analysis regresses end-of-year grades on an Immigrant × Teacher-IAT interaction, controlling for INVALSI scores and teacher fixed effects. Grades are from the 2011–2016 cohorts. If the IAT measures something that drives discriminatory grading, teachers with higher scores should show a larger immigrant penalty at equal test performance.

Working paper (2018). The authors analyzed math and literature separately.

• For math, one SD higher on the IAT (0.26 D-units) was associated with a 0.033-point larger immigrant penalty (SE 0.019). This was significant only at p < .10 in every specification.
• For literature, the estimate was zero (−0.001, SE 0.017).
• The literature null was explained after the fact by lenience toward first-generation immigrants. The supporting interaction was itself not significant.

Published version (2024). The subjects are pooled, standard errors are clustered by teacher, and the IAT is entered in raw D-units.

• The interaction is −0.075 (SE 0.050), p ≈ .13. The authors state that it is not statistically significant.
• Per SD of IAT, that is about 0.02 grade points, or about 0.02 SD of grades.
• The 95% CI excludes any effect larger than about 0.04 SD per SD of IAT.

So with a smallest effect of interest of d = .10, the data are evidence of no meaningful predictive validity, not merely a lack of evidence for it.

What changed between versions

1. The main test became a footnote to subgroup analyses. After the pooled null, the AER version splits students by INVALSI score.

• For high-ability students: −0.139 (SE 0.080), p ≈ .08.
• For low-ability students: −0.031 (SE 0.060).
• No test of the difference between subgroups is reported.

Even the high-ability estimate implies at most about 0.07 SD per SD of IAT at the CI bound. This split does not appear in the working paper. Yet the abstract-level summary in the introduction now says that higher IAT scores are associated with lower grades for high-performing immigrants. The conclusion calls the pattern “strongly suggestive of bias.”

2. Measurement error was invoked, and a new measure was introduced. The authors attribute the null partly to noise in teacher-level bias estimates. In an online appendix, they show that a “naïve” teacher-level bias measure is not significantly related to the IAT. An empirical-Bayes-shrunken version of the same measure is. This alternative outcome appears only after the preregistered-looking specification fails, which is a classic forking path.

3. The measurement error that matters most is in the IAT itself. The AER version reports a new fact (footnote 6): the male-name and female-name immigrant IATs correlate only 0.28. That implies a reliability of roughly .44 for the averaged score. The authors present this as reassuring because 76% of teachers received consistent categorical feedback. But a test-retest correlation of .28 places a low ceiling on any criterion correlation. It also means many teachers were told they were “moderately” or “strongly” biased on the basis of noise.

4. The figures and text overstate the tables. The text describing Figure 5 says higher IAT is associated with significantly lower grades for immigrants. Table 3, which quantifies the same relationship, says it is not significant.

5. The framing moved from prediction to intervention. The working paper’s abstract led with the IAT–grading correlation. The AER abstract omits it and emphasizes the experiments. The working paper’s key moderator result—that the feedback effect was driven by teachers without explicit anti-immigrant views—has moved to an online appendix.

6. The IAT critique literature is acknowledged, then set aside. The AER version cites the predictive-validity critiques, including Blanton et al. (2009), Oswald et al. (2013), and Schimmack (2021). It responds that those studies used small samples and lacked real-world behavior. This study has a large sample and real-world behavior, and it reproduces the weak-to-null pattern those critiques describe.

The experiments

Field experiment. Receiving IAT feedback before grading narrowed the native–immigrant grade gap by 0.35 points (pooled ITT; SE 0.11). This is larger than the entire conditional gap the paper documents (about 0.10 points). It came from raising immigrants’ grades and lowering natives’ grades.

Did teachers with higher IAT scores respond more? The triple interaction is 0.214 (SE 0.302), which is null. In the field, the IAT predicts neither discrimination nor responsiveness to feedback. A reaction to being labeled biased explains the effect as well as correction of a bias does.

Online experiment. This was added for the AER version. Of 179 teachers who completed the survey, 146 are analyzed, and the attrition isn’t explained in the main text. Each teacher graded ten vignette tests with randomly assigned names.

• The average effect of personalized feedback versus a generic debiasing message is zero (0.017, SE 0.122).
• The headline result is a moderator effect: teachers with higher IAT scores responded more to their own feedback.

One result here does favor the IAT. In the active control group (about 74 teachers), higher IAT was associated with lower grades for immigrant-named tests (−0.426, SE 0.157). Three things should be weighed against it:

• It comes from a small sample, in an exploratory interaction model.
• The outcome is vignette grading, not real grades.
• The group had just received a debiasing message, and teachers with IAT = 0 graded immigrant names 0.42 points higher than native names.

It does not replicate in the field data with real grades and about 780 teachers.

The AEA registry IDs (AEARCTR-0003647 and -0013570) appear to postdate the respective data collections. If so, the moderator analyses were not preregistered. This is worth confirming against the registry.

Conclusion

The working paper’s main predictive-validity claim was a marginal (p < .10) association in math and a null in literature. In the published version, with subjects pooled and errors clustered by teacher, the IAT does not significantly predict discriminatory grading. The estimate is small enough to rule out even modest effects.

The publication process did not converge on this conclusion. Instead it added subgroup splits, an alternative bias measure, and a second, small online experiment. That gave the final paper a narrative of implicit bias that is “strongly suggestive” in the conclusion but not supported by the primary test.

Two of the paper’s own facts deserve more attention than they received. First, about 70% of teachers have “moderate to severe” IAT scores, yet grading discrimination is about 0.1 points. Second, the two IATs given to the same teachers on the same day correlate .28. Both are hard to reconcile with the view that the IAT measures a stable implicit bias that shapes behavior.

In real classroom grading of immigrant students, the IAT failed to predict discrimination.

---

van Zwet, E., Gelman, A., Greenland, S., Imbens, G., Schwab, S., & Goodman, S. N. (2024). A New Look at P Values for Randomized Clinical Trials. NEJM evidence, 3(1), EVIDoa2300003. https://doi.org/10.1056/EVIDoa2300003

Introduction

Randomized controlled trials occupy a privileged position in evidence-based medicine because randomization protects against many sources of confounding that complicate observational research. Yet influential critics have questioned whether even randomized evidence can be trusted at face value. Ioannidis’s Why Most Published Research Findings Are False argued that low power, bias, and low prior probabilities can make statistically significant findings more likely to be false than true, and his argument explicitly included clinical trials (Ioannidis, 2005). Andrew Gelman has expressed related skepticism, arguing that evidence-based medicine encounters serious problems when the evidence is weak and that even “clean randomized clinical trials can fail to replicate.”

However, Ioannidis and Gelman differ fundamentally about what makes a significant finding false. Ioannidis’s framework assumes that some tested effects are exactly zero. A significant result is then a false discovery when it rejects one of these true nil hypotheses. Gelman rejects this zero-versus-nonzero framing as a useful description of scientific research. In his discussion with O’Rourke, he argued that researchers are usually not studying effects that are exactly zero and questioned attempts to infer scientific truth by separating published results into exact-null and non-null components (Gelman & O’Rourke, 2014).

This disagreement became explicit in the debate over Jager and Leek’s attempt to estimate the false discovery rate in medical research. Using 5,322 reported $p$-values from five major medical journals, they estimated that approximately 14% of significant findings were false discoveries (Jager & Leek, 2014). Ioannidis strongly criticized their analysis and argued that the estimate was too reassuring (Ioannidis, 2014). Gelman and O’Rourke (2014) objected for a different reason: they questioned whether a science-wide false discovery rate based on dividing effects into exact zeros and nonzeros was a scientifically meaningful quantity at all.

Gelman and Carlin (2014) proposed focusing instead on Type S and Type M errors. Type M errors concern exaggeration of effect magnitude. Type S errors are more fundamental: a statistically significant result has the wrong sign—for example, a trial concludes that a treatment is beneficial when its true effect is harmful (Gelman & Carlin, 2014). A related quantity, the false-sign rate, has subsequently been developed in the multiple-testing literature (Stephens, 2017).

Jager and Leek’s estimated null component need not be interpreted literally as a population of effects that are exactly zero. Very small nonzero effects generate nearly uniform $p$-value distributions and are difficult to distinguish from exact nulls in a two-component mixture. If their estimated 14% false-discovery component instead represented effects very close to zero, approximately half of these significant findings would be expected to have the wrong sign. This suggests a false-sign rate of roughly 7% from this component, with only a comparatively small additional contribution expected from the stronger-effect component.

Thus, Jager and Leek’s results do not suggest that most significant medical findings either reject a true nil or point in the wrong direction. Nevertheless, their estimate was challenged on several grounds and had little influence on subsequent debates about the credibility of medical research.

Schimmack and Bartoš (2023) approached the problem differently. Rather than estimating the actual false discovery rate, we estimated an upper bound, which we called the false discovery risk. Sorić (1989) showed that the maximum false discovery rate is determined by the discovery rate—the proportion of all tests that are significant—without requiring an estimate of how many true effects are exactly zero. Using a new sample of clinical trials reported in medical journals, we estimated a false discovery risk of 13%, with a 95% confidence interval from approximately 8% to 21%.

This distinction is important because false-discovery methods do not require investigators to identify which hypotheses are truly null. The broader logic has long been used in large-scale multiple testing, particularly in genomics, where controlling the false discovery rate became an alternative to controlling the probability of any false positive (Benjamini & Hochberg, 1995; Storey, 2003). Z-curve extends this logic to literatures affected by publication selection: it estimates the discovery rate that would be expected without selection and uses this rate to obtain an upper bound on the FDR (Bartoš & Schimmack, 2022). Importantly, this approach does not require the assumption that exact nil effects actually exist. The purpose of estimating the false discovery risk is to examine how credible rejections of the nil hypothesis are.

A year after our study, van Zwet, Gelman, and colleagues analyzed 23,551 randomized clinical trials from Cochrane reviews (van Zwet et al., 2024). Their main concern was effect-size exaggeration, but they also estimated the probability that statistically significant trials had the wrong sign. Their model implied a false-sign rate of only about 2%. This remarkably low rate sits uneasily beside broad claims that statistically significant results from low-powered clinical trials are generally untrustworthy. It suggests that the principal problem identified by their model is not that significant clinical trials usually reach the wrong directional conclusion, but that their estimates of effect magnitude are noisy and selected upward.

Thus, three analyses of medical research appear to produce somewhat different pictures: Jager and Leek estimated an actual FDR of approximately 14%, Schimmack and Bartoš estimated a maximum FDR of approximately 13%, and van Zwet and colleagues estimated a false-sign rate of only about 2%. These quantities are not identical, but they address closely related questions about the credibility of statistically significant clinical-trial results. One important reason for their differences is the assumed distribution of true effects.

The present analysis examines this issue directly. Using the same Cochrane data, I fit several substantially different mixture models and examine which conclusions are robust to the choice of mixture and which depend on interpreting the fitted components as real populations of true effects.

The Credibility of Z-Curve

In a series of posts on Andrew Gelman’s blog, van Zwet criticized z-curve, the statistical method that we used to estimate the false discovery risk. His concerns included bootstrap confidence intervals in some settings and sensitivity of the expected discovery rate to misspecification of z-curve’s default discrete mixture. In particular, he showed examples in which a true noncentrality fell between z-curve’s fixed component locations and the expected discovery rate was biased.

These are legitimate concerns about model uncertainty. I responded by further developing z-curve and releasing zcurve3. One important extension is that zcurve3 no longer requires the traditional discrete mixture. Users can fit mixtures of normal distributions and vary the locations and variances of the components, making it possible to examine directly whether substantive conclusions depend on the particular representation of the latent distribution.

This extension is particularly useful here because van Zwet, Gelman, and colleagues modeled the Cochrane data with a mixture of normal distributions centered at zero. Zcurve3 makes it possible to fit the same Cochrane data with their zero-centered normal mixture, with a more flexible normal mixture in which both means and standard deviations are estimated, and with the traditional discrete z-curve model.

This provides a direct robustness test. If the principal z-curve estimands change substantially across these models, concerns about the discrete-component approximation are justified. If they remain stable despite substantial differences in the estimated mixture components, the estimands are more robust than the latent mixture itself. The same comparison can determine whether estimates of the false-sign rate, which depend directly on the inferred distribution of true effects, show the same robustness.

Reproducibility Code:
https://github.com/UlrichSchimmack/zcurve3_development_functions/blob/main/SecondLook.Cochrane.R

Zero-centered normal mixture

The first model specified three normal components with their means fixed at zero and their standard deviations freely estimated, closely reproducing the model used by van Zwet and colleagues. Zcurve3 fits absolute $z$-values and therefore represents these components as normal distributions truncated at zero. For zero-centered normal distributions, this is simply the folded representation of the same symmetric model and has essentially no substantive impact on the fit.

There are two additional differences. Zcurve3 can explicitly model selection for statistical significance, and confidence intervals were obtained with cluster bootstrap resampling to account for the nesting of individual study results within Cochrane reviews.

The z-curve plot shows that the mixture closely traces the distribution of significant results. It also predicts the nonsignificant distribution well. The similarity between the observed and expected discovery rates indicates little evidence of selection for significance in the Cochrane data. This finding is informative in its own right. Nonsignificant trials are not missing from these meta-analyses; they are considerably more common than significant trials. This does not rule out other sources of effect-size inflation, but strong publication selection against nonsignificant trials does not appear to characterize this dataset.

The estimated false discovery risk is approximately 20%, with the upper end of the 95% confidence interval at about 25%. Thus, even when z-curve is fitted with a latent distribution closely resembling the one preferred by van Zwet and colleagues, no more than approximately one quarter of significant findings could be exact-null false discoveries. This result is broadly consistent with our earlier analysis of $p$-values reported in medical-journal abstracts, despite the very different dataset and mixture specification.

Normal mixture with free means

The second model relaxed the assumption that all component means are fixed at zero. Fixing the means at zero served other purposes in van Zwet and colleagues’ application, including producing a symmetric reference distribution and symmetric shrinkage toward zero. However, the restriction could matter if the latent effect distribution were centered or concentrated away from zero.

In the Cochrane data, relaxing the restriction made little difference. The means of the two dominant components were estimated to be close to zero, and the principal z-curve estimates were virtually unchanged. Thus, the zero-mean restriction happens to be fairly benign for these data.

Default discrete z-curve

The third model fitted the default z-curve specification with seven discrete components. Once again, the principal results changed only slightly.

This illustrates an important property of z-curve. Its principal estimands are functions of the fitted distribution of test statistics rather than interpretations of individual mixture components. Different mixture models can assign very different weights and parameters to their latent components while producing nearly identical fitted densities. If they reproduce the relevant distribution of $z$-values equally well, they can therefore produce very similar estimates of the expected discovery rate, expected replication rate, and false discovery risk.

This result directly addresses one aspect of van Zwet’s criticism. A fixed discrete approximation can be biased in some data-generating scenarios, and this possibility should not be ignored. But zcurve3 makes the concern empirically testable. Researchers can fit alternative mixture specifications as a sensitivity analysis. In the Cochrane data, replacing the traditional discrete mixture with normal mixtures does not materially alter the principal z-curve conclusions.

Robust estimands, unstable components

The picture changes when the individual mixture components themselves are interpreted as latent populations.

In the two continuous normal-mixture models, the probability of an effect being exactly zero is zero by construction. The actual exact-null FDR is therefore zero under these models. This should not be confused with the false discovery risk, which remains around 20%. The latter asks how high the FDR could be without assuming that the chosen continuous latent model is literally true.

The zero-centered and free-mean normal mixtures imply false-sign rates of approximately 3.2%, reasonably close to van Zwet et al.’s reported estimate of about 2%.

The default discrete z-curve gives a different latent interpretation. About 4.5% of significant results are assigned to the component with a noncentrality parameter of zero. If this component is interpreted literally, the estimated actual FDR is therefore about 4.5%. If, instead, the zero component is regarded as a discrete approximation to a continuous collection of very small positive and negative effects, exact-zero FDR disappears and approximately half of the significant results in this component become sign errors. Under this interpretation, the false-sign rate is approximately 2.7%.

Thus, these relatively flexible models all imply a false-sign rate below about 5%. However, this apparent agreement should not be mistaken for identification of the latent distribution.

To illustrate the problem, I fitted another discrete mixture with components at noncentralities $0,2,4,$ and $6$. Removing the component at $z=1$ prevents the model from representing weak positive effects explicitly. Many of the low-power studies must therefore be assigned to the zero component instead.

The principal z-curve estimands changed only slightly. The latent interpretation changed dramatically. After accounting for significant observations above $z=6$, the zero component implies an actual FDR of approximately 23%, close to the maximum FDR permitted by the discovery rate. If the zero component is instead interpreted as a symmetric collection of effects extremely close to zero, approximately half of these findings have the wrong sign, producing an estimated false-sign rate of about 11.5%.

Nothing about the observed Cochrane data changed. Only the latent mixture specification changed.

This example illustrates why fitted mixture components should not automatically be interpreted as literal data-generating populations. Their main statistical advantage is precisely their flexibility: different mixtures can approximate the same observed density. Quantities that depend mainly on the fitted density can therefore be robust even when the decomposition into latent components is not. In contrast, quantities that require a literal interpretation of the components—including estimates of the actual proportion of exact zeros, false-sign rates, and some shrinkage quantities—can be substantially more model dependent.

This is not unique to z-curve. Stephens’s (2017) empirical-Bayes approach to false-sign rates, for example, obtains greater stability by imposing a substantive shape constraint: the latent effect distribution is assumed to be unimodal with its mode at zero. Such assumptions may be reasonable, but estimates obtained from them are conditional on those assumptions rather than determined by the observed data alone.

Conclusion

Mixture models remain relatively uncommon in meta-analysis and meta-science, and their use can invite a basic misunderstanding. The most serious mistake is to interpret the fitted components as if they were empirically identified populations of studies or true effects. Component locations, variances, and weights can be highly sensitive to model specification. Different mixtures can fit essentially the same observed distribution while implying substantially different latent decompositions. Good model fit alone therefore cannot establish that one particular decomposition is the true data-generating process.

Z-curve was designed to avoid relying on this interpretation. Its principal estimands—the expected discovery rate and expected replication rate—are global properties of the fitted distribution. The false discovery risk is subsequently obtained from the estimated discovery rate using Sorić’s bound. Consequently, substantially different mixture specifications can produce similar answers as long as they reproduce the relevant features of the observed distribution.

This distinction also clarifies the role of the nil hypothesis. It is not necessary to assume that some fixed proportion of scientific effects are literally zero in order to use false-discovery risk as a credibility criterion. If exact-zero effects do not exist, the actual exact-null FDR is zero. The Sorić bound remains useful because it asks the more conservative question: given the discovery rate, how large could the false discovery rate be? This is consistent with the broader logic of false-discovery methods already established in large-scale multiple testing.

The same point helps clarify discussions of “low power.” Van Zwet and colleagues define the signal-to-noise ratio as the true effect divided by its standard error and translate it into conventional power to reject $H_0:\theta=0$. This is mathematically legitimate, but if exact-zero effects are assumed never to occur, low power against zero cannot itself imply that significant findings are false. It primarily signals that effects are small relative to their sampling error, which creates imprecise estimates, magnitude exaggeration after selection, and some risk of sign errors. How much of this makes a study scientifically untrustworthy requires an explicit criterion.

The present Cochrane analysis provides one such criterion. Across substantially different mixture specifications, the estimated discovery and replication rates and the maximum false discovery rate are remarkably stable. In contrast, actual FDR and false-sign estimates can change substantially when the latent components are interpreted literally.

The lesson is therefore not that one mixture model is correct and another is wrong. Conclusions should be trusted to the extent that they survive reasonable changes in mixture specification. In the Cochrane data, the principal z-curve estimands pass this test. Literal interpretations of the latent mixture components do not.

References

Bartoš F, Schimmack U. Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology. 2022;6:2021.2720. doi:10.15626/MP.2021.2720.

Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Series B. 1995;57:289–300. doi:10.1111/j.2517-6161.1995.tb02031.x.

Gelman A, Carlin JB. Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspect Psychol Sci. 2014;9:641–651. doi:10.1177/1745691614551642.

Gelman A, O’Rourke K. Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics. 2014;15:18–23. doi:10.1093/biostatistics/kxt034.

Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124. doi:10.1371/journal.pmed.0020124.

Ioannidis JPA. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics. 2014;15:28–36. doi:10.1093/biostatistics/kxt036.

Jager LR, Leek JT. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics. 2014;15:1–12. doi:10.1093/biostatistics/kxt007.

Schimmack U, Bartoš F. Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values. PLoS ONE. 2023;18:e0290084. doi:10.1371/journal.pone.0290084.

Sorić B. Statistical “discoveries” and effect-size estimation. J Am Stat Assoc. 1989;84:608–610. doi:10.2307/2289950.

Stephens M. False discovery rates: A new deal. Biostatistics. 2017;18:275–294. doi:10.1093/biostatistics/kxw041.

Storey JD. The positive false discovery rate: A Bayesian interpretation and the q-value. Ann Stat. 2003;31:2013–2035. doi:10.1214/aos/1074290335.