Zajenkowski, M., van der Linden, D., & Rogoza, R. (2022). Self-assessed intelligence, objective intelligence and the higher-order structure of personality. Personality and Individual Differences, 190, 111553. https://doi.org/10.1016/j.paid.2022.111553
What happens when an AI can read a published article, identify a methodological problem, and independently reanalyze the data?
From Chess Computers to Scientific Research
When the first chess computers appeared, they were not particularly good. They could beat beginners, but experienced players had little reason to fear them. Over time, however, chess programs improved. Eventually, they became competitive with grandmasters. Today, even the world’s best human chess players cannot consistently compete with the strongest chess engines.
Chess computers did not merely become better opponents. They also changed how humans play chess. Professional players now use computer analysis to evaluate positions, identify mistakes, and prepare strategies. Even the best chess players can learn from their computers.
A similar transformation is underway in scientific research.
Artificial intelligence is already contributing to research in fields such as molecular biology, chemistry, physics, and mathematics. AI systems have predicted the structures of hundreds of millions of proteins, identified promising new materials, and contributed to mathematical discoveries.
These are technically demanding fields in which scientific progress requires sophisticated theories, complex methods, and specialized knowledge.
Psychology should not expect to be immune to this transformation.
Indeed, much psychological research involves comparatively simple statistical analyses of correlations among questionnaire responses. If AI can assist with problems in molecular biology and mathematics, it should hardly be surprising that it can also evaluate a personality psychology article.
I recently put this possibility to the test.
An AI Reviews a Published Personality Psychology Article
I have been critically examining research on the General Factor of Personality (GFP), a proposed broad personality disposition that supposedly explains why socially desirable characteristics tend to be positively correlated.
The GFP is typically extracted from correlations among Big Five personality measures. Researchers who believe that it is a substantive personality factor interpret it as a general disposition toward social effectiveness or adaptability.
An alternative explanation is that the correlations partly reflect halo: people who evaluate themselves or others favorably on one characteristic tend to provide favorable ratings on other characteristics.
These explanations are fundamentally different. The first proposes an actual personality disposition. The second concerns the measurement process.
One way to distinguish them is to examine relationships between questionnaire-based personality ratings and characteristics measured independently of those ratings.
I therefore asked ChatGPT to evaluate an article by Zajenkowski, van der Linden, and Rogoza (2022), titled Self-assessed intelligence, objective intelligence and the higher-order structure of personality, published in Personality and Individual Differences.
The article was particularly interesting because the authors measured intelligence in two ways. Participants rated their own intelligence, but they also completed objective cognitive ability tests.
This provides an opportunity to examine whether the GFP is associated with actual intelligence or merely with people’s beliefs about their intelligence.
The article had undergone the usual scientific publication process. Researchers conducted the studies, wrote the manuscript, submitted it to a scientific journal, received peer review, and obtained editorial acceptance.
I wanted to know whether AI could identify problems that this process had apparently failed to resolve.
The First Step: Reading the Article Critically
I uploaded the published article to ChatGPT and asked whether it provided evidence for the GFP.
ChatGPT examined the study’s hypotheses, measurements, methods, results, and statistical tables.
It quickly identified an important pattern.
Across two studies, the GFP was positively associated with self-assessed intelligence, but its correlations with objectively measured intelligence were much weaker.
| Study 1 | Study 2 | |
|---|---|---|
| GFP and self-assessed intelligence | .29 | .35 |
| GFP and objective intelligence | .15 | .05 |
People who described their personalities favorably also tended to describe themselves as intelligent. However, their actual performance on intelligence tests showed little relationship with the GFP.
This pattern is consistent with a general positive self-evaluation factor.
But ChatGPT also noticed something more important.
Objective intelligence was primarily associated with one specific Big Five domain: Openness/Intellect.
The correlations were .34 in Study 1 and .23 in Study 2. Correlations with the other four domains were substantially smaller, and some were negative.
This raises an obvious question.
Does objective intelligence have anything to do with a general personality factor, or is its association with the GFP merely a consequence of including Openness/Intellect in the GFP composite?
The published analyses did not answer this question.
The Second Step: Identifying the Statistical Problem
The authors constructed their higher-order personality factors through a series of factor analyses.
First, they extracted Plasticity from Extraversion and Openness/Intellect. They extracted Stability from Emotional Stability, Agreeableness, and Conscientiousness.
They then conducted another factor analysis to extract a GFP from Plasticity and Stability.
This is a questionable way to test a hierarchical theory.
A one-factor solution involving only two variables does not provide a meaningful test of whether a general latent personality factor explains the observed correlations. The general factor is essentially imposed rather than evaluated against alternative models.
More importantly, the procedure does not distinguish variance that is common to different personality traits from variance that is specific to individual traits.
Imagine that intelligence is positively correlated with Openness/Intellect but unrelated to the common variance among the Big Five.
Because Openness/Intellect contributes to the GFP score, the GFP may still correlate with intelligence.
Researchers could then report a relationship between intelligence and the GFP even though the relationship is entirely attributable to a specific personality characteristic.
This is precisely the distinction that a hierarchical structural equation model can examine.
The obvious next step was to fit such a model.
The Third Step: AI Conducts Its Own Statistical Analysis
Here is where the experience became particularly interesting.
I pointed out that the article published the complete correlation matrices for both studies. Consequently, we did not need access to the individual participants’ responses. We could fit covariance-structure models directly to the published correlations.
I asked ChatGPT to do so.
It wrote and executed Python code that reconstructed the correlation matrices, specified alternative latent-variable models, estimated their parameters using maximum likelihood, and calculated model-fit statistics.
It then compared three explanations:
- General-factor model: Objective intelligence is associated with a common factor underlying the Big Five.
- Specific-trait model: Intelligence is associated with variance specific to Openness/Intellect rather than with the general factor.
- Combined model: Intelligence is allowed to relate to both the general factor and the specific Openness/Intellect component.
The comparison is straightforward. If a substantive GFP explains the association with intelligence, the general-factor relationship should be necessary to account for the data. If intelligence is related specifically to Openness/Intellect, the specific-trait model should account for the association without requiring a relationship with the GFP.
ChatGPT fitted these models to the first study.
The results were striking.
The Results: The Alternative Model Fits Better
The first model, in which intelligence was associated only with the general personality factor, fit poorly:
χ²(9) = 38.44, RMSEA = .119.
The alternative model, allowing intelligence to relate specifically to Openness/Intellect, fit substantially better:
χ²(9) = 13.63, RMSEA = .047.
The difference becomes even clearer when the competing paths are tested within the combined model.
Adding the Openness-specific relationship to the general-factor model produced a substantial improvement:
Δχ²(1) = 25.04, p < .000001.
In contrast, adding a GFP–intelligence relationship to the specific-trait model produced virtually no improvement:
Δχ²(1) = 0.24, p = .627.
In other words, the specific association with Openness/Intellect was needed to explain the results. An additional relationship with the general factor was not.
This is exactly the kind of theoretically informative analysis that the original article could have conducted.
The second study presented an additional problem.
A single general-factor model of the five personality domains failed to fit the correlations adequately:
χ²(5) = 30.20, RMSEA = .146.
The fitted solution also approached a boundary, making the general-versus-specific decomposition unreliable in that sample.
Nevertheless, a conventional multiple regression using all five personality domains again showed that objective intelligence was primarily associated with Openness/Intellect. Controlling for the other four domains, its standardized regression coefficient was approximately .24.
The results therefore did not provide a convincing replication of a general-factor explanation for intelligence.
What Did AI Actually Demonstrate?
There are two important qualifications.
First, the analysis does not prove that the GFP is entirely a halo artifact. Establishing that conclusion would require a measurement model capable of separating evaluative bias from substantive personality variance, ideally using multiple methods and informants.
Second, the models were fitted to Big Five domain scores rather than individual questionnaire items. The analysis therefore tests a simplified hierarchical structure rather than a complete measurement model.
Nevertheless, the reanalysis establishes something important.
The original article’s small correlations between objective intelligence and the GFP do not require a substantive GFP–intelligence relationship. In Study 1, a model involving specific Openness/Intellect variance fits much better. In Study 2, the proposed general-factor measurement structure itself is not adequately supported.
The broader pattern is consistent with an evaluative interpretation: favorable self-assessments of personality correlate strongly with favorable self-assessments of intelligence, whereas objective intelligence is primarily related to a specific personality domain.
That is a more informative interpretation than simply reporting correlations between constructed GFP scores and intelligence.
Why Didn’t Peer Review Catch This?
The most interesting question is not whether ChatGPT could perform these analyses.
It could.
The interesting question is why a published article did not already contain them.
The authors had access to the complete data. They were familiar with the GFP literature and its competing interpretations. They had the statistical tools needed to fit hierarchical models.
The reviewers and editor also had an opportunity to evaluate whether the analyses adequately tested the theoretical claims.
Yet the manuscript was published without this critical comparison.
We cannot know what the reviewers noticed, requested, or discussed. Peer-review reports are generally not publicly available. But we can evaluate the published product.
And the published product leaves an important theoretical question unanswered even though the necessary data were available.
This illustrates a broader limitation of human peer review.
Reviewers are often selected because they are knowledgeable about a particular literature. They may share the conceptual assumptions and methodological conventions of the researchers whose work they evaluate.
Within an established research paradigm, reviewers may focus on whether a study follows accepted procedures rather than whether those procedures actually distinguish competing explanations.
An article can therefore receive favorable reviews while making little progress toward resolving the theoretical questions that supposedly motivate the research.
Publication is evidence that a manuscript passed a particular editorial process. It is not evidence that its statistical analyses were the most informative analyses available.
AI Does Not Have to Respect Established Paradigms
An AI assistant has a potentially important advantage.
It does not have to build an academic career within the GFP paradigm. It does not need publications, grants, citations, or approval from established researchers in the field.
Of course, AI can reproduce prevailing scientific assumptions because it has learned from the existing literature. It can also make mistakes, including statistical mistakes.
But it can be asked to examine competing explanations directly, without regard to whether a particular interpretation is fashionable or supported by influential researchers.
In this case, the AI was initially asked to evaluate a published article. It identified a measurement problem, examined the correlation tables, and implemented a more informative statistical analysis.
This is not the same as replacing the entire scientific process with an autonomous machine. I supplied the scientific question and prompted the alternative analysis.
Nevertheless, the AI carried out substantial technical work independently, including writing and executing the statistical analysis and evaluating the resulting model comparisons.
Only a few years ago, many psychologists would have regarded AI primarily as a tool for improving writing, summarizing literature, or answering factual questions.
The ability to move from a published article to a new statistical analysis is a much more consequential development.
Is AI Already a Better Reviewer?
Chess provides a useful analogy, but there is an important difference.
In chess, there are unambiguous rules and an objective criterion for success. A computer can evaluate positions and test moves against clear outcomes.
Scientific research is more complicated. Researchers must decide which questions matter, which constructs are meaningful, whether measurements are valid, and what conclusions follow from imperfect evidence.
No statistical model can resolve all these questions automatically.
However, many components of scientific research are sufficiently formalized to permit computational evaluation.
An AI can examine whether a statistical model is identified, whether its restrictions are theoretically justified, whether alternative models fit better, and whether the conclusions follow from the reported results.
It can also reconstruct analyses from published summary statistics, provided the necessary information is available.
These are precisely the kinds of technical issues that peer reviewers are expected to evaluate.
The chess analogy therefore suggests a plausible future in which AI is not merely a convenient research assistant but an increasingly capable methodological critic.
Human researchers may continue to provide important theoretical insights and scientific judgment. But they should not assume that they will remain superior at statistical analysis or at detecting weaknesses in published research.
The present example demonstrates that AI can already perform some of these tasks remarkably well.
The Future of Psychological Science
The implications extend well beyond the GFP debate.
Thousands of psychological articles are published every year. Many rely on standard statistical procedures, including correlations, regressions, factor analyses, and structural equation models.
Published articles often contain enough statistical information to evaluate alternative explanations without collecting new data.
AI makes such evaluations dramatically more accessible.
Researchers no longer need to spend hours manually reconstructing covariance matrices, programming models, and calculating fit statistics simply to investigate whether a published conclusion survives a reasonable alternative analysis.
More ambitious projects could systematically examine entire research literatures, identifying articles that rely on poorly fitting models, questionable measurement assumptions, or interpretations that are not uniquely supported by the evidence.
This could change the incentives of psychological research.
At present, publication remains the primary currency of academic success. Researchers are rewarded for producing publishable findings, and journals generally rely on a small number of human reviewers to evaluate manuscripts.
But if independent AI analyses routinely identify serious weaknesses in published research, the distinction between publishable research and scientifically convincing research may become increasingly visible.
In that environment, journals could improve quality control by requiring computational checks and systematic comparisons with plausible alternative models.
AI would not eliminate the need for human judgment. It could, however, make it much harder for weak statistical arguments to survive merely because they conform to established research practices.
Conclusion
The example discussed here is not a demonstration that AI is infallible or that human scientists are obsolete.
It demonstrates something more concrete.
A published article used correlations between intelligence and a constructed General Factor of Personality to investigate the higher-order structure of personality.
An AI assistant examined the article, identified a missing theoretical comparison, reconstructed the published correlation matrices, and fitted alternative structural equation models.
The alternative model accounted for the findings better than the general-factor explanation.
The original authors, reviewers, and editor did not provide this comparison in the published article.
That is a significant result, not because it settles the GFP debate, but because it illustrates the changing relationship between human expertise and artificial intelligence.
The question is no longer whether AI can help psychologists write their articles. It is whether AI can evaluate psychological research more critically than the humans who write, review, and publish it.
This case suggests that, for at least some important methodological tasks, the answer is already yes.
References
Zajenkowski, M., van der Linden, D., & Rogoza, R. (2022). Self-assessed intelligence, objective intelligence and the higher-order structure of personality. Personality and Individual Differences, 190, 111553. https://doi.org/10.1016/j.paid.2022.111553
Wang, H., et al. (2023). Scientific discovery in the age of artificial intelligence. Nature, 620, 47–60. https://doi.org/10.1038/s41586-023-06221-2
Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. https://doi.org/10.1038/s41586-021-03819-2
Silver, D., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362, 1140–1144. https://doi.org/10.1126/science.aar6404
Reanalysis note
The analyses described above were generated using ChatGPT from the correlation matrices published in Tables 1 and 2 of Zajenkowski et al. (2022). Models were estimated using Python and maximum likelihood. Correlations were published to two decimal places, so reported fit statistics are approximate. The code and reconstructed matrices should be independently checked before treating the findings as a definitive published reanalysis.