Tag Archives: Personality Structure

Chatting with ChatGPT: Correlated Residuals in SEM

ChatGPT, confirmatory factor analysis, Correlated Residuals, Questionable Measurement Practices, SEM, , , , ,

Highlight: ChatGPT agreed that McCrae et al. used Questionable Measurement Practices to hide problems with their structural model of personality. Using PCA rather than CFA is a questionable measurement practice because PCA hides correlated residuals in the data.

Abstract

Key Points on Correlated Residuals in SEM

  1. Distinction Between Correlated Constructs vs. Correlated Residuals
    • Correlated constructs (latent variables): Expected if a higher-order factor explains their relationship.
    • Correlated residuals: Indicate shared variance that is not explained by the model and require theoretical justification.
  2. Correlated Residuals in Measurement vs. Structural Models
    • Measurement models (CFA): Residual correlations suggest method effects, poor measurement design, or missing latent factors.
    • Structural models: Correlated residuals indicate theory misspecification, requiring better explanations (e.g., an unmeasured causal link or omitted variable).
  3. Why Hiding Correlated Residuals is Problematic
    • Creates an illusion of a “clean” model that misrepresents the data.
    • Leads to biased parameter estimates and misinterpretation of relationships.
    • Prevents theoretical progress by ignoring unexplained variance.
  4. Tactics to Hide Correlated Residuals (Bad Science)
    • Using PCA instead of CFA to avoid residual correlations entirely.
    • Aggregating variables (e.g., item sum scores) to smooth over unexplained variance.
    • Introducing unnecessary latent factors to absorb residual variance artificially.
    • Adjusting error variances or allowing cross-loadings to “soak up” residual relationships.
    • Cherry-picking fit indices to claim model adequacy despite misfit.
  5. A Better Approach: Transparency & Theory Improvement
    • Report correlated residuals honestly and explore potential theoretical explanations.
    • Use alternative model specifications to examine different structures.
    • Refine the measurement model (improving item design, removing method effects).
    • Expand the theoretical model to include missing latent factors or direct causal links.
  6. The Case of the Big Five & McCrae and Costa
    • Their CFA model showed correlated residuals between some facets, meaning the Big Five did not fully explain trait relationships.
    • Instead of modifying the model, they rejected CFA entirely in favor of PCA, which does not allow residual correlations or test model fit.
    • This approach avoided the issue rather than solving it, highlighting the danger of prioritizing theory preservation over theory improvement.

Final Takeaway

✔ Residual correlations are clues, not problems. They signal missing theory, measurement issues, or model limitations.
✔ A “clean” model isn’t always the right model. The goal is theoretical accuracy, not just statistical fit.
✔ Transparency > Simplification. Rather than hiding residual correlations, researchers should use them to refine measurement and advance theory.

Full Discussion

Q: What are correlated residuals or correlated disturbances in structural equation models?

In Structural Equation Modeling (SEM), correlated residuals (also called correlated disturbances) refer to situations where the residual terms (errors) of two or more observed or latent variables are not independent but instead share some degree of correlation.

What Are Residuals in SEM?

Residuals in SEM represent the unexplained variance in a variable after accounting for the effects of predictors in the model. They can arise in:

  • Observed variables (measurement model residuals)
  • Latent variables (structural model disturbances)

Why Do Residuals Get Correlated?

Residuals should ideally be uncorrelated if the model is correctly specified. However, correlations can occur due to:

  1. Omitted Variables: A common cause—when a relevant factor influencing both variables is missing from the model.
  2. Measurement Errors: If two observed variables share methodological similarities, such as being measured on the same scale or instrument, their errors may correlate.
  3. Shared Method Variance: When the same respondent answers multiple survey questions, systematic biases (e.g., social desirability bias) can lead to correlated residuals.
  4. Misspecified Relationships: If two variables have a direct causal relationship that isn’t modeled, their residuals may pick up the unaccounted association.
  5. Time-Series or Cluster Effects: In longitudinal or multilevel models, residuals may be correlated due to dependencies across time points or groups.

How Are Correlated Residuals Represented in SEM?

  • In path diagrams, they are usually represented as double-headed arrows connecting the error terms of two variables.
  • Mathematically, they are expressed as covariances between error terms.

Implications of Correlated Residuals

  • Model Fit: Allowing for correlated residuals can improve model fit, but arbitrarily adding them can lead to overfitting.
  • Interpretation Issues: If residuals are correlated due to a missing variable, the structural relationships might be biased or misleading.
  • Model Modification: SEM software (e.g., AMOS, Mplus, or Lavaan in R) often suggests adding correlated residuals to improve fit, but this should be done theoretically rather than based solely on fit indices.

Best Practices

  • Theoretical Justification: Only include correlated residuals when there is a theoretical or substantive reason.
  • Alternative Model Testing: Check if the correlation disappears when additional predictors or paths are added.
  • Residual Diagnostics: Examine modification indices and residual correlation matrices for patterns.

Would you like me to demonstrate how to test for correlated residuals in R or another software?

Yes, can you provide an example of testing for correlated residuals?

I’ll walk you through an example using R and the lavaan package to test for correlated residuals in a Structural Equation Model (SEM).

Example: Testing for Correlated Residuals in R (lavaan)

We’ll analyze a simple measurement model with four observed variables measuring a latent construct (e.g., “Well-being”). We will:

  1. Fit an initial model assuming uncorrelated residuals.
  2. Examine the modification indices to detect potential correlated residuals.
  3. Modify the model by adding correlated residuals if justified.

Step 1: Load Required Libraries

library(lavaan)

library(semPlot) # For visualizing the SEM model

Step 2: Simulate Data

We create four observed variables (Y1, Y2, Y3, Y4) that measure a latent construct (WellBeing).

set.seed(123)

# Simulate data for a latent construct with some correlation between residuals

n <- 200

WellBeing <- rnorm(n, mean = 50, sd = 10)

Y1 <- WellBeing + rnorm(n, 0, 5)

Y2 <- WellBeing + rnorm(n, 0, 5)

Y3 <- WellBeing + rnorm(n, 0, 5)

Y4 <- WellBeing + rnorm(n, 0, 5)

# Introduce correlated residuals between Y2 and Y3

Y2 <- Y2 + rnorm(n, 0, 2)

Y3 <- Y3 + 0.3 * Y2 + rnorm(n, 0, 4)

# Combine into a data frame

data <- data.frame(Y1, Y2, Y3, Y4)

Step 3: Define and Fit an Initial CFA Model

We assume that all indicators load onto a single latent factor (WellBeing), without any correlated residuals.

# Define a simple CFA model

model1 <- ‘

  WellBeing =~ Y1 + Y2 + Y3 + Y4

‘

# Fit the model

fit1 <- cfa(model1, data = data)

# Show summary with fit indices

summary(fit1, fit.measures = TRUE, standardized = TRUE)

Step 4: Examine Modification Indices

We check the modification indices to see if the model suggests adding correlated residuals.

# Show modification indices above a threshold (e.g., >10)

modindices(fit1, sort = TRUE)

If the output suggests a high modification index (MI) for Y2 ~~ Y3, it means that correlating their residuals could significantly improve model fit.

Step 5: Modify the Model by Adding a Correlated Residual

We now modify the model to allow Y2 and Y3 to have correlated residuals.

# Modify model by adding a correlated residual

model2 <- ‘

  WellBeing =~ Y1 + Y2 + Y3 + Y4

  Y2 ~~ Y3  # Correlated residual

‘

# Fit the new model

fit2 <- cfa(model2, data = data)

# Show summary with fit indices

summary(fit2, fit.measures = TRUE, standardized = TRUE)

Step 6: Compare Model Fit

To check if the new model is better, compare the Chi-square values, CFI, and RMSEA:

anova(fit1, fit2)  # Model comparison

  • If the new model has significantly better fit (lower Chi-square, higher CFI, lower RMSEA), then adding the correlated residual is justified.
  • If not, it suggests that the initial model was already sufficient.

Step 7: Visualize the SEM Model

We can visualize the modified SEM model, showing the correlated residual.

semPaths(fit2, whatLabels = “std”, layout = “tree”, residuals = TRUE,

         edge.label.cex = 1.2, sizeMan = 6, sizeLat = 8)

Interpreting Results

  1. If model fit improves and modification indices strongly suggested the correlation, we may keep the correlated residual.
  2. If model fit does not improve, the correlation may not be necessary or may indicate a misspecification elsewhere.
  3. Theory should always justify correlated residuals—avoid adding them solely to improve fit!

Would you like me to adjust this example for another software (e.g., Mplus, Stata, or Python)?

Q: Ok. I get it, but for most people who do not use SEM this is not helpful. Let’s get back to your definition of residuals. You make an important distinction between observed variables and latent variables. What is the theoretical difference between correlated residuals for observed variables and for latent variables?

Great question! The distinction between correlated residuals for observed variables and correlated residuals for latent variables is critical in Structural Equation Modeling (SEM) because it reflects different theoretical and methodological concerns.

1. Correlated Residuals for Observed Variables

Correlated residuals at the observed variable level occur when the error terms of two measured indicators (i.e., directly observed variables) are correlated. This typically happens in measurement models (such as Confirmatory Factor Analysis, CFA).

Theoretical Interpretation

  • Correlated residuals between observed variables suggest that those indicators share something beyond their common latent factor.
  • This often implies method effects or measurement bias, meaning the two variables may be related due to:
    • Similar wording in survey questions (e.g., two items measuring anxiety both containing the phrase “nervous feelings”).
    • Common method variance (e.g., all responses were given in the same testing session).
    • A missing latent construct influencing both items.
    • Multicollinearity or redundancy in measurement.

Example

Suppose we have a latent factor “Job Satisfaction” measured by four observed items:

  • Y1: “I enjoy my job.”
  • Y2: “I feel motivated at work.”
  • Y3: “My job makes me happy.”
  • Y4: “I feel fulfilled by my work.”

If we find that the residuals of Y1 and Y3 are correlated, it might be because they both use emotionally charged language (e.g., “enjoy” and “happy”), suggesting a common method bias rather than a true substantive link.

Implication in SEM:
Correlated residuals at the observed level generally indicate a misspecified measurement model and may suggest poor construct validity.

2. Correlated Residuals for Latent Variables

Correlated residuals at the latent variable level occur when the disturbances (errors) of two unobserved constructs (latent variables) are correlated. This typically happens in structural models (where relationships between latent variables are tested).

Theoretical Interpretation

  • Correlated residuals between latent variables suggest that there is unexplained shared variance between two constructs.
  • This could indicate:
    • A missing causal path: If two latent constructs have correlated disturbances, it may mean that one should be predicting the other rather than just correlating.
    • A shared omitted variable: Both latent factors may be influenced by an unmeasured common cause.
    • Specification error: The model may be oversimplified, failing to capture important interactions.

Example

Suppose we have two latent constructs:

  • “Work Engagement” (measured by vigor, dedication, and absorption)
  • “Job Performance” (measured by supervisor ratings, self-evaluations, and task completion)

If the residuals (disturbances) of these two latent variables are correlated, it could mean:

  1. There is an unmodeled common cause (e.g., “Organizational Culture” affects both Work Engagement and Job Performance, but it’s missing from the model).
  2. A causal path is missing (e.g., “Work Engagement” should be directly predicting “Job Performance”).
  3. Conceptual overlap (e.g., Work Engagement and Job Performance are not fully distinct constructs).

Implication in SEM:
Correlated residuals at the latent level usually indicate a misspecified structural model and may signal a missing theoretical link.

Key Differences: Observed vs. Latent Correlated Residuals

AspectObserved VariablesLatent Variables
Location in ModelMeasurement ModelStructural Model
What is correlated?Measurement errors (e.g., Y1, Y2)Disturbances of latent constructs (e.g., Work Engagement, Job Performance)
Possible CausesMethod effects, common measurement bias, omitted shared indicator effectsMissing causal paths, unmodeled common cause, conceptual overlap
ImplicationPotential measurement bias, poor validityPossible model misspecification, theoretical gaps
SolutionImprove measurement items, control for common method variance, refine latent constructsAdd missing paths, introduce omitted variables, reconsider theoretical framework

Final Thought

  • Correlated residuals at the observed level often point to measurement problems (e.g., bad survey items, method bias).
  • Correlated residuals at the latent level suggest model misspecifications (e.g., missing theoretical links, omitted variables).

Both require careful theoretical justification before modifying a model. Simply adding correlated residuals without a theoretical rationale can distort the meaning of the SEM results.

Would you like a real-world application of this, such as an example in psychology or social sciences?

Q: I like your answer. it shows the importance of distinguishing between measurement models and structural models. Measurement models relate variation in instruments created by a researcher to measure a theoretical construct. Structural models relate theoretical constructs to each other. We can eliminate correlated residuals in measurement models by dropping bad instruments or creating better ones. This is under the control of researchers. In contrast, structural models aim to represent relationships among theoretical constructs. If two constructs are related more than a theory predicts, we cannot just omit the residual correlation between the two constructs. We have to modify the theory to explain the relationship. While we seem to agree, it seems that many researchers confuse correlated residuals in measurement models with correlated residuals in structural models and try to build simplistic theories that avoid correlated residuals.

You’re absolutely right! Many researchers fail to distinguish between correlated residuals in measurement models and correlated residuals in structural models, leading to two common problems:

  1. Over-reliance on Statistical Fixes in Measurement Models
    • Instead of refining their instruments or improving construct validity, researchers sometimes “patch up” poor measurement models by adding correlated residuals arbitrarily.
    • This leads to measurement errors being absorbed into the model rather than being addressed at the instrument level.
  2. Over-simplification in Structural Models
    • Some researchers try to eliminate correlated residuals in structural models simply by dropping them, ignoring them, or fitting the cleanest model possible.
    • This results in theoretical under-specification—failing to acknowledge missing constructs, causal feedback loops, or complex relationships between constructs.

Why Does This Confusion Happen?

  1. Software Makes It Too Easy
    • Many SEM tools (AMOS, Mplus, lavaan) automatically suggest correlated residuals to improve model fit.
    • Researchers, especially those without a strong theoretical grounding in SEM, often accept these suggestions without questioning whether they belong in a measurement model or a structural model.
  2. Fit Indices Drive Decision-Making
    • Researchers often chase good fit indices (e.g., CFI > 0.95, RMSEA < 0.05) without critically assessing whether their model is theoretically sound.
    • They may add or remove correlated residuals just to achieve a desirable model fit, rather than questioning why the correlations exist.
  3. Misunderstanding the Nature of Theoretical Constructs
    • Some researchers treat latent variables as objective truths, rather than theoretical approximations of reality.
    • This leads them to assume that a “clean” structural model should have no correlated residuals, even when real-world phenomena are more complex than their theory predicts.

How to Address This Issue

For Measurement Models (Fixing Instruments)

✔ Assess Item Quality – Check for poor items and revise them rather than relying on statistical fixes.
✔ Address Method Bias – If items share method variance, consider using techniques like marker variables or multi-trait multi-method (MTMM) analysis.
✔ Examine Conceptual Redundancy – Ensure that items aren’t redundant or measuring overlapping constructs.

For Structural Models (Improving Theories)

✔ Accept That Some Residuals Should Be Correlated – If two constructs are more related than the theory predicts, the solution is not to eliminate the correlation but to explain it.
✔ Re-evaluate the Theory – Instead of treating correlated residuals as “errors,” consider what real-world mechanisms might be driving them.
✔ Consider Unmeasured Constructs – If two latent variables have correlated residuals, it may signal an omitted third variable influencing both.

Final Thought: Avoiding the “Fit-First, Theory-Later” Trap

Many researchers take an “adjust first, explain later” approach—modifying their models until fit indices look good, then retroactively justifying the changes. This is a dangerous practice because it prioritizes statistical outcomes over theoretical clarity.

A better approach is a “theory-first, model-second” mindset:

  • Start by carefully defining constructs and their relationships.
  • Build a measurement model that minimizes correlated residuals through better instrument design.
  • Accept that some correlated residuals in the structural model are meaningful and require theoretical explanation, not statistical elimination.

In short: Don’t model your way out of a bad theory—improve the theory instead.

Your insights are sharp, and I really appreciate this discussion. Would you like to explore examples of historical cases where theory had to evolve because of unexpected correlations (e.g., psychology, economics, or social sciences)?

Yes, I would like to see some examples. Let’s start with psychology.

Historical Examples in Psychology Where Theory Had to Evolve Due to Unexpected Correlations

Psychology has several landmark cases where researchers found unexpected correlated residuals—relationships between variables that their theories didn’t predict. Instead of ignoring or eliminating these correlations, these anomalies forced theoretical advancements. Here are some classic examples:

1. Intelligence and Working Memory: The Evolution of Fluid Intelligence Theory

Original Assumption:

  • Early intelligence theories (e.g., Spearman’s g-factor) suggested that intelligence (IQ) was a singular construct.
  • Working memory was initially thought to be separate from intelligence.

Unexpected Correlation:

  • Research using latent variable models found stronger-than-expected correlations between fluid intelligence (Gf) and working memory capacity (WMC).
  • If intelligence and working memory were distinct, why were their residuals consistently correlated?

Theoretical Shift:

  • Psychologists like Randall Engle and John Duncan argued that fluid intelligence and working memory share executive attention mechanisms.
  • Instead of treating them as separate constructs, researchers developed a new working memory-based model of intelligence.

Lesson for SEM:

  • If a theory says two constructs are separate but their residuals are highly correlated, the solution isn’t to “fix” the correlation statistically—it’s to revise the theory to account for the overlap.

2. Depression and Anxiety: The Rise of the Negative Affect Model

Original Assumption:

  • Depression and anxiety were considered distinct mental disorders (DSM-III).
  • They were expected to have low residual correlation in measurement models.

Unexpected Correlation:

  • SEM studies showed that residuals of depression and anxiety measures were highly correlated across multiple studies.
  • Even after accounting for shared environmental risk factors, genetic influences, and life stressors, their residual correlation remained too high to ignore.

Theoretical Shift:

  • David Watson and Lee Anna Clark proposed the Tripartite Model of Emotion:
    • Negative Affect (NA) (e.g., sadness, fear) is shared between anxiety and depression.
    • Low Positive Affect (PA) is more specific to depression.
    • Physiological Hyperarousal is more specific to anxiety.
  • Instead of forcing the residuals to be independent, they introduced a latent factor (NA) to explain their shared variance.

Lesson for SEM:

  • When two constructs show high residual correlation, consider whether a higher-order latent factor could explain their shared variance.

3. Personality Traits: The Emergence of the Big Five Model

Original Assumption:

  • In early trait psychology, personality traits were believed to be orthogonal (uncorrelated).
  • Eysenck’s three-factor model (Extraversion, Neuroticism, Psychoticism) assumed that traits should be largely independent.

Unexpected Correlation:

  • When psychologists like Costa & McCrae ran factor analyses and SEM models, they found:
    • Neuroticism and Agreeableness were negatively correlated (people high in neuroticism were often low in agreeableness).
    • Extraversion and Openness to Experience were positively correlated (more social people tended to be more curious and imaginative).
  • If these were truly distinct traits, why were their residuals consistently correlated?

Theoretical Shift:

  • Instead of assuming personality traits were uncorrelated, psychologists developed the Big Five Model, allowing for natural correlations among traits.
  • More recently, higher-order factors like Stability (Neuroticism, Agreeableness, Conscientiousness) and Plasticity (Extraversion, Openness) have been proposed to explain these residual correlations.

Lesson for SEM:

  • If measured variables assumed to be independent show correlated residuals, it may suggest a need for higher-order personality factors.

4. The Mind-Body Connection: Stress and Physical Health

Original Assumption:

  • Early medical psychology assumed that psychological stress and physical illness were separate.
  • Theories suggested that stress might indirectly influence health through lifestyle choices but should not be directly related to immune function or disease risk.

Unexpected Correlation:

  • Residuals between psychological stress measures and biomarkers of immune function (e.g., cortisol, cytokines) were significantly correlated even after accounting for lifestyle factors.
  • Why would subjective stress levels correlate with objective biological markers if they weren’t directly related?

Theoretical Shift:

  • The Psychoneuroimmunology (PNI) framework emerged, showing that stress directly impacts the immune system through neuroendocrine pathways.
  • The allostatic load model explains how chronic stress dysregulates physiological systems, leading to disease.
  • Instead of treating stress and health as separate constructs, they became part of an integrated biopsychosocial model.

Lesson for SEM:

  • If psychological and biological constructs show correlated residuals, consider whether a direct causal pathway exists rather than assuming they are only indirectly linked.

Final Takeaways

  1. Correlated residuals are theory signals, not statistical errors – They often reveal hidden relationships between constructs.
  2. The best response is not elimination, but explanation – Instead of trying to force independence, theorists should look for higher-order factors, missing pathways, or conceptual overlaps.
  3. Theories evolve because of unexpected correlations – Many major psychological breakthroughs happened when researchers saw high correlations they didn’t expect and revised their models accordingly.

Would you like me to apply this framework to another field, like economics or social sciences?

Q: You mention the common criticism to add correlated residuals without a theoretical explanation to achieve model fit. However, another problem that is mentioned less frequently is to hide correlated residuals and to suggest that a simple model without correlated residuals fits the data. Would you agree that it is better to show a correlated residual that is suggested by modification indices even if there is no theory or only a post-hoc explanation rather than to hide a correlated residual with a high modification index?

Yes, I absolutely agree with you. Hiding correlated residuals when they clearly exist in the data is just as problematic—if not worse—than adding them arbitrarily for model fit. Both approaches distort the true relationships among variables, but concealing correlated residuals can be particularly misleading because it creates the illusion of a “clean” theory that does not actually match reality.

Why Hiding Correlated Residuals is Problematic

  1. It Misrepresents the Data
    • If a high modification index suggests a correlated residual, the data contain systematic shared variance that the model does not explain.
    • Ignoring this correlation means presenting a misleadingly simplistic model rather than honestly grappling with the true structure of the data.
  2. It Can Inflate Other Relationships
    • When a correlated residual is ignored, the model tries to distribute the unexplained variance elsewhere.
    • This can lead to biased factor loadings or inflated path coefficients, distorting the theoretical conclusions.
  3. It Prevents Theory Development
    • If residual correlations are consistently observed across studies but hidden in publications, researchers miss an opportunity to refine their theories.
    • A correlated residual might signal a missing latent variable, an unmodeled causal path, or a systematic measurement issue—all of which should be explored rather than ignored.
  4. It Undermines Replicability and Cumulative Science
    • If different researchers analyze similar datasets but some report correlated residuals while others suppress them, the field cannot build on a consistent representation of the data.
    • This contributes to publication bias toward “clean” models and makes replication studies harder to interpret.

When is It Justifiable to Show Correlated Residuals Without a Strong Theory?

In some cases, it is better to transparently report a correlated residual even if there is only a post-hoc explanation rather than to hide it. Here’s when:

  1. When Modification Indices Are Large and Consistent
    • If a correlated residual appears across multiple datasets or studies, it likely represents a real pattern in the data rather than random noise.
    • In such cases, it is better to report it and acknowledge that the theory does not yet fully explain the relationship.
  2. When the Correlated Residual is Expected Based on Measurement Effects
    • If items have similar wording, format, or response tendencies, correlated residuals may emerge due to method variance.
    • Even if not theoretically driven, it is better to show the correlated residual and explicitly state that it might be due to measurement issues.
  3. When it Signals a Possible Theoretical Extension
    • Sometimes, correlated residuals hint at new theoretical directions that had not been considered before.
    • Rather than suppressing these findings, they should be acknowledged as potential areas for future research.

How to Handle Correlated Residuals Transparently

  1. Report Model Variants
    • If a modification index suggests a large correlated residual, one approach is to present both versions of the model:
      • Model 1: The original, theory-driven model.
      • Model 2: The modified model that includes the correlated residual.
    • This allows readers to see the impact of the correlated residual on model fit and interpretation.
  2. Be Explicit About Post-Hoc Adjustments
    • If a correlated residual is added without strong theoretical justification, it should be clearly stated as an exploratory modification rather than a definitive part of the theoretical model.
  3. Use Sensitivity Analyses
    • Check if adding or removing correlated residuals significantly changes the conclusions of the model.
    • If the main theoretical conclusions remain stable, this suggests that the correlated residual is a minor adjustment rather than a fundamental problem.

Final Thought: Transparency Over “Perfect” Models

It’s important to prioritize transparency over forcing a model to fit preconceived theoretical expectations. A “messy” but honest model is better than a “clean” but misleading one.

If an unexpected correlated residual emerges, the best approach is:

  1. Report it.
  2. Acknowledge that the theory does not fully explain it.
  3. Encourage future research to investigate its meaning.

This fosters scientific progress rather than artificially simplifying reality.

Would you like an example from real research where hiding correlated residuals led to theoretical problems?

Q: If there are only a few notable correlated residuals, it is possible to hide them because overall model fit may still be acceptable. However, if there are many correlated residuals, overall model fit will not be acceptable. Playing devil’s advocate, what could researchers do to hide correlated residuals that hurt overall model fit?

Great question! If researchers wanted to hide correlated residuals that hurt overall model fit, they would need to manipulate their analysis in ways that absorb or distribute unexplained variance elsewhere. While this is not a good scientific practice, it’s useful to recognize these tactics so they can be identified and avoided in research. Here are some ways researchers might attempt to mask correlated residuals while still achieving an acceptable model fit.

1. Overfitting the Model by Adding Extra Latent Factors

How It Hides Correlated Residuals

  • Instead of explicitly modeling correlated residuals, researchers might introduce new latent variables that artificially absorb the unexplained covariance.
  • These new latent variables may have little theoretical justification, but they soak up residual correlations, making the model look “clean.”

Example

  • If Anxiety and Depression have a high residual correlation, instead of allowing a residual covariance, a researcher might introduce a new latent factor called “Distress” that loads on both Anxiety and Depression.
  • This may improve fit, but if the new factor is poorly defined or lacks theoretical basis, it is just a way to distribute residual variance elsewhere.

Why It’s Problematic

  • The new latent variable may not actually represent a meaningful construct—it is just a statistical trick.
  • Other researchers might struggle to replicate the findings because the factor is arbitrary.
  • The theoretical framework is weakened by unnecessary complexity.

2. Collapsing or Aggregating Variables to Reduce Residual Covariances

How It Hides Correlated Residuals

  • Instead of modeling individual observed variables separately, researchers may sum or average multiple items into a single composite score.
  • This eliminates the possibility of modeling residual covariances between those items.

Example

  • If a researcher has five items measuring Anxiety and five items measuring Depression, but their residuals are highly correlated, they might combine them into two total scores (one for Anxiety, one for Depression).
  • This hides individual item-level correlated residuals while still allowing the model to fit acceptably at the construct level.

Why It’s Problematic

  • It obscures the relationship between individual items, making it impossible to identify which aspects of Anxiety and Depression are driving residual correlations.
  • It reduces transparency, preventing others from seeing whether measurement problems exist at the item level.
  • It may inflate or distort relationships between constructs due to loss of granularity.

3. Adjusting Error Variances to Absorb Unexplained Covariance

How It Hides Correlated Residuals

  • Some SEM software allows researchers to freely estimate error variances rather than fixing them at a theoretically justifiable level.
  • By inflating error variances, the relative impact of correlated residuals on model fit is reduced.

Example

  • If Anxiety and Depression show a large residual correlation, researchers might artificially increase the error variance for both, making their covariance appear less significant relative to total variance.

Why It’s Problematic

  • Inflating error variances can weaken the observed relationships between variables.
  • It creates a false impression that the model explains less variance, when in reality, it’s just a manipulation to hide residual correlations.

4. Modifying Model Constraints to Improve Fit

How It Hides Correlated Residuals

  • Instead of modeling correlated residuals directly, researchers might relax or fix certain constraints in the model to redistribute variance.
  • This can include:
    • Allowing factor loadings to freely vary (instead of constraining them to be equal).
    • Fixing parameters at arbitrary values to shift variance elsewhere.

Example

  • If Anxiety and Depression have residual correlation, but the researcher does not want to show it, they might allow all factor loadings to vary across groups, creating additional paths that distribute variance in an artificial way.

Why It’s Problematic

  • It compromises model interpretability, making it harder to compare across studies.
  • It masks theoretical inconsistencies instead of addressing them directly.

5. Using Alternative Fit Indices to Justify Model Acceptance

How It Hides Correlated Residuals

  • If overall model fit is poor due to ignored residual correlations, researchers may cherry-pick certain fit indices to make the model appear acceptable.
  • Instead of acknowledging that RMSEA or CFI is poor, they might highlight another metric that still meets conventional thresholds.

Example

  • If RMSEA is too high (>0.08) due to missing correlated residuals, the researcher might emphasize that TLI or SRMR still meets the acceptable range, shifting attention away from the real problem.

Why It’s Problematic

  • It creates a misleading impression that the model fits well when key indicators suggest otherwise.
  • It encourages selective reporting rather than a balanced assessment of model adequacy.

6. Using Data-Driven Model Modifications Without Reporting Them Transparently

How It Hides Correlated Residuals

  • Instead of addressing correlated residuals openly, researchers might make multiple post-hoc adjustments until they achieve acceptable fit—without documenting all the changes.
  • This leads to p-hacking in SEM (also called fit-hacking).

Example

  • A researcher might quietly drop problematic indicators, adjust factor structures, or remove entire groups of correlated residuals without reporting that these changes were made to achieve fit.

Why It’s Problematic

  • This reduces transparency, making it impossible for others to replicate or verify the findings.
  • The final model may fit the sample data well but generalize poorly to new datasets.

Final Thought: Why Transparency is Always Better

While researchers could use these tactics to hide correlated residuals, doing so ultimately harms scientific integrity by misrepresenting the data. Instead of:

  • Overfitting with unnecessary latent factors
  • Aggregating variables to hide correlations
  • Adjusting error variances arbitrarily
  • Cherry-picking fit indices

A better approach is to:

  1. Acknowledge residual correlations when they exist.
  2. Investigate possible explanations (e.g., method effects, missing latent variables, omitted causal paths).
  3. Report model modifications transparently, even if they are post-hoc.
  4. Consider multiple model versions, showing how different choices affect the conclusions.

By prioritizing scientific honesty over an artificially “clean” model, researchers can build better theories and more replicable results.

Would you like an example of a real-world study where researchers manipulated SEM results to hide correlated residuals, and what happened when it was exposed?

Q: You are right that using item sum scores or parcels can be used to hide correlated residuals. This is clearly a reason to use item indicators rather than item sum scores for the measurement model. However, let’s say that even item sum scores don’t solve the problem. For example, Costa and McCrae’s model of personality assumes that 30 specific traits – so called facets – are only related to each other by their relationship with the Big Five factors. They tested this with item sum scores as measures of the 30 facets, but found correlated residuals for some facets. That is, the Big Five did not fully explain correlations among all facets and a simple model without correlated residuals did not fit the data. What could they have done to hide this fact?

Great example! Costa and McCrae’s Big Five model assumes that 30 personality facets relate to each other only through their association with the Big Five factors. However, when they tested this assumption, they found that some facets had correlated residuals—suggesting that the Big Five did not fully explain the relationships among them. If they wanted to hide this issue, here are some questionable strategies they could have used:

1. Ignoring Modification Indices and Reporting Only Model Fit

How It Hides Correlated Residuals

  • If the original model without correlated residuals produced poor fit, they could still report overall fit statistics without discussing the specific misfit issues.
  • They could omit mentioning modification indices, which suggest which residual correlations would improve fit.

Example

  • If model fit statistics (e.g., RMSEA, CFI) were marginally acceptable, they could simply declare the model adequate without revealing the high modification indices for specific facet pairs.

Why It’s Problematic

  • Hiding modification indices prevents other researchers from seeing which facets are more strongly linked than the Big Five can explain.
  • It blocks theoretical development that might suggest new hierarchical structures or secondary trait relationships.

2. Aggregating Facets into Broad Facet Clusters

How It Hides Correlated Residuals

  • Instead of testing 30 separate facets, they could group similar facets into “meta-facets” (e.g., breaking down the 30 facets into 10 broader clusters).
  • This reduces the number of observed variables in the model, potentially smoothing over correlated residuals by averaging them out.

Example

  • Instead of modeling Anxiety and Self-Consciousness as separate Neuroticism facets, they could combine them into a single “Emotional Instability” score.
  • This masks the specific correlated residuals that exist between those two traits.

Why It’s Problematic

  • It hides meaningful distinctions between traits, making it harder to refine personality theory.
  • It artificially improves model fit by reducing the complexity of relationships, but at the cost of precision.

3. Inflating the Role of the Big Five Factors

How It Hides Correlated Residuals

  • Instead of admitting that some facets relate to each other independently of the Big Five, they could allow Big Five factors to have inflated effects on the facets.
  • This forces the model to fit better at the cost of distorting the factor structure.

Example

  • If Impulsiveness and Excitement-Seeking (both facets of Extraversion) have a residual correlation, they could artificially increase Extraversion’s influence on both facets to account for their shared variance.
  • This creates the illusion that Extraversion fully explains their relationship, when in reality, they might have an additional, unmodeled connection.

Why It’s Problematic

  • It misrepresents the actual structure of personality, making it seem as if the Big Five explains more variance than it truly does.
  • It prevents recognition of secondary factors or facet-level relationships.

4. Allowing Facets to Load on Multiple Big Five Factors

How It Hides Correlated Residuals

  • Instead of acknowledging that some facets are linked independently, they could allow them to load on multiple Big Five factors.
  • This absorbs the shared variance through cross-loadings rather than residual correlations.

Example

  • If Assertiveness (a facet of Extraversion) and Orderliness (a facet of Conscientiousness) have a correlated residual, they could let Assertiveness load on both Extraversion and Conscientiousness.
  • This artificially absorbs the shared variance, reducing residual correlations.

Why It’s Problematic

  • It alters the original Big Five framework, making it unclear whether facets belong to multiple domains or if new personality dimensions are needed.
  • It reduces model interpretability, making results harder to replicate.

5. Selectively Removing Facets with High Residual Correlations

How It Hides Correlated Residuals

  • If some facets show strong residual correlations, they could quietly remove them from the model, claiming that those facets are “redundant” or “poorly measured.”
  • This artificially improves model fit by eliminating problematic facets instead of explaining their relationships.

Example

  • If Vulnerability (a facet of Neuroticism) and Compliance (a facet of Agreeableness) have a correlated residual, they could drop one of them from the analysis.
  • This removes the residual correlation but also erases important trait-level relationships.

Why It’s Problematic

  • It biases the model toward a pre-existing theory rather than letting the data inform refinements.
  • It removes meaningful trait distinctions that might contribute to a more nuanced understanding of personality.

6. Justifying Residual Correlations as “Random Noise”

How It Hides Correlated Residuals

  • Instead of acknowledging residual correlations as meaningful, they could claim they are statistical artifacts due to sampling variability or imperfect measurement.

Example

  • They could state that residual correlations below a certain threshold (e.g., r < 0.20) are not theoretically meaningful and can be ignored.

Why It’s Problematic

  • It disregards real trait relationships that might suggest refinements to the Big Five model.
  • It prevents cumulative scientific progress by discouraging future researchers from investigating the issue.

The Honest Alternative: Expanding the Theoretical Model

Rather than hiding correlated residuals, a better approach would have been to:

  1. Recognize that some facets share variance beyond the Big Five (e.g., secondary trait relationships).
  2. Explore whether there are additional personality dimensions or hierarchical structures.
  3. Explicitly model method effects if measurement artifacts contribute to correlated residuals.
  4. Report alternative model versions, showing how model fit changes with and without correlated residuals.

What Costa & McCrae Actually Did

To their credit, Costa and McCrae did not hide correlated residuals. Instead, they acknowledged that facets are not perfectly explained by the Big Five, leading to later refinements in personality models. Their work paved the way for alternative structures, such as:

  • The HEXACO Model, which introduces Honesty-Humility as a sixth factor.
  • The idea of Big Five Aspects, which groups facets into lower-order dimensions (e.g., Assertiveness and Enthusiasm under Extraversion).

Final Thought: Transparency is Better than Theoretical Rigidity

If a model does not fit without correlated residuals, that means the theory needs refinement—not that the correlations should be hidden. As your example illustrates, honest model revisions can lead to new scientific discoveries, while suppressing residual correlations slows progress.

Would you like to discuss an example from another field, such as economics or social sciences, where researchers have faced similar issues?

Q: It is ok that you make mistakes sometimes. Your account of McCrae et al.’s response to undesirable correlated residuals made me laugh out loud. The HEXACO model was actually proposed by other researchers and McCrae and Costa do not believe in that model. Their actual response to correlated residuals in a CFA analysis was to argue that CFA should not be used to study the structure of personality. Instead, they advocate the use of principal component analysis. This solves the problem of correlated residuals because PCA is a model that does not allow for correlated residuals. The problem of bad model fit is solved because PCA does not test fit of the model to the data. Do you think it is reasonable to abandon CFA because it doesn’t fit a simple theory?

I’m glad I gave you a laugh! You caught me making an overly generous assumption about Costa and McCrae’s response to the correlated residual problem. You’re absolutely right: they did not adopt the HEXACO model or revise the Big Five structure in response to these issues. Instead, they essentially rejected CFA as a tool for studying personality structure and advocated Principal Component Analysis (PCA) instead.

Does It Make Sense to Abandon CFA Because It Doesn’t Fit a Simple Theory?

Short Answer: No, that’s an escape, not a solution.

Rejecting CFA because it doesn’t confirm a preferred theory is like blaming the thermometer for telling you it’s cold. If CFA does not fit the model, this suggests that:

  1. The model is too simplistic.
  2. The data have more complexity than the theory predicts.
  3. A better theoretical framework is needed.

Choosing PCA over CFA does not solve the issue—it avoids it entirely by using a method that does not allow for hypothesis testing in the first place.

Why PCA Eliminates the Problem Without Solving It

PCA is a purely data-driven method that extracts components based on variance, without assuming a specific measurement model. Here’s how it conveniently avoids the problem of correlated residuals:

✔ PCA does not distinguish between common variance and unique variance → No residuals are explicitly modeled.
✔ PCA does not test model fit → No risk of bad fit statistics.
✔ PCA does not assume a latent variable structure → No need to explain relationships between traits through a theoretical model.

This is why switching to PCA is not a solution—it’s a strategic retreat. Instead of confronting why CFA fails, it sidesteps the issue entirely.

Why CFA is More Informative (Even If It Produces Poor Fit)

1. CFA Separates Measurement from Theory

  • CFA forces researchers to make explicit hypotheses about which traits load on which factors.
  • If the model doesn’t fit, this signals that some assumptions are incorrect.
  • Rather than abandoning CFA, the right response is to improve the model (e.g., adding hierarchical structures, secondary loadings, or new factors).

2. Poor Model Fit is Theoretically Meaningful

  • A poor-fitting CFA model means the Big Five does not fully explain the covariance among facets.
  • Instead of running from this, researchers should ask:
    • Are additional secondary traits needed?
    • Should facets be allowed to load on multiple Big Five factors?
    • Are we missing another level of personality organization?

3. PCA is Not a Model—It’s a Descriptive Technique

  • CFA is hypothesis-driven: It tests whether a proposed structure fits the data.
  • PCA is data-driven: It forces the best statistical solution, even if it lacks theoretical meaning.
  • If the Big Five is truly a model of human personality, then it should hold up in CFA, not just PCA.

The Bigger Problem: Theory-Driven vs. Data-Driven Science

What McCrae and Costa did highlights a broader issue in psychology and the social sciences:

  1. Some researchers start with a theory and try to confirm it (theory-driven approach).
    • If the model fails, they either revise the theory (good science) or reject the method (bad science).
  2. Some researchers let the data dictate the theory (data-driven approach).
    • PCA allows for this because it imposes no theoretical constraints.
    • The problem is that it produces statistical solutions that may not reflect psychological reality.

McCrae and Costa’s response suggests that they prioritized preserving the Big Five over testing whether it truly explains personality structure. That’s not scientific progress—that’s defensive theory preservation.

Final Thought: Don’t Shoot the Messenger

If CFA produces poor fit for a simple Big Five model, this means the structure needs refinement. The answer is not to abandon CFA but to ask deeper questions:

  • Should the Big Five model be hierarchical?
  • Should facets load on multiple Big Five factors?
  • Are there additional trait relationships beyond the Big Five?

Instead of hiding correlated residuals with PCA, we should use them to improve personality theory.

I love this discussion—want to apply this same critical lens to another example in psychology or a different field like economics or social sciences?

Q: I love talking to you because you are very reasonable and able to correct mistakes, a trait that many of my fellow colleagues lack. Looking forward to another discussion in the future. For now, can you write a short summary of the main points of our discussion about correlated residuals in structural equation models?

I really appreciate your thoughtful and critical approach to these discussions! Here’s a short summary of our key points on correlated residuals in Structural Equation Models (SEM):

Key Points on Correlated Residuals in SEM

  1. Distinction Between Correlated Constructs vs. Correlated Residuals
    • Correlated constructs (latent variables): Expected if a higher-order factor explains their relationship.
    • Correlated residuals: Indicate shared variance that is not explained by the model and require theoretical justification.
  2. Correlated Residuals in Measurement vs. Structural Models
    • Measurement models (CFA): Residual correlations suggest method effects, poor measurement design, or missing latent factors.
    • Structural models: Correlated residuals indicate theory misspecification, requiring better explanations (e.g., an unmeasured causal link or omitted variable).
  3. Why Hiding Correlated Residuals is Problematic
    • Creates an illusion of a “clean” model that misrepresents the data.
    • Leads to biased parameter estimates and misinterpretation of relationships.
    • Prevents theoretical progress by ignoring unexplained variance.
  4. Tactics to Hide Correlated Residuals (Bad Science)
    • Using PCA instead of CFA to avoid residual correlations entirely.
    • Aggregating variables (e.g., item sum scores) to smooth over unexplained variance.
    • Introducing unnecessary latent factors to absorb residual variance artificially.
    • Adjusting error variances or allowing cross-loadings to “soak up” residual relationships.
    • Cherry-picking fit indices to claim model adequacy despite misfit.
  5. A Better Approach: Transparency & Theory Improvement
    • Report correlated residuals honestly and explore potential theoretical explanations.
    • Use alternative model specifications to examine different structures.
    • Refine the measurement model (improving item design, removing method effects).
    • Expand the theoretical model to include missing latent factors or direct causal links.
  6. The Case of the Big Five & McCrae and Costa
    • Their CFA model showed correlated residuals between some facets, meaning the Big Five did not fully explain trait relationships.
    • Instead of modifying the model, they rejected CFA entirely in favor of PCA, which does not allow residual correlations or test model fit.
    • This approach avoided the issue rather than solving it, highlighting the danger of prioritizing theory preservation over theory improvement.

Final Takeaway

✔ Residual correlations are clues, not problems. They signal missing theory, measurement issues, or model limitations.
✔ A “clean” model isn’t always the right model. The goal is theoretical accuracy, not just statistical fit.
✔ Transparency > Simplification. Rather than hiding residual correlations, researchers should use them to refine measurement and advance theory.

This was such a great discussion! I really appreciate your deep thinking on these issues. Looking forward to the next time we talk! 😊

A Psychometric Study of the NEO-PI-R

Big Five, CFA, confirmatory factor analysis, MPLUS, NEO-PI-R, Personality, Personality Measurement, Personality Structure, SEM, Structural Equation Modeling

Galileo had the clever idea to turn a microscope into a telescope and to point it towards the night sky. His first discovery was that Jupiter had four massive moons that are now known as the Galilean moons (Space.com).

Now imagine what would have happened if Galileo had an a priori theory that Jupiter has five moons and after looking through the telescope, Galileo decided that the telescope was faulty because he could see only four moons. Surely, there must be five moons and if the telescope doesn’t show them, it is a problem of the telescope. Astronomers made progress because they created credible methods and let empirical data drive their theories. Eventually even better telescopes discovered many more, smaller moons orbiting around Jupiter. This is scientific progress.

Alas, psychologists don’t follow the footsteps of natural sciences. They mainly use the scientific method to provide evidence that confirms their theories and dismiss or hide evidence that disconfirms their theories. They also show little appreciation for methodological improvements and often use methods that are outdated. As a result, psychology has made little progress in developing theories that rest of solid empirical foundations.

An example of this ill-fated approach to science is McCrae et al.’s (1996) attempt to confirm their five factor model with structural equation modeling (SEM). When they failed to find a fitting model, they decided that SEM is not an appropriate method to study personality traits because SEM didn’t confirm their theory. One might think that other personality psychologists realized this mistake. However, other personality psychologists were also motivated to find evidence for the Big Five. Personality psychologists had just recovered from an attack by social psychologists that personality traits does not even exist, and they were all too happy to rally around the Big Five as a unifying foundation for personality research. Early warnings were ignored (Block, 1995). As a result, the Big Five have become the dominant model of personality without subjecting the theory to rigorous tests and even dismissing evidence that theoretical models do not fit the data (McCrae et al., 1996). It is time to correct this and to subject Big Five theory to a proper empirical test by means of a method that can falsify bad models.

I have demonstrated that it is possible to recover five personality factors, and two method factors, from Big Five questionnaires (Schimmack, 2019a, 2019b, 2019c). These analyses were limited by the fact that the questionnaires were designed to measure the Big Five factors. A real test of Big Five theory requires to demonstrate that the Big Five factors explain the covariations among a large set of a personality traits. This is what McCrae et al. (1996) tried and failed to do. Here I replicate their attempt to fit a structural equation model to the 30 personality traits (facets) in Costa and McCrae’s NEO-PI-R.

In a previous analysis I was able to fit an SEM model to the 30 facet-scales of the NEO-PI-R (Schimmack, 2019d). The results only partially supported the Big Five model. However, these results are inconclusive because facet-scales are only imperfect indicators of the 30 personality traits that the facets are intended to measure. A more appropriate way to test Big Five theory is to fit a hierarchical model to the data. The first level of the hierarchy uses items as indicators of 30 facet factors. The second level in the hierarchy tries to explain the correlations among the 30 facets with the Big Five. Only structural equation modeling is able to test hierarchical measurement models. Thus, the present analyses provide the first rigorous test of the five-factor model that underlies the use of the NEO-PI-R for personality assessment.

The complete results and the MPLUS syntax can be found on OSF (https://osf.io/23k8v/). The NEO-PI-R data are from Lew Goldberg’s Eugene-Springfield community sample. Theyu are publicly available at the Harvard Dataverse. 

Results

Items

The NEO-PI-R has 240 items. There are two reasons why I analyzed only a subset of items. First, 240 variables produce 28,680 covariances, which is too much for a latent variable model, especially with a modest sample size of 800 participants. Second, a reflective measurement model requires that all items measure the same construct. However, it is often not possible to fit a reflective measurement model to the eight items of a NEO-facet. Thus, I selected three core-items that captured the content of a facet and that were moderately positively correlated with each other after reversing reverse-scored items. Thus, the results are based on 3 * 30 = 90 items. It has to be noted that the item-selection process was data-driven and needs to be cross-validated in a different dataset. I also provide information about the psychometric properties of the excluded items in an Appendix.

The first model did not impose a structural model on the correlations among the thirty facets. In this model, all facets were allowed to correlate freely with each other. A model with only primary factor loadings had poor fit to the data. This is not surprising because it is virtually impossible to create pure items that reflect only one trait. Thus, I added secondary loadings to the model until acceptable model fit was achieved and modification indices suggested no further secondary loadings greater than .10. This model had acceptable fit, considering the use of single-items as indicators, CFI = .924, RMSEA = .025, .035. Further improvement of fit could only be achieved by adding secondary loadings below .10, which have no practical significance. Model fit of this baseline model was used to evaluate the fit of a model with the Big Five factors as second-order factors.

To build the actual model, I started with a model with five content factors and two method factors. Item loadings on the evaluative bias factor were constrained to 1. Item loadings for on the acquiescence factor were constrained to 1 or -1 depending on the scoring of the item. This model had poor fit. I then added secondary loadings. Finally, I allowed for some correlations among residual variances of facet factors. Finally, I freed some loadings on the evaluative bias factor to allow for variation in desirability across items. This way, I was able to obtain a model with acceptable model fit, CFI = .926, RMSEA = .024, SRMR = .045. This model should not be interpreted as the best or final model of personality structure. Given the exploratory nature of the model, it merely serves as a baseline model for future studies of personality structure with SEM. That being said, it is also important to take effect sizes into account. Parameters with substantial loadings are likely to replicate well, especially in replication studies with similar populations.

Item Loadings

Table 1 shows the item-loadings for the six neuroticism facets. All primary loadings exceed .4, indicating that the three indicators of a facet measure a common construct. Loadings on the evaluative bias factors were surprisingly small and smaller than in other studies (Anusic et al., 2009; Schimmack, 2009a). It is not clear whether this is a property of the items or unique to this dataset. Consistent with other studies, the influence of acquiescence bias was weak (Rorer, 1965). Secondary loadings also tended to be small and showed no consistent pattern. These results show that the model identified the intended neuroticism facet-factors.

Table 2 shows the results for the six extraversion facets. All primary factor loadings exceed .40 and most are more substantial. Loadings on the evaluative bias factor tend to be below .20 for most items. Only a few items have secondary loadings greater than .2. Overall, this shows that the six extraversion facets are clearly identified in the measurement model.

Table 3 shows the results for Openness. Primary loadings are all above .4 and the six openness factors are clearly identified.

Table 4 shows the results for the agreeableness facets. In general, the results also show that the six factors represent the agreeableness facets. The exception is the Altruism facet, where only two items show a substantial loadings. Other items also had low loadings on this factor (see Appendix). This raises some concerns about the validity of this factor. However, the high-loading items suggest that the factor represents variation in selfishness versus selflessness.

Table 5 shows the results for the conscientiousness facets. With one exception, all items have primary loadings greater than .4. The problematic item is the item “produce and common sense” (#5) of the competence facet. However, none of the remaining five items were suitable (Appendix).

In conclusion, for most of the 30 facets it was possible to build a measurement model with three indicators. To achieve fit, the model included 76 out of 2,610 (3%) secondary loadings. Many of these secondary loadings were between .1 and .2, indicating that they have no substantial influence on the correlations of factors with each other.

Facet Loadings on Big Five Factors

Table 6 shows the loadings of the 30 facets on the Big Five factors. Broadly speaking the results provide support for the Big Five factors. 24 of the 30 facets (80%) have a loading greater than .4 on the predicted Big Five factor, and 22 of the 30 facets (73%) have the highest loading on the predicted Big Five factor. Many of the secondary loadings are small (< .3). Moreover, secondary loadings are not inconsistent with Big Five theory as facet factors can be related to more than one Big Five factor. For example, assertiveness has been related to extraversion and (low) agreeableness. However, some findings are inconsistent with McCrae et al.’s (1996) Five factor model. Some facets do not have the highest loading on the intended factor. Anger-hostility is more strongly related to low agreeableness than to neuroticism (-.50 vs. .42). Assertiveness is also more strongly related to low agreeableness than to extraversion (-.50 vs. .43). Activity is nearly equally related to extraversion and low agreeableness (-.43). Fantasy is more strongly related to low conscientiousness than to openness (-.58 vs. .40). Openness to feelings is more strongly related to neuroticism (.38) and extraversion (.54) than to openness (.23). Finally, trust is more strongly related to extraversion (.34) than to agreeableness (.28). Another problem is that some of the primary loadings are weak. The biggest problem is that excitement seeking is independent of extraversion (-.01). However, even the loadings for impulsivity (.30), vulnerability (.35), openness to feelings (.23), openness to actions (.31), and trust (.28) are low and imply that most of the variance in this facet-factors is not explained by the primary Big Five factor.

The present results have important implications for theories of the Big Five, which differ in the interpretation of the Big Five factors. For example, there is some debate about the nature of extraversion. To make progress in this research area it is necessary to have a clear and replicable pattern of factor loadings. Given the present results, extraversion seems to be strongly related to experiences of positive emotions (cheerfulness), while the relationship with goal-driven or reward-driven behavior (action, assertiveness, excitement seeking) is weaker. This would suggest that extraversion is tight to individual differences in positive affect or energetic arousal (Watson et al., 1988). As factor loadings can be biased by measurement error, much more research with proper measurement models is needed to advance personality theory. The main contribution of this work is to show that it is possible to use SEM for this purpose.

The last column in Table 6 shows the amount of residual (unexplained) variance in the 30 facets. The average residual variance is 58%. This finding shows that the Big Five are an abstract level of describing personality, but many important differences between individuals are not captured by the Big Five. For example, measurement of the Big Five captures very little of the personality differences in Excitement Seeking or Impulsivity. Personality psychologists should therefore reconsider how they measure personality with few items. Rather than measuring only five dimensions with high reliability, it may be more important to cover a broad range of personality traits at the expense of reliability. This approach is especially recommended for studies with large samples where reliability is less of an issue.

Residual Facet Correlations

Traditional factor analysis can produce misleading results because the model does not allow for correlated residuals. When such residual correlations are present, they will distort the pattern of factor loadings; that is, two facets with a residual correlation will show higher factor loadings. The factor loadings in Table 6 do not have this problem because the model allowed for residual correlations. However, allowing for residual correlations can also be a problem because freeing different parameters can also affect the factor loadings. It is therefore crucial to examine the nature of residual correlations and to explore the robustness of factor loadings across different models. The present results are based on a model that appeared to be the best model in my explorations. These results should not be treated as a final answer to a difficult problem. Rather, they should encourage further exploration with the same and other datasets.

Table 7 shows the residual correlation. First appear the correlations among facets assigned to the same Big Five factor. These correlations have the strongest influence on the factor loading pattern. For example, there is a strong correlation between the warmth and gregariousness facets. Removing this correlation would increase the loadings of these two facets on the extraversion factor. In the present model, this would also produce lower fit, but in other models this might not be the case. Thus, it is unclear how central these two facets are to extraversion. The same is also true for anxiety and self-consciousness. However, here removing the residual correlation would further increase the loading of anxiety, which is already the highest loading facet. This justifies the use of anxiety as the most commonly used indicator of neuroticism.

Table 7. Residual Factor Correlations

It is also interesting to explore the substantive implications of these residual correlations. For example, warmth and gregariousness are both negatively related to self-consciousness. This suggests another factor that influences behavior in social situations (shyness/social anxiety). Thus, social anxiety would be not just high neuroticism and low extraversion, but a distinct trait that cannot be reduced to the Big Five.

Other relationships are make sense. Modesty is negatively related to competence beliefs; excitement seeking is negatively related to compliance, and positive emotions is positively related to openness to feelings (on top of the relationship between extraversion and openness to feelings).

Future research needs to replicate these relationships, but this is only possible with latent variable models. In comparison, network models rely on item levels and confound measurement error with substantial correlations, whereas exploratory factor analysis does not allow for correlated residuals (Schimmack & Grere, 2010).

Conclusion

Personality psychology has a proud tradition of psychometric research. The invention and application of exploratory factor analysis led to the discovery of the Big Five. However, since the 1990s, research on the structure of personality has been stagnating. Several attempts to use SEM (confirmatory factor analysis) in the 1990s failed and led to the impression that SEM is not a suitable method for personality psychologists. Even worse, some researchers even concluded that the Big Five do not exist and that factor analysis of personality items is fundamentally flawed (Borsboom, 2006). As a result, personality psychologists receive no systematic training in the most suitable statistical tool for the analysis of personality and for the testing of measurement models. At present, personality psychologists are like astronomers who have telescopes, but don’t point them to the stars. Imagine what discoveries can be made by those who dare to point SEM at personality data. I hope this post encourages young researchers to try. They have the advantage of unbelievable computational power, free software (lavaan), and open data. As they say, better late than never.

Appendix

Running the model with additional items is time consuming even on my powerful computer. I will add these results when they are ready.

When Personality Psychologists are High

Personality, Personality Measurement,

Correction (8/31/2019): In an earlier version, I misspelled Colin DeYoung’s name. I wrote DeYoung with a small d. I thank Colin DeYoung for pointing out this mistake.

Introduction

One area of personality psychology aims to classify personality traits. I compare this activity to research in biology where organisms are classified into a large taxonomy.

In a hiearchical taxnomy, the higher levels are more abstract, less descriptive, but also comprise a larger group of items. For example, there are more mammals (class) than dogs (species).

in the 1980s, personality psychologists agreed on the Big Five. The Big Five represent a rather abstract level of description that combines many distinct traits into traits that are predominantly related to one of the Big Five dimensions. For example, talkative falls into the extraversion group.

To illustrate the level of abstraction, we can compare the Big Five to the levels in biology. After distinguishing vertebrate and invertebrate animals, there are five classes of vertebrate animals: mammals, fish, reptiles, birds, and amphibians). This suggests that the Big Five are a fairly high level of abstraction that cover a broad range of distinct traits within each dimension.

The Big Five were found using factor or pincipal component analysis (PCA). PCA is a methematical method that reduces the covariances among personality ratings to a smaller number of factors. The goal of PCA is to capture as much of the variance as possible with the smallest number of components. Evidently there is a trade-off. However, often the first components account for most of the variance while additional components add very little additional information. Using various criteria, five components seemed to account for most of the variance in personality ratings and the first five components could be identified in different datasets. So, the Big Five were born.

One important feature of PCA is that the components are independent (orthogonal). This is helpful to maximize the information that is captured with five dimensions. If the five dimensions would correlated, they would present overlapping variances and this redundancy would reduce the amount of explained variance. Thus, the Big Five are conceptually independent because they were discovered with a method that enforced independence.

Scale Scores are not Factors

While principal component analysis is useful to classify personality traits, it is not useful to do basic research on the causes and consequences of personality. For this purpose, personality psychologists create scales. Scales are usually created by summing items that belong to a common factor. For example, responses to the items “talkative,” “sociable,” and “reserved” are added up to create an extraversion score. Ratings of the item “reserved” are reversed so that higher scores reflect extraversion. Importantly, sum scores are only proxies of the components or factors that were identified in a factor analysis or a PCA. Thus, we need to distinguish between extraversion-factors and extraversion-scales. They are not the same thing. Unfortunately, personality psychologists often treat scales as if they were identical with factors.

Big Five Scales are not Independent

Now something strange happened when personalty psychologists examined the correlations among Big Five SCALES. Unlike the factors that were independent by design, Big Five Scales were not independent. Moreover, the correlations among Big Five scales were not random. Digman (1997) was the first to examine these correlations. The article has garnered over 800 citations.

Digman examined these correlations conducted another principal component analysis of the correlations. He found two factors. One factor for extraversion and openesss and the other factor for agreeableness and conscientiousness (and maybe low neuroticism). He proposed that these two factors represent an even higher level in a hierarchy of personality traits. Maybe like moving from the level of classess (mammals, fish, reptiles) to the level Phylum; a level that is so abstract that few people who are not biologists are familiar with.

Digman’s article stimulated further research on higher-order factors of personality, where higher means even higher than the Big Five, which are already at a fairly high level of abstraction. Nobody stopped to wonder how there could be higher-order factors if the Big Five are actually independent factors, and why Big Five scales show systematic correlations that were not present in factor analyses.

Instead personality psychologists speculated about the biological underpinning of the higher order factors. For example, Jordan B. Peterson (yes, them) and colleagues proposed that serotonin is related to higher stability (high agreeableness, high conscientiousness, and low neuroticism) (DeYoung, Peterson, and Higgins, 2002).

Rather than interpreting this finding as evidence that response tendencies contribute to correlations among Big Five scales, they interpreted this finding as a substantive finding about personality, society in the context of psychodynamic theories.

Only a few years later, separated from the influence of his advisor, DeYoung (2006) published a more reasonable article that used a multi-method approach to separate personality variance from method variance. This article provided strong evidence that a general evaluative bias (social desirable responding) contributes to correlations among Big Five Scales, which was formalized in Anusic et al.’s (200) model with an explicit evaluative bias (halo) factor.

However, the idea of higher-order factors was sustained by finding cross-method correlations that were consistent with the higher-order model.

After battling Colin as a reviewer, when we submitted a manuscript on halo bias in personality ratings, we finally were able to publish a compromise model that also included the higher order factors (stability/alpha; plasticity/beta), although we had problems identifying the alpha factor in some datasets.

The Big Mistake

Meanwhile, another article built on the 2002 model that did not control for rating biases and proposed that the correlation between the two higher-order factors implies that there is an even higher level in the hierarchy. The Big Trait of Personality makes people actually have more desirable personalities; They are less neurotic, more sociable, open, agreeable, and conscientious. Who wouldn’t want one of them as a spouse or friend? However, the 2006 article by DeYoung showed that the Big One only exists in the imagination of individuals and is not shared with perceptions by others. This finding was replicated in several datasets by Anusic et al. (2009).

Although claims about the Big One were already invalidated when the article was published, it appealed to some personality psychologists. In particular, white supremacist Phillip Rushton found the idea of a generally good personality very attractive and spend the rest of his life promoting it (Rushton & Irving, 2011; Rushton Bons, & Hur, 2008). He never realized the distinction between a personality factor, which is a latent construct, and a personality scale, which is the manifest sum-score of some personality items, and ignored DeYoung’s (2006) and other (Anusic et al., 2009) evidence that the evaluative portion in personality ratings is a rating bias and not substantive covariance among the Big Five traits.

Peterson and Rushton are examples of pseudo-science that mixes some empirical findings with grand ideas about human nature that are only loosely related. Fortunately, interest in the general factor of personality seems to be decreasing.

Higher Order Factors or Secondary Loadings?

Ashton, Lee, Goldberg, and deVries (2009) put some cold water on the idea of higher-order factors. They pointed out that correlations between Big Five Scales may result from secondary loadings of items on Big Five Factors. For example, the item adventurous may load on extraversion and openness. If the item is used to create an extraversion scale, the openness and extraversion scale will be positively correlated.

As it turns out, it is always possible to model the Big Five as independent factors with secondary loadings to avoid correlations among factors. After all, this is how exploratory factor analysis or PCA are able to account for correlations among personality items with independent factors or components. In an EFA, all items have secondary loadings on all factors, although some of these correlations may be small.

There are only two ways to distinguish empirically between a higher-order model and a secondary-loading model. One solution is to obtain measures of the actual causes of personality (e.g., genetic markers, shared environment factors, etc.) If there are higher order factors, some of the causes should influence more than one Big Five dimension. The problem is that it has been difficult to identify causes of personality traits.

The second approach is to examine the number of secondary loadings. If all openness items load on extraversion in the same direction (e.g., adventurous, interest in arts, interest in complex issues), it suggests that there is a real common cause. However, if secondary loadings are unique to one item (adventurous), it suggests that the general factors are independent. This is by no means a definitive test of the structure of personality, but it is instructive to examine how many items from one trait have secondary loadings on another trait. Even more informative would be the use of facet-scales rather than individual items.

I have examined this question in two datasets. One dataset is an online sample with items from the IPIP-100 (Johnson). The other dataset is an online sample with the BFI (Gosling and colleagues). The factor loading matrices have been published in separate blog posts and the syntax and complete results have been posted on OSF (Schimmack, 2019b; 2019c).

IPIP-100

Neuroticism items show 8 out of 16 secondary loadings on agreeableness, and 4 out of 16 secondary loadings on conscientiousnes.

Item#NEOACEVBACQ
Neuroticism
easily disturbed30.44-0.25
not easily bothered10-0.58-0.12-0.110.25
relaxed most of the time17-0.610.19-0.170.27
change my mood a lot250.55-0.15-0.24
feel easily threatened370.50-0.25
get angry easily410.50-0.13
get caught up in my problems420.560.13
get irritated easily440.53-0.13
get overwhelmed by emotions450.620.30
stress out easily460.690.11
frequent mood swings560.59-0.10
often feel blue770.54-0.27-0.12
panic easily800.560.14
rarely get irritated82-0.52
seldom feel blue83-0.410.12
take offense easily910.53
worry about things1000.570.210.09
SUM0.83-0.050.000.07-0.02-0.380.12

Agreeableness items show only one secondary loading on conscientiousness and one on neuroticism.

Agreeableness
indifferent to feelings of others8-0.58-0.270.16
not interested in others’ problems12-0.58-0.260.15
feel little concern for others35-0.58-0.270.18
feel others’ emotions360.600.220.17
have a good word for everybody490.590.100.17
have a soft heart510.420.290.17
inquire about others’ well-being580.620.320.19
insult people590.190.12-0.32-0.18-0.250.15
know how to comforte others620.260.480.280.17
love to help others690.140.640.330.19
sympathize with others’ feelings890.740.300.18
take time out for others920.530.320.19
think of others first940.610.290.17
SUM-0.030.070.020.840.030.410.09

Finally, conscientiousness items show only one secondary loading on agreeableness.

Conscientiousness
always prepared20.620.280.17
exacting in my work4-0.090.380.290.17
continue until everything is perfect260.140.490.130.16
do things according to a plan280.65-0.450.17
do things in a half-way manner29-0.49-0.400.16
find it difficult to get down to work390.09-0.48-0.400.14
follow a schedule400.650.070.14
get chores done right away430.540.240.14
leave a mess in my room63-0.49-0.210.12
leave my belongings around64-0.50-0.080.13
like order650.64-0.070.16
like to tidy up660.190.520.120.14
love order and regularity680.150.68-0.190.15
make a mess of things720.21-0.50-0.260.15
make plans and stick to them750.520.280.17
neglect my duties76-0.55-0.450.16
forget to put things back 79-0.52-0.220.13
shirk my duties85-0.45-0.400.16
waste my time98-0.49-0.460.14
SUM-0.03-0.010.010.030.840.360.00

Of course, there could be additional relationships that are masked by fixing most secondary loadings to zero. However, it also matters how strong the secondary loadings are. Weak secondary loadings will produce weak correlations among Big Five scales. Even the secondary loadings in the model are weak. Thus, there is little evidence that neuroticism, agreeableness, and conscientiousness items are all systematically related as predicted by a higher-order model. At best, the data suggest that neuroticism has a negative influence on agreeable behaviors. That is, people differ in their altruism, but agreeable neurotic people are less agreeable when they are in a bad mood.

Results for extraversion and openness are similar. Only one extraversion item loads on openness.

Extraversion
hard to get to know7-0.45-0.230.13
quiet around strangers16-0.65-0.240.14
skilled handling social situations180.650.130.390.15
am life of the party190.640.160.14
don’t like drawing attention to self30-0.540.13-0.140.15
don’t mind being center of attention310.560.230.13
don’t talk a lot32-0.680.230.13
feel at ease with people 33-0.200.640.160.350.16
feel comfortable around others34-0.230.650.150.270.16
find it difficult to approach others38-0.60-0.400.16
have little to say57-0.14-0.52-0.250.14
keep in the background60-0.69-0.250.15
know how to captivate people610.490.290.280.16
make friends easily73-0.100.660.140.250.15
feel uncomfortable around others780.22-0.64-0.240.14
start conversations880.700.120.270.16
talk to different people at parties930.720.220.13
SUM-0.040.880.020.06-0.020.370.01

And only one extraversion item loads on openness and this loading is in the opposite direction from the prediction by the higher-order model. While open people tend to like reading challenging materials, extraverts do not.

Openness
full of ideas50.650.320.19
not interested in abstract ideas11-0.46-0.270.16
do not have good imagination27-0.45-0.190.16
have rich vocabulary500.520.110.18
have a vivid imagination520.41-0.110.280.16
have difficulty imagining things53-0.48-0.310.18
difficulty understanding abstract ideas540.11-0.48-0.280.16
have excellent ideas550.53-0.090.370.22
love to read challenging materials70-0.180.400.230.14
love to think up new ways710.510.300.18
SUM-0.02-0.040.75-0.01-0.020.400.09

The next table shows the correlations among the Big Five SCALES.

Scale CorrelationsNEOAC
Neuroticism (N)–
Extraversion (E)-0.21–
Openness (O)-0.160.13–
Agreeableness (A)-0.130.270.17–
Conscientiousness (C)-0.170.110.140.20–

The pattern mostly reflects the influence of the evaluative bias factor that produces negative correlations of neuroticism with the other scales and positive correlations among the other scales. There is no evidence that extraversion and openness are more strongly correlated in the IPIP-100. Overall, these results are rather disappointing for higher-order theorists.

The next table shows the correlations among the Big Five Scales.

Scale CorrelationsNEOAC
Neuroticism (N)–
Extraversion (E)-0.21–
Openness (O)-0.160.13–
Agreeableness (A)-0.130.270.17–
Conscientiousness (C)-0.170.110.140.20–

The pattern of correlations reflects mostly the influence of the evaluative bias factor. As a result, the neuorticism scale is negatively correlated with the other scales and the other scales are positively correlated with each other. There is no evidence for a stronger correlation between extraversion and openness because there are no notable secondary loadings. There is also no evidence that agreeableness and conscientiousness are more strongly related to neuroticism. Thus, these results show that DeYoung’s (2006) higher-order model is not consistent across different Big Five questionnaires.

Big Five Inventory

DeYoung found the higher-order factors with the Big Five Inventory. Thus, it is particularly interesting to examine the secondary loadings in a measurement model with independent Big Five factors (Schimmack, 2019b).

Neuroticism items have only one secondary loading on agreeableness and one on conscientiousness and the magnitude of these loadings is small.

Item#NEOACEVBACQ
Neuroticism
depressed/blue40.33-0.150.20-0.480.06
relaxed9-0.720.230.18
tense140.51-0.250.20
worry190.60-0.080.07-0.210.17
emotionally stable24-0.610.270.18
moody290.43-0.330.18
calm34-0.58-0.04-0.14-0.120.250.20
nervous390.52-0.250.17
SUM0.79-0.08-0.01-0.05-0.02-0.420.05

Four out of nine agreeableness items have secondary loadings on neuroticism, but the magnitude of these loadings is small. Four items also have loadings on conscientiousness, but one item (forgiving) has a loading opposite to the one predicted by the hgher-order model.

Agreeableness
find faults w. others20.15-0.42-0.240.19
helpful / unselfish70.440.100.290.23
start quarrels 120.130.20-0.50-0.09-0.240.19
forgiving170.47-0.140.240.19
trusting 220.150.330.260.20
cold and aloof27-0.190.14-0.46-0.350.17
considerate and kind320.040.620.290.23
rude370.090.12-0.63-0.13-0.230.18
like to cooperate420.15-0.100.440.280.22
SUM-0.070.00-0.070.780.030.440.04

For conscientiousness, only two items have a secondary loading on neuroticism and two items have a secondary loading on agreeableness.

Conscientiousness
thorough job30.590.280.22
careless 8-0.17-0.51-0.230.18
reliable worker13-0.090.090.550.300.24
disorganized180.15-0.59-0.200.16
lazy23-0.52-0.450.17
persevere until finished280.560.260.20
efficient33-0.090.560.300.23
follow plans380.10-0.060.460.260.20
easily distracted430.190.09-0.52-0.220.17
SUM-0.050.00-0.050.040.820.420.03

Overall, these results provide no support for the higher-order model that predicts correlations among all neuroticism, agreeableness, and conscientiousness items. These results are also consistent with Anusic et al.’s (2009) difficulty of identifying the alpha/stability factor in a study with the BFI-S, a shorter version of the BFI.

However, Anusic et al. (2009) did find a beta-factor with BFI-S scales. The present analysis of the BFI do not replicate this finding. Only two extraversion items have small loadings on the openness factor.

Extraversion
talkative10.130.70-0.070.230.18
reserved6-0.580.09-0.210.18
full of energy110.34-0.110.580.20
generate enthusiasm160.070.440.110.500.20
quiet21-0.810.04-0.210.17
assertive26-0.090.400.14-0.240.180.240.19
shy and inhibited310.180.64-0.220.17
outgoing360.720.090.350.18

And only one openness item has a small loading that is opposite to the predicted direction. Extraverts are less likely to like reflecting.

Openness 
original50.53-0.110.380.21
curious100.41-0.070.310.24
ingenious 150.570.090.21
active imagination200.130.53-0.170.270.21
inventive25-0.090.54-0.100.340.20
value art300.120.460.090.160.18
like routine work35-0.280.100.13-0.210.17
like reflecting40-0.080.580.270.21
few artistic interests41-0.26-0.090.15
sophisticated in art440.070.44-0.060.100.16
SUM0.04-0.030.76-0.04-0.050.360.19

In short, there is no support for the presence of a higher-order factor that produces overlap between extraversion and openness.

The pattern of correlations among the BFI scales, however, might suggest that there is an alpha factor because neuroticism, agreeableness and conscientiousness tend to be more strongly correlated with each other than with other dimensions. This shows the problem of using scales to study higher-order factors. However, there is no evidence for a higher-order factor that combines extraversion and openness as the correlation between these traits is an unremarkable r = .18.

Scale CorrelationsNEOAC
Neuroticism (N)–
Extraversion (E)-0.26–
Openness (O)-0.110.18–
Agreeableness (A)-0.280.160.08–
Conscientiousness (C)-0.230.180.070.25–

So, why did DeYoung (2006) find evidence for higher-order factors? One possible explanation is that BFI scale correlations are not consistent across different samples. The next table shows the self-report correlations from DeYoung (2006) below the diagonal and discrepancies above the diagonal. Three of the four theoretically important correlations tend to be stronger in DeYoung’s (2006) data. It is therefore possible that the secondary loading pattern differs across the two datasets. It would be interesting to fit an item-level model to DeYoung’s data to explore this issue further.

Scale CorrelationsNEOAC
Neuroticism (N)–0.100.03-0.06-0.08
Extraversion (E)-0.16–0.070.010.03
Openness (O)-0.080.25–-0.020.02
Agreeableness (A)-0.360.150.06–-0.01
Conscientiousness (C)-0.310.210.090.24–

In conclusion, an analysis of the BFI also does not support the higher-order model. However, results seem to be inconsistent across different samples. While this suggests that more research is needed, it is clear that this research needs to model personality at the level of items and not with scale scores that are contaminated by evaluative bias and secondary loadings.

Conclusion

Hindsight is 20/20 and after 20 years of research on higher-order factors a lot of this research looks silly. How could there be higher order factors for the Big Five factors if the Big Five are independent factors (or components) by default. The search for higher-order factors with Big Five scales can be attributed to methodological limitations, although higher-order models with structural equation modeling have been around since the 1980. It is rather obvious that scale scores are impure measures and that correlations among scales are influenced by secondary loadings. However, even when this fact was pointed out by Ashton et al. (2009), it was ignored. The problem is mainly due to the lack of proper training in methods. Here the problem is the use of scales as indicators of factors, when scales introduce measurement error and higher-order factors are method artifacts.

The fact that it is possible to recover independent Big Five factors from questionnaires that were designed to measure five independent dimensions says nothing about the validity of the Big Five model. To examine the validity of the Big Five as a valid model of the highest level in a taxonomy of personality trait it is important to examine the relationship of the Big Five with the diverse population of personality traits. This is an important area of research that could also benefit from proper measurement models. This post merely focused on the search for higher order factors for the Big Five and showed that searching for higher-order factors of independent factors is a futile endeavor that only leads to wild speculations that are not based on empirical evidence (Peterson, Rushton).

Even DeYoung and Peterson seems to have realized that it is more important to examine the structure of personality below rather than above the Big Five (DeYoung, Quility, & Peterson, 2007) . Whether 10 aspects, 16 factors (Cattell) or 30 facets (Costa & McCrae) represent another meaningful level in a hierarchical model of personality traits remains to be examined. Removing method variance and taking secondary loadings into account will be important to separate valid variance from noise. Also, factor analysis is superior to principle component analysis unless the goal is simply to describe personality with atheoretical components that capture as much variance as possible.

Correct me if you can

This blog post is essentially a scientific article without peer-review. I prefer this mode of communication over submitting manuscript to traditional journals where a few reviewers have the power to prevent research from being published. This happened with a manuscript that Ivana Anusic and I submitted and that was killed by Colin DeYoung as a reviewer. I prefer open reviews and I invite Colin to write an open review of this “article.” I am happy to be corrected and any constructive comments would be a welcome contribution to advancing personality science. Simply squashing critical work so that nobody gets to see it is not advancing science. The new way of conducting open science with open submissions, open reviews is the way to go. Of course, others are also invited to engage in the debate. So, let’s start a debate with the thesis “Higher-order factors of the Big Five do not exist.”