The past decade has not been kind to experimental social psychology. Study after study failed to replicate and entire literatures have turned out to be built on nothing (a.k.a. statistical noise mining).
“Another day, another idol falls. This one has been teetering for years, so the collapse didn’t come as a shock. But that doesn’t make it any less painful.” (Michael Inzlicht).
It all started with a leading journal publishing an article with the crazy claim that people can foresee the future and practicing after a test can improve exam scores (Bem, 2011). This claim was quickly revealed to be false (and possibly a hoax, Gelman) after a big replication study failed to show the same results (Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P., 2012).
In a media interview Bem explained that his experiments were never meant to be taken seriously. (Daniel Engber, 2017, Slate).
“If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?’
While the past decade has not been good for experimental social psychologists, it has produced a new group of psychologists to examine the causes of the replication crisis in experimental social psychology. As they look at the practices of research psychologists, they are meta-psychologists, psychologists who study other psychologists.
One of them is Blake McShane, who did his dissertation on statistical models to analyze time-series data (McShane, 2010). Given his background in statistics, managerial science, applied economics, and marketing, it is fair to say that he entered this field without first-hand experience of research practices that produced the replication crisis. He also does not cite seminal papers that foreshadowed the crisis by Cohen (1962, 1990, 1994). Instead, his main approach to examining meta-psychological questions appears to rely on his expertise in conducting simulation studies (McShane & Böckenholt, 2014, McShane, Böckenholt, & Hansen, 2016, 2020).
The problem with these simulation studies is that they repeat the same problems that plagued experimental social psychology at the meta-level. Just like Bem’s studies are not empirical tests, but rhetorical devices, McShane’s simulations are rhetorical devices to illustrate a point that does not require simulation evidence, namely.
[models] perform reasonably well in the setting for which they were designed, …[but] they are sensitive to deviations from their model assumptions.
In the 2016 article, the simulations violated assumptions of models that assume homogeneity and they failed. However, the simulations met the assumptions of another model and (no surprise) it worked well. However, McShane did not cite an earlier study that showed the model also has problems when its assumptions are not met (Hedges & Vevea, 1996).
Later simulation studies further confirmed that McShane’s preferred model does not work so well under realistic conditions (Carter et al., 2019), a finding not cited by McShane et al. in 2020. Pressed on this point that his simulations favored his preferred model, he might reply
“If you looked at all my past simulations, they were always rhetorical devices. I created conditions to show how things work when assumptions are met. I used simulations as a point of persuasion, and I never really worried about, ‘Does this apply to real data’ ”
In conclusion, a simulation that shows a model works when its assumptions are true and does not work when its assumptions are false is merely a demonstration, not an evaluation of a model under realistic conditions.