A science that cannot face its history has no future. (Anonymous).
ABSTRACT
Bem (2011) presented incredible results that seemed to provide strong empirical evidence (p < .05^9). The article was published because it passed “a rigorous review process, involving a large set of extremely thorough reviews by distinguished experts in social cognition” and “editors can only take the author at his word that his data are in fact genuine.” Here I show that social psychologists have avoided discussing the broader implications of the method crisis in social psychology. The same standards that were used for Bem’s article are still used for most articles published today: a few significant p-values, peer-review, and trust that researchers are honest are supposed to ensure that results are robust and replicable. However, the replication crisis has shown that this is not the case. Consumers of social psychology need to be aware that even 10 years after Bem’s infamous article, evidence for social psychological theories is no more credible than Bem’s evidence for extra-sensory perception.
Introduction
Daryl Bem was a highly respected social psychologist, until he published his “Feeling the Future” article in 2011.
The article became a poster child of everything that is wrong with research methods in social psychology and has been cited over 300 times.
The article was also accompanied by an editorial that justified the publication of an article that seemed to provide evidence for the incredible claim that humans, or at least extraverts, can feel events that haven’t happened yet.
The editorial suggests that Bem’s article will “stimulate further thoughts about appropriate methods in research on social cognition and attitudes” (p. 406). Ten years later, we can see whether publishing Bem’s article had the intended effect.
The high citation count shows that the article did indeed generate lot’s of discussion about research practices in social cognition research and social psychology more broadly. However, an inspection of these citations shows that most of this discussion occurred outside of social psychology, by meta-psychologists who reflected on research practices by social psychologists. In stark contrast, critical self-reflection by social psychologists is insignificant.
Here I provide a bibliography of these citations. An examination of these citations shows that social psychologists have carefully avoided asking themselves the most important question that follows from Bem’s (2011) article.
If we cannot trust Bem’s article that reported nine statistically significant results, which article in social psychology can we trust?
Etienne P. LeBel & Kurt R. Peters (2011) Review of General Psychology
This article clearly spells out the problems of QRPs and uses Bem’s article to raise questions about other research findings. The first author was trained as a graduate student by Gawronski, but is no longer in social psychology.
Charles M. Judd, Jacob Westfall, & David A. Kenny (2012) JPSP
This article implies that the problem was inappropriate treatment of variation across stimuli. It does not mention the use of QRPs in social psychology, nor does it mention evidence that Bem (2011) used QRPs (Francis, 2012; Schimmack, 2012).
Sander L. Koole and Daniël Lakens (2012) PoPS
Jeff Galak, Robyn A. LeBoeuf, Jeff Galak, & Leif D. Nelson (2012). JPSP
suggest that QRPS were used, but do not cite John et al. (2012); also do not cite Schimmack (2012) or Francis (2012) as evidence that QRPs were used.
cite John et al. (2012) and Schimmack (2012) but do not cite Francis or Schimmack as evidence that Bem used QRPs.
Andrew J. Vonasch and Roy F. Baumeister (2013) British JSP
Does not mention John et al. (2012) and does not cite Francis or Schimmack as evidence that Bem used QRPs even though the second author was reviewer of Schimmack (2012)
Joachim I. Krueger, David Freestone, Mika L.M. MacInnis (2013) New Ideas
does not mention QRPs (John et al., 2012) or that Bem used QRPs (Francis, Schimmack, 2012)
mention John et al. (2012) and imply use of QRPs, cite Schimmack (2012), but do not cite Schimmack or Francis as evidence that Bem used QRPs.
Jonathan W. Schooler (2015) PoPS
Does not mention that Francis or Schimmack showed Bem (2011) used QRPs, although he was a reviewer of the Schimmack (2012) article.
Eli J. Finkel, Paul W. Eastwick, and Harry T. Reis
Cite John et al., Schimmack, and Francis, but not to point out that Bem used QRPs to get his results.
Brian D.Earp & DavidTrafimow, 2015, Frontiers in Psychology
cite John et al. (2012), but do not cite Francis or Schimmack as evidence that Bem used QRPs.
Klaus Fiedler (2016) SOCIAL PSYCHOLOGY OF MORALITY
cites John et al.’s (2012) to criticize it. does not cite Francis or Schimmack as evidence that Bem used QRPs.
Joachim Hüffmeier, Jens Mazei, & Thomas Schultze (2016) JESP
do cite John et al. (2012) for QRPs, do not cite Francis; cite Schimmack (2012), but not for evidence that Bem (2011) used QRPs.
does not cite John et al. (2012) for QRPs, does not cite Francis or Schimmack (2012) for presence of QRPs in Bem’s (2011) article.
mention QRPs (John et al., 2012) and describe Bem’s results as questionable, but do not mention that Bem used QRPs.
Mark Rubin (2017) Review of General Psychology
mentions QRPs (John, 2012) and labels Bem’s research as questionable, but does not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.
Roger Giner-Sorolla (2019) European Review of Social Psychology
does not cite John et al., but does cite Schimmack (2012) as evidence that Bem “almost certaintly” used QRPs.
Lee Jussim, Jon A. Krosnick, Sean T. Stevens & Stephanie M. Anglin (2019) Psy Belgica
cite John et al. (2012), but do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs. However, they cite Schimmack (2017) as evidence that social psychology has “dramatically improved”
Jonathan Baron & John T. Jost (2019) PoPS
do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.
Gandalf Nicolas, Xuechunzi Bai, & Susan T. Fiske (2019). PoPS
do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) for evidence that Bem used QRPs.
do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.
Conclusion
Bem (2011) presented incredible results that seemed to provide strong empirical evidence (p < .05^9). The article was published because it passed “a rigorous review process, involving a large set of extremely thorough reviews by distinguished experts in social cognition” and “we can only take the author at his word that his data are in fact genuine.” The same is true for all other articles published in social psychology. A few significant p-values, peer-review, and trust are supposed to ensure that results are robust, replicable. However, the replication crisis has shown that this is not the case. As research practices have not dramatically changed, consumers of social psychology need to be warned that even 10 years after Bem’s article published results in social psychology are no more trustworthy than Bem’s claims of extra-sensory perception.