Social Psychologists’ (Non) Response to Bem (2011)

A science that cannot face its history has no future. (Anonymous).

ABSTRACT

Bem (2011) presented incredible results that seemed to provide strong empirical evidence (p < .05^9). The article was published because it passed “a rigorous review process, involving a large set of extremely thorough reviews by distinguished experts in social cognition” and “editors can only take the author at his word that his data are in fact genuine.” Here I show that social psychologists have avoided discussing the broader implications of the method crisis in social psychology. The same standards that were used for Bem’s article are still used for most articles published today: a few significant p-values, peer-review, and trust that researchers are honest are supposed to ensure that results are robust and replicable. However, the replication crisis has shown that this is not the case. Consumers of social psychology need to be aware that even 10 years after Bem’s infamous article, evidence for social psychological theories is no more credible than Bem’s evidence for extra-sensory perception.

Introduction

Daryl Bem was a highly respected social psychologist, until he published his “Feeling the Future” article in 2011.

The article became a poster child of everything that is wrong with research methods in social psychology and has been cited over 300 times.

The article was also accompanied by an editorial that justified the publication of an article that seemed to provide evidence for the incredible claim that humans, or at least extraverts, can feel events that haven’t happened yet.

The editorial suggests that Bem’s article will “stimulate further thoughts about appropriate methods in research on social cognition and attitudes” (p. 406). Ten years later, we can see whether publishing Bem’s article had the intended effect.

The high citation count shows that the article did indeed generate lot’s of discussion about research practices in social cognition research and social psychology more broadly. However, an inspection of these citations shows that most of this discussion occurred outside of social psychology, by meta-psychologists who reflected on research practices by social psychologists. In stark contrast, critical self-reflection by social psychologists is insignificant.

Here I provide a bibliography of these citations. An examination of these citations shows that social psychologists have carefully avoided asking themselves the most important question that follows from Bem’s (2011) article.

If we cannot trust Bem’s article that reported nine statistically significant results, which article in social psychology can we trust?

Etienne P. LeBel & Kurt R. Peters (2011) Review of General Psychology

This article clearly spells out the problems of QRPs and uses Bem’s article to raise questions about other research findings. The first author was trained as a graduate student by Gawronski, but is no longer in social psychology.

Charles M. Judd, Jacob Westfall, & David A. Kenny (2012) JPSP

This article implies that the problem was inappropriate treatment of variation across stimuli. It does not mention the use of QRPs in social psychology, nor does it mention evidence that Bem (2011) used QRPs (Francis, 2012; Schimmack, 2012).

Sander L. Koole and Daniël Lakens (2012) PoPS

Jeff Galak, Robyn A. LeBoeuf, Jeff Galak, & Leif D. Nelson (2012). JPSP

suggest that QRPS were used, but do not cite John et al. (2012); also do not cite Schimmack (2012) or Francis (2012) as evidence that QRPs were used.

Jens B. Asendorpf, Mark Conner, Filip De Fruyt, Jan de Houwer, Jaap J. A. Denissen, Klaus Fiedler, Susann Fiedler, David C. Funder, Reinhold, Kliegel, Brian A. Nosek, Marco Perugini, Brent. W. Roberts, Manfred Schmitt, Marcel A. G. Van Aken, Hannelore Weber, & Jelte M. Wicherts (2013). EJPers

cite John et al. (2012) and Schimmack (2012) but do not cite Francis or Schimmack as evidence that Bem used QRPs.

Andrew J. Vonasch and Roy F. Baumeister (2013) British JSP

Does not mention John et al. (2012) and does not cite Francis or Schimmack as evidence that Bem used QRPs even though the second author was reviewer of Schimmack (2012)

Joachim I. Krueger, David Freestone, Mika L.M. MacInnis (2013) New Ideas

does not mention QRPs (John et al., 2012) or that Bem used QRPs (Francis, Schimmack, 2012)

David C. Funder, John M. Levine, Diane M. Mackie, Carolyn C. Morf, Carol Sansone, Simine Vazire, and Stephen G. West (2014), PSPR

mention John et al. (2012) and imply use of QRPs, cite Schimmack (2012), but do not cite Schimmack or Francis as evidence that Bem used QRPs.

Jonathan W. Schooler (2015) PoPS

Does not mention that Francis or Schimmack showed Bem (2011) used QRPs, although he was a reviewer of the Schimmack (2012) article.

Eli J. Finkel, Paul W. Eastwick, and Harry T. Reis

Cite John et al., Schimmack, and Francis, but not to point out that Bem used QRPs to get his results.

Brian D.Earp & DavidTrafimow, 2015, Frontiers in Psychology

cite John et al. (2012), but do not cite Francis or Schimmack as evidence that Bem used QRPs.

Klaus Fiedler (2016) SOCIAL PSYCHOLOGY OF MORALITY

cites John et al.’s (2012) to criticize it. does not cite Francis or Schimmack as evidence that Bem used QRPs.

Joachim Hüffmeier, Jens Mazei, & Thomas Schultze (2016) JESP

do cite John et al. (2012) for QRPs, do not cite Francis; cite Schimmack (2012), but not for evidence that Bem (2011) used QRPs.

Mark Schaller (2016) JESP

does not cite John et al. (2012) for QRPs, does not cite Francis or Schimmack (2012) for presence of QRPs in Bem’s (2011) article.

Matt Motyl, Alexander P. Demos, Timothy S. Carsel, Brittany E. Hanson, Zachary J. Melton,
Allison B. Mueller, J. P. Prims, Jiaqing Sun, Anthony N. Washburn, Kendal M. Wong, Caitlyn Yantis, and Linda J. Skitka

mention QRPs (John et al., 2012) and describe Bem’s results as questionable, but do not mention that Bem used QRPs.

Mark Rubin (2017) Review of General Psychology

mentions QRPs (John, 2012) and labels Bem’s research as questionable, but does not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.

Roger Giner-Sorolla (2019) European Review of Social Psychology

does not cite John et al., but does cite Schimmack (2012) as evidence that Bem “almost certaintly” used QRPs.

Lee Jussim, Jon A. Krosnick, Sean T. Stevens & Stephanie M. Anglin (2019) Psy Belgica

cite John et al. (2012), but do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs. However, they cite Schimmack (2017) as evidence that social psychology has “dramatically improved”

Jonathan Baron & John T. Jost (2019) PoPS

do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.

Gandalf Nicolas, Xuechunzi Bai, & Susan T. Fiske (2019). PoPS

do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) for evidence that Bem used QRPs.

Zachary G. Baker, Ersie-Anastasia Gentzis, Emily M. Watlington, Sabrina Castejon, Whitney E. Petit1, Maggie Britton, Sana Haddad, Angelo M. DiBello, Lindsey M. Rodriguez, Jaye L. Derrick, C. Raymond Knee (2020). Personal Relationships

do not cite John et al. (2012) and do not cite Francis or Schimmack (2012) as evidence that Bem used QRPs.

Conclusion

Bem (2011) presented incredible results that seemed to provide strong empirical evidence (p < .05^9). The article was published because it passed “a rigorous review process, involving a large set of extremely thorough reviews by distinguished experts in social cognition” and “we can only take the author at his word that his data are in fact genuine.” The same is true for all other articles published in social psychology. A few significant p-values, peer-review, and trust are supposed to ensure that results are robust, replicable. However, the replication crisis has shown that this is not the case. As research practices have not dramatically changed, consumers of social psychology need to be warned that even 10 years after Bem’s article published results in social psychology are no more trustworthy than Bem’s claims of extra-sensory perception.

Leave a Reply