The danger of bias is probably greatest when textbook writers write about their own research. Gilovich, Keltner, Chen, and Nisbett (2019; Social Psychology, 5th edition) could have chosen many research topics to illustrate how social psychologists conduct research, but they chose Nisbett’s work on the culture of honor.

We tie the methods of social psychology together by showing how many of them can be applied to a single problem: the nature of the “culture of honor.” (p. x).

The authors further suggest that this chapter is “oriented toward providing the critical thinking skills that are the hallmark of social psychology” (p. x).

We show how the tools of social psychology can be used to critique research in the behavioral and medical sciences that students encounter online and in magazines and newspapers.

Importantly, they do not promise to provide tools to think critically about social psychology or about studies presented in their textbook. Presumably, they are flawless.

CHAPTER 2

Chapter 2 starts with field experiments by Cohen and Nisbett (1997). The authors sent fictitious job applications to business owners in the North and the South of the United States. The applicant admitted that he had been convicted of a felony. The textbook describes the results as follows.

Cohen and Nisbett found some distinct patterns in the replies. Retailers from the South complied with the applicant’s requests more than reatilers from the North. And the notes from Southern business owners were much warmer and more sympathetic than those from the North.

The table from the original article shows that some of these differences were not statistically significant.

This would provide an opportunity to teach students about research practices in social psychology. How should a p-value of .06 be interpreted? Would the study have been published if all of the p-values were greater than .10?

The next example illustrates the use of surveys.

Nisbett and Cohen (1996) used surveys to try to find out why U.S. Southerners were more likely to commit homicide.

Southerners were more likely to favor violence in response to insults and to think that a man would be justified to fight an acquaintance who “looks over his girlfriend and talks to her in a suggestive way.”

No further information is provided how many homicides are due to insults or flirting or whether incidence rates for these types of homicides differ between the North and the South. The reason is probably that social psychologists rarely conduct correlational studies and do not value these types of studies.

The best way to be sure about causality is to conduct an experiment (p. 47)

The power of experiments is illustrated with Cohen, Nisbett, Bowdle, & Schwarz’s (1996) studies of Northerners and Southerns reactions to an insult under standardized laboratory conditions (fortunately, no homicides were committed).

Participants were randomly assigned to an “Insult” and a control condition. In the insult condition, a confederate of the experimenter had to make room for the participant and acted annoyed; that is, he slammed a file-drawer shut and called the participant an A**hole.

The textbook describes the results as follows.

First, observers noted the participants’ immediate reactions after the insult. Insulted Southerners usually showed a flash of anger; insulted Northerners were more likely to shrug their shoulders or to appear amused.

Second, participants were asked to read a story in which a man made a pass at another man’s fiancee and then to provide an ending to the story. Southerners who had been insulted were much more likely to provide a violent ending to the story than Southerners who hadn’t been insulted, whereas the endings provided by Northerners were unaffected by the insult.

Third, the participants’ level of testosterone, the hormone that mediates aggression in males, was tested both before and after the insult. The level of testosterone increased for Southerners who had been insulted, but it did not increase for Southerners who hadn’t been insulted or for Northerners, whether insulted or not.

Forth, participants were asked to walk back down the narrow hallway, and this time another assistant to the experimenter walked toward the participant. The assistant was very tall and muscular and his instructions were to talk down the middle of the hall, forcing the participant to doge out of his way. The dependent variable was how far away the participant was when he finally swerved out of the assistant’s way. The investigators thought that the insulted Southerners would be put into such an aggressive mood that they would play “chicken” with the assistant, waiting until the last moment to swerve aside. And indeed they did. Northerners, whether insulted or not, served aside at a distance of about 5 feet (1.4 meters) from the assistant. Southerners, who are known for their politeness, stood aside at around 9 feet (2.75 meters) if not insulted, but pushed ahead until 3 feet away (less than 1 meter) if they had been insulted.

One could not imagine a more perfect result; four dependent variables all confirmed the authors’ expectations. Unfortunately, this is exactly what students are reading. An authors’ idealized recollection of studies with a lot more dependent variables and more mixed results.

Fact Check 1. Insulted Southerners usually showed a flash of anger. 

Study 1
Southern participants tended to be more angry than northern participants (northern M = 2.34, southern M= 3.05), F(41) = 1.61, .10 < p < .15.
[A result with p > .10 is very rarely presented as marginally significant].
Experiment 2 (as well as Experiment 3, which is reported subsequently) yielded weak and inconsistent results regarding the emotional reaction to the bump

In short, none of the three studies showed a significant result supporting the textbook claim.

Fact Check 2.  Insulted Southerners’ Violent Ending

To examine the interaction between region and insult, we performed an analysis of variance (ANOVA) on a three-level variable (no violence, violence suggested, actual violence). Higher numbers indicated greater violence, and means were: southern insult = 2.30, southern control =
1.40, northern insult = 1.73, and northern control = 2.05, interaction
F( 1,78) = 7.65, p<. 005.

This result was not replicated in Study 2 with a different scenario.

There was no effect for region, insult, or the interaction on
whether participants expected the ambiguous scenarios to end
with either physical or verbal aggression (all Fs < 1).

Fact Check 3. Hormone Levels

Hormones were only assessed in Study 2. Cortisol and testosterone showed just significant effects in an analysis with planned contrasts.

It is only after provocation that we expected southerners to show cortisol increases over the level of northerners and over the level of control groups. The appropriate contrast to test this prediction is +3, – 1 , – 1 , – 1 . This contrast— indicating that the effect of the insult was seen only for
southerners, not for northerners—described the data well and
was significant, t( 165) = 2.14, p < .03

As may be seen in Figure 2, testosterone levels rose 12% for insulted southerners and 4% for control southerners. They rose 6% for insulted northerners and 4% for control northerners. Again, we used the +3, – 1 , – 1 , -1 contrast indicating that change was expected only for insulted southerners. The contrast was significant at p < .03, /(165) = 2.19.’ *

Fact Check 4. Chicken Game

The +3, – 1 , – 1 , – 1 interaction contrast was significant, p < .001, t(142) = 3.45. 

The chicken game is the only finding with a clear statistical result. However, even results like this need to be replicated to be convincing, especially if many dependent variables were used.

The authors also comment on the dependent variables that did not produce significant results that are not mentioned in the textbook.

It also is important to note that there were several measures—
the neutral projective hostility tasks of Experiment 1,
the ambiguous insult scenarios of Experiment 2, the shockacceptance
measure of Experiment 2, and the masculine protest
items of Experiment 3—on which northerners and southerners
were nor differentially affected by the insult. These null
results suggest that the insult did not create a generalized hostility
or perceived threat to self that colored everything southern
participants did or thought. Measures that were irrelevant
or ambiguous with respect to issues of affront and status,
that were uninvolving because they were paper-and-pencil,
and that were ecologically unnatural did not show an effect
of the insult. Instead, the effect of the affront was limited to
situations that concerned issues of honor, were emotionally
involving, and had actual consequences for the participant’s
masculine status and reputation.

Here the authors make the mistake to interpret patterns of significant findings and non-significant findings without testing whether these findings are significantly different from each other. Given how weak some of the significant results are, this is unlikely.

Chapter 2 ends with a section on replication (see Schimmack, 2018). In this section, the authors emphasize the importance of replication studies.

One of the ways in which science is different from other modes of inquiry is the importance placed on replication. (p. 54).

However, the textbook finding does not mention any replication studies of Cohen et al.’s findings. An article from 2014 (The Lost Cause? Examining the Southern Culture of Honor Through Defensive Gun Use) lists three articles by Cohen and colleagues as evidence from ethnographic experiments.

Closer inspection of the 1998 and 1999 articles shows that they were not experiments, but survey studies. Thus, to the best of my knowledge, the featured experiments were never replicated to see whether the results can be reproduced.

Conclusion 

The textbook gives the impression that clear results from an experiment provide important information about the causes of differences in homicide rates between the North and the South of the USA. Closer inspection shows that the results are far from clear. In addition, they provide only circumstantial evidence regarding the causes of homicide. The ability of laboratory experiments to illuminate causes of real-world phenomena like homicides is exaggerated. Even if Southerners respond with more aggression to insults, it does not mean that they are more willing to kill in these situations.

Ironically, the choice of this study to illustrate methods in social psychology is instructive. Social psychologists like to tell interesting stories and use data when they fit the story to give them the allure of being scientific. When the data do not fit the story, they are usually not reported. Students who received any introduction to the scientific method may realize that this is not how science works.