Rickard Carlsson and I submitted a letter to the Proceedings of the National Academy of Sciences. The format allows only 500 words (PDF). Here is the long version of our concerns about Johnson et al.’s PNAS article about racial disparities in police shootings. An interesting question for meta-psychologists is how the authors and reviewers did not catch an error that led to the implausible result that police officers are six times more likely to shoot White civilians than Black civilians when they felt threatened by a civilian.

Police Officers are not Six Times more Likely to Shoot White Civilians than Black Civilians: A Coding Error in Johnson et al. (2019)

Ulrich Schimmack Rickard Carlsson
University of Toronto, Mississauga Lineaus University

The National Academy of Sciences (NAS) was founded in 1863 by Abraham Lincoln to provide independent, objective advice to the nation on matters related to science and technology (1).  In 1914, NAS established the Proceedings of the National Academy of Sciences (PNAS) to publish scientific findings of high significance.  In 2019, Johnson, Tress, Burke, Taylor, and Cesario published an article on racial disparities in fatal shootings by police officers in PNAS (2).  Their publication became the topic of a heated exchange in the Oversight Hearing on Policing Practices in the House Committee on the Judiciary on September 19, 2019. Heather Mac Donald cited the article as evidence that there is no racial disparity in fatal police shootings. Based on the article, she also claimed “In fact, black civilians are shot less, compared with whites, than their rates of violent crime would predict” (3). Immediately after her testimony, Phillip Atiba Goff challenged her claims and pointed out that the article had been criticized (4). In a rebuttal, Heather MacDonald cited Johnson from the authors response that the authors stand by their finding (5).  Here we show that the authors’ conclusions are based on a statistical error in their analyses.

The authors relied on the Guardian’s online database about fatal use of force (7). The database covers 1,146 incidences in 2015.  One aim of the authors’ research was to examine the influence of officers’ race on the use of force. However, because most officers are White, they only found 12 incidences (N = 12, 5%) where a Black citizen was fatally shot by a Black officer. This makes it impossible to examine statistically reliable effects of officers’ race.  In addition, the authors examined racial disparities in fatal shootings with regression models that related victims’ race to victims, officers, and counties’ characteristics. The results showed that “a person fatally shot by police was 6.67 times less [italics added] likely (OR = 0.15 [0.09, 0.27]) to be Black than White” (p. 15880).  This finding would imply for every case of a fatal use of force with a Black citizen like Eric Garner or Tamir Rice, there should be six cases similar cases with White citizens.  The authors explain this finding with depolicing; that is, officers may be “less likely to fatally shoot Black civilians for fear of public and legal reprisal” (p. 15880).  The authors also conducted several additional analyses that are reported in their supplementary materials.  However, they claim that their results are robust and “do not depend on which predictors are used” (p. 15881). We show that all of these statements are invalidated by a coding mistake in their statistical model.

Table 1
Racial Disparity in Race of Fatally Shot Civilians

Model	County Predictor	Odds-Ratio (Black/White), 95%CI
M1	Homicide Rates	0.31 (0.23, 0.42)
M2	Population Rates	2.03 (1.21, 3.41)
M3	Population & Homicide Rates	0.89 (0.44, 1.80)

The authors did not properly code categorical predictor variables. In a reply, the authors acknowledge this mistake and redid the analyses with proper weighted effect coding of categorical variables. Their new results are reported in Table. 1   The correct results show that the choice of predictor variables does have a strong influence on the conclusions.  In a model that only uses homicide rates as predictor (M1), the intercept still shows a strong anti-White bias, with 3 White civilians being killed for every 1 Black civilian in a county with equal proportions of Black and White citizens. In the second model with population proportions as predictor, the data show anti-Black bias. When both predictors are used, the data show parity, but with a wide margin of error that ranges from a ratio of 2 White civilians for 1 Black civilian to 2 Black civilians for 1 Black civilian.  Thus, after correcting the statistical mistake, the results are no longer consistent and it is important to examine which of these models should be used to make claims about racial disparities.

We argue that it is necessary to include population proportions in the model.  After all, there are many counties in the dataset with predominantly White populations and no shootings of Black civilians. This is not surprising. For officers to encounter and fatally shoot a Black resident, there have to be Black civilians. To ignore the demographics would be a classic statistical mistake that can lead to false conclusions, such as the famous example that is used to teach the difference between correlation and causation. In this example, it appears as if Christians commit more homicides because homicide rates are positively correlated with the number of churches. This inference is wrong because the correlation between churches and homicides simply reflects the fact that counties with a larger population have more churches and more homicides.  Thus, the model that uses only population ratios as predictor is useful because it tells us whether White or Black people are shot more often than we would expect if race was unrelated to police shootings. Consistent with other studies, including an article by the same authors, we see that Black citizens are shot disproportionally more often than White citizens (8,9).

The next question that a scientific study of police shootings can examine is why there exist racial disparities in police shootings.  Importantly, answering this question does not make racial disparities disappear. Even if Black citizens are shot more often because they are more often involved in crimes, as the authors claim, there exists a racial disparity.  It didn’t disappear, nor does this explanation account for incidences like the death of Eric Garner or Tamir Rice.  However, the authors’ conclusion that “racial disparity in fatal shootings is explained by non-Whites’ greater exposure to the police through crime” (p. 15881) is invalid for several reasons.

First of all, the corrected results for the model that takes homicide rates and population rates into account no longer provides conclusive evidence about racial disparities. The data still allow for a racial disparity where Black civilians are shot at twice the rate as White civilians.  Moreover, this model ignores the authors’ own finding that victims’ age is a significant predictor of victims’ race.  Parity is obtained for the average age of 37, but the age effect implies that 20-year old victims are significantly more likely to be Black, OR(B/W) = 3.26, 95%CI = 1.26 to 8.43 while 55-year old victims are significantly more likely to be White, OR(B/W) = 0.24, 95%CI = 0.08 to 0.71.  Thus, even when homicide rates are included in the model, the authors’ data are consistent with the public perception that officers are more likely to use force with young Black men than with young White men.

The second problem is that the model does not include other potentially relevant predictor variables, such as poverty rates, and that an analysis across counties is unable to distinguish between actual and spurious predictors because all statistics are highly correlated with counties’ demographics (r > .9).

A third problem is that it is questionable to rely on statistics about homicide victims as a proxy for police encounters. The use of homicide rates implies that most victims of fatal use of force are involved in homicides. However, the incidences in the Guardian database show that many victims were involved in less severe crimes.

Finally, it is still possible that there is racial disparity in unnecessary use of force even if fatal incidences are proportional to violent crimes. If police encounter more Black people in ambiguous situations because Black people are disproportionally more involved in violent crime, they would still accidentally shoot more Black citizens than White citizens. It is therefore important to distinguish between racial bias of officers and racial disparities in fatal incidences of use of force.  Racial bias is only one of several factors that can produce racial disparities in the use of excessive force.

Conclusion

During a hearing on policing practices in the House Committee on the Judiciary, Heather MacDonald cited Johnson et al.’s (2019) article as evidence that crime accounts for racial disparities in the use of lethal force by police officers and that “black civilians are shot less, compared with whites, than their rates of violent crime would predict.” Our analysis of Johnson et al.’s (2019) article shows that these statements are to a large extent based on a statistical error.  Thus, the article cannot be used as evidence to claim that there are no racial disparities in policing or as evidence that police officers are even more reluctant to use excessive force with Black suspects than with White civilians.  The only lesson that we can learn from this article is that social scientists make mistakes and that pre-publication peer-review alone does not ensure that these mistakes are caught and corrected. It is puzzling how the authors and reviewers did not detect a statistical mistake when the results implied that police officers fatally shoot 6 White suspects for every Black suspect. It was this glaring finding that made us conduct our own analyses and to detect the mistake. This shows the importance of post-publication peer review to ensure that scientific information that informs public policy is as objective and informative as it can be

References

1. National Academy of Sciences.  Mission statement of the (http://www.nasonline.org/about-nas/mission/)

2. Johnson, D. J., Trevor T., Nicole, B., Carley, T., & Cesario, J. (2019). Officer characteristics and racial disparities in fatal officer-involved shootings. Proceedings of the National Academy of Sciences, 116(32), 15877–15882.

3. MacDonald, H. (2019). False Testimony, https://www.city-journal.org/police-shootings-racial-bias

4. Knox, D. & Mummolo, J. (2019). Making inferences about racial disparities in police violence. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3431132

5. Johnson, D. J., & Cesario, J. (2019). Reply to Knox and Mummolo: Critique of Johnson et al. (2019). https://psyarxiv.com/dmhpu/

6. Johnson, D. J., & Cesario, J. (2019). Reply to Schimmack: Critique of Johnson et al. (2019).

7. “The counted.” The Guardian. https://www.theguardian.com/us-news/ng-interactive/2015/jun/01/the-counted-police-killings-us-database#

8. J. Cesario, D. J. Johnson, W. Terrill, Is there evidence of racial disparity in police use of deadly force? Analyses of officer-involved fatal shootings in 2015–2016. Soc. Psychol. Personal. Sci. 10, 586–595 (2018).

9. Edwards, F., Lee, H., Esposito, M. (2019). Risk of being killed by police use of force in the United States by age, race-ethnicity, and sex. Proceedings of the National Academy of Sciences, 116(34), 16793-16798. doi: 10.1073/pnas.1821204116