2021 Replicability Report for the Psychology Department at the University of Michigan


Since 2011, it is an open secret that many published results in psychology journals do not replicate. The replicability of published results is particularly low in social psychology (Open Science Collaboration, 2015).

A key reason for low replicability is that researchers are rewarded for publishing as many articles as possible without concerns about the replicability of the published findings. This incentive structure is maintained by journal editors, review panels of granting agencies, and hiring and promotion committees at universities.

To change the incentive structure, I developed the Replicability Index, a blog that critically examined the replicability, credibility, and integrity of psychological science. In 2016, I created the first replicability rankings of psychology departments (Schimmack, 2016). Based on scientific criticisms of these methods, I have improved the selection process of articles to be used in departmental reviews.

1. I am using Web of Science to obtain lists of published articles from individual authors (Schimmack, 2022). This method minimizes the chance that articles that do not belong to an author are included in a replicability analysis. It also allows me to classify researchers into areas based on the frequency of publications in specialized journals. Currently, I cannot evaluate neuroscience research. So, the rankings are limited to cognitive, social, developmental, clinical, and applied psychologists.

2. I am using department’s websites to identify researchers that belong to the psychology department. This eliminates articles that are from other departments.

3. I am only using tenured, active professors. This eliminates emeritus professors from the evaluation of departments. I am not including assistant professors because the published results might negatively impact their chances to get tenure. Another reason is that they often do not have enough publications at their current university to produce meaningful results.

Like all empirical research, the present results rely on a number of assumptions and have some limitations. The main limitations are that
(a) only results that were found in an automatic search are included
(b) only results published in 120 journals are included (see list of journals)
(c) published significant results (p < .05) may not be a representative sample of all significant results
(d) point estimates are imprecise and can vary based on sampling error alone.

These limitations do not invalidate the results. Large difference in replicability estimates are likely to predict real differences in success rates of actual replication studies (Schimmack, 2022).

University of Michigan

I used the department website to find core members of the psychology department. I counted 55 professors and 11 associate professors, which makes it one of the largest departments in North America. Not all researchers conduct quantitative research and report test statistics in their result sections. I limited the analysis to 29 professors and 5 associate professors who had at least 100 test statistics.

Figure 1 shows the z-curve for all 12,365 tests statistics in articles published by these 19 faculty members. I use the Figure to explain how a z-curve analysis provides information about replicability and other useful meta-statistics.

1. All test-statistics are converted into absolute z-scores as a common metric of the strength of evidence (effect size over sampling error) against the null-hypothesis (typically H0 = no effect). A z-curve plot is a histogram of absolute z-scores in the range from 0 to 6. The 1,781 z-scores greater than 6 are not shown because z-scores of this magnitude are extremely unlikely to occur when the null-hypothesis is true (particle physics uses z > 5 for significance). Although they are not shown, they are included in the meta-statistics.

2. Visual inspection of the histogram shows a steep drop in frequencies at z = 1.96 (dashed blue/red line) that corresponds to the standard criterion for statistical significance, p = .05 (two-tailed). This shows that published results are selected for significance. The dashed red/white line shows significance for p < .10, which is often used for marginal significance. Thus, there are more results that are presented as significant than the .05 criterion suggests.

3. To quantify the amount of selection bias, z-curve fits a statistical model to the distribution of statistically significant results (z > 1.96). The grey curve shows the predicted values for the observed significant results and the unobserved non-significant results. The full grey curve is not shown to present a clear picture of the observed distribution. The statistically significant results (including z > 6) make up 41% of the total area under the grey curve. This is called the expected discovery rate because the results provide an estimate of the percentage of significant results that researchers actually obtain in their statistical analyses. In comparison, the percentage of significant results (including z > 6) includes 72% of the published results. This percentage is called the observed discovery rate, which is the rate of significant results in published journal articles. The difference between a 72% ODR and a 41% EDR provides an estimate of the extent of selection for significance. The difference of~ 30 percentage points is fairly large, but other departments have even bigger discrepancies. The upper level of the 95% confidence interval for the EDR is 57%. Thus, the discrepancy is not just random. To put this result in context, it is possible to compare it to the average for 120 psychology journals in 2010 (Schimmack, 2022). The ODR is similar (both 72%), but the EDR is higher (41% vs. 28%).

4. The EDR can be used to estimate the risk that published results are false positives (i.e., a statistically significant result when H0 is true), using Soric’s (1989) formula for the maximum false discovery rate. An EDR of 41% implies that no more than 8% of the significant results are false positives, however the lower limit of the 95%CI of the EDR, 28%, allows for 13% false positive results. Most readers are likely to agree that this is too high. One solution to this problem is to lower the conventional criterion for statistical significance (Benjamin et al., 2017). Figure 2 shows that alpha = .01 reduces the point estimate of the FDR to 3% with an upper limit of the 95% confidence interval of 5%. Thus, without any further information readers could use this criterion to interpret results published in articles by U Michigan faculty members.

Replicability varies across disciplines. Before 2015 , replicability was particularly low in social psychology (Schimmack, 2021). This difference is also visible in separate analyses of researches from different fields at the University of Michigan. For the 9 social psychologists with sufficient data, the EDR is only 27% , which also implies a higher false positive risk of 14%, but these point estimates have wide confidence intervals.

There were only six cognitive researchers with usable data. Their z-curve shows less selection bias and a higher EDR estimate than the EDR estimate for social psychologists. The difference between social and cognitive psychologists is significant (i.e., the 95%CI do not overlap).

However, it is notable that the z-curve overestimates the number of z-scores that are just significant (z = 2 to 2.2), while it underestimates the percentage of z-scores between 2.2 and 2.4. This may suggests that the selection model is wrong and that sometimes just significant p-values are not published. A sensitivity (or multiverse) analysis can use different selection models. Using only z-scores about 2.2 (the vertical blue dotted line in the figure below), doesn’t change the ERR estimate much, but the EDR estimate is considerably lower, 43%, and the 95%CI goes as low as 18%. This leads to higher false discovery risks with an upper limit of the 95%Ci of 24%. Caution would therefore suggest to be careful with p-values greater than .01.

University of Michigan has a large group of developmental psychologists. The 9 faculty with usable data provided 4,382 test statistics. The results are better than those for social psychology, but not as good as those for cognitive psychology when the standard selection criterion is used. These results are consistent with typical differences between these disciplines that are reflected in analyses of 120 psychology journals (Schimmack, 2022).

Most of the faculty at U Michigan are full professors. Only 5 associate professors provided sufficient data for a z-curve analysis. The total number of test-statistics was 1,100.

The z-curve shows no evidence that research practices of associate professors are different from those of full professors.

Another way to look at research practices is to limit the analysis to articles published since 2016, which is the first year in which some journals show an increase in replicability (Schimmack, 2022). However, there is no notable difference to the z-curve for all years. This is in part due to the relative (not absolute) good performance of University of Michigan. Other departments have a lot more room for improvement.

The table below shows the meta-statistics of all 19 faculty members. You can see the z-curve for each faculty member by clicking on their name.

1Patricia A. Reuter-Lorenz7881762
2Daniel H. Weissman7479692
3John Jonides7376702
4William J. Gehring7275682
5Cindy Lustig6973663
6Henry M. Wellman6973653
7Terri D. Conley6871643
8Allison Earl6572574
9Terry E. Robinson6569613
10Arnold K. Ho6572574
11Felix Warneken6365604
12Stephanie A. Fryberg6265604
13Twila Tardif6265584
14Martin F. Sarter6064564
15Ashley N. Gearhardt5774418
16Thad A. Polk54763211
17Ethan Kross5264408
18Susan A. Gelman52782615
19Julie E. Boland5163398
20Shinobu Kitayama51713211
21David Dunning50683211
22Christopher S. Monk47742021
23Priti Shah44701824
24Patricia J. Deldin43523410
25Robert M. Sellers42632120
26Brenda L. Volling40483311
27Joshua M. Ackerman40621726
28Abigail J. Stewart37492615
29Nestor L. Lopez-Duran33521434
30Kent C. Berridge32501433
31Sheryl L. Olson32511237
32Denise Sekaquaptewa31521046
33Fiona Lee2542956

Leave a Reply