Introduction
Since 2011, it is an open secret that many published results in psychology journals do not replicate. The replicability of published results is particularly low in social psychology (Open Science Collaboration, 2015).
A key reason for low replicability is that researchers are rewarded for publishing as many articles as possible without concerns about the replicability of the published findings. This incentive structure is maintained by journal editors, review panels of granting agencies, and hiring and promotion committees at universities.
To change the incentive structure, I developed the Replicability Index, a blog that critically examined the replicability, credibility, and integrity of psychological science. In 2016, I created the first replicability rankings of psychology departments (Schimmack, 2016). Based on scientific criticisms of these methods, I have improved the selection process of articles to be used in departmental reviews.
1. I am using Web of Science to obtain lists of published articles from individual authors (Schimmack, 2022). This method minimizes the chance that articles that do not belong to an author are included in a replicability analysis. It also allows me to classify researchers into areas based on the frequency of publications in specialized journals. Currently, I cannot evaluate neuroscience research. So, the rankings are limited to cognitive, social, developmental, clinical, and applied psychologists.
2. I am using department’s websites to identify researchers that belong to the psychology department. This eliminates articles that are from other departments.
3. I am only using tenured, active professors. This eliminates emeritus professors from the evaluation of departments. I am not including assistant professors because the published results might negatively impact their chances to get tenure. Another reason is that they often do not have enough publications at their current university to produce meaningful results.
Like all empirical research, the present results rely on a number of assumptions and have some limitations. The main limitations are that
(a) only results that were found in an automatic search are included
(b) only results published in 120 journals are included (see list of journals)
(c) published significant results (p < .05) may not be a representative sample of all significant results
(d) point estimates are imprecise and can vary based on sampling error alone.
These limitations do not invalidate the results. Large difference in replicability estimates are likely to predict real differences in success rates of actual replication studies (Schimmack, 2022).
Department Rankings
The main results of the replicability analysis are included in this table. Detailed analyses of departments and faculty members can be found by clicking on the hyperlink of a university.
The table is sorted by the all time actual replication prediction (ARP). It is easy to sort the table by other meta-statistics.
The ERR is the expected replication rate that is estimated based on the average power of studies with significant results (p < .05).
The EDR is the expected discovery rate that is estimated based on the average power of studies before selection for significance. It is estimated using the distribution of significant p-values converted into z-scores.
Bias is the discrepancy between the observed discovery rate (i.e., the percentage of significant results in publications) and the expected discovery rate. Bias reflects the selective reporting of significant results.
The FDR is the false discovery risk. It is estimated using Soric’s formula that converts the expected discovery rate into an estimate of the maximum percentage of false positive results under the assumption that true hypothesis are tested with 100% power.
For more information about these statistics, please look for tutorials or articles on z-curve on this blog.
| University | ARP-All | ERR-ALL | EDR-ALL | Bias-All | FDR-All | ARP-5Y | ERR-5Y | EDR-5Y | Bias-5Y | FDR-5Y | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| University of Michigan | 55 | 69 | 41 | 31 | 8 | 58.5 | 72 | 45 | 27 | 6 | ||
| Western University | 54.5 | 70 | 39 | 29 | 8 | 73.5 | 77 | 70 | 1 | 2 | ||
| University of Toronto | 54 | 67 | 41 | 28 | 8 | 56 | 69 | 43 | 24 | 7 | ||
| Princeton University | 52.5 | 65 | 40 | 30 | 8 | 67.5 | 74 | 61 | 8 | 3 | ||
| University of Amsterdam | 50.5 | 66 | 35 | 35 | 10 | 47 | 69 | 25 | 41 | 15 | ||
| Harvard University | 48 | 69 | 27 | 40 | 14 | 55 | 68 | 42 | 22 | 7 | ||
| Yale University | 48 | 65 | 31 | 38 | 12 | 55 | 70 | 40 | 31 | 8 | ||
| University Texas - Austin | 46.5 | 66 | 27 | 44 | 14 | 55.5 | 70 | 41 | 24 | 8 | ||
| University of British Columbia | 44 | 67 | 21 | 47 | 20 | 47 | 65 | 29 | 34 | 13 | ||
| McGill University | 43.5 | 66 | 21 | 57 | 20 | 43.5 | 69 | 18 | 57 | 23 | ||
| Columbia University | 41.5 | 62 | 21 | 49 | 19 | 39 | 61 | 17 | 50 | 26 | ||
| New York University | 41 | 62 | 20 | 50 | 20 | 48 | 70 | 26 | 43 | 15 | ||
| Stanford University | 41 | 60 | 22 | 45 | 18 | 58 | 66 | 50 | 15 | 5 |