Update: 2/27/2021
This post has been replaced by a new post. The rankings here are only shown for the sake of transparency, but scores have been replaced and the new scores should be used (https://replicationindex.com/2021/01/19/personalized-p-values/). The correlation between these scores and the new scores is r ~ .5. This is fairly low and I have been trying to figure out the reason for the discrepancy. However, I am not able to reproduce the results posted here. The problem is that I did not document the criteria for journal selection and I did not store the files with the list of articles for each author. Thus, these results are not reproducible. In contrast, the new results are reproducible and the data are openly shared to allow others to reproduce the results.
Original Post from November 8, 2018
Social psychology has a replication problem. The reason is that social psychologists used questionable research practices to increase their chances of reporting significant results. The consequence is that the real risk of a false positive result is higher than the stated 5% level in publications. In other words, p < .05 no longer means that at most 5% of published results are false positives (Sterling, 1959). Another problem is that selection for significance with low power produces inflated effect sizes estimates. Estimates suggests that on average published effect sizes are inflated by 100% (OSC, 2015). These problems have persisted for decades (Sterling, 1959), but only now psychologists are recognizing that published results provide weak evidence and might not be replicable even if the same study were replicated exactly.
How should consumers of empirical social psychology (textbook writers, undergraduate students, policy planners) respond to the fact that published results cannot be trusted at face value? Jerry Brunner and I have been working on ways to correct published results for the inflation introduced by selection for significance and questionable practices. Z-curve estimates the mean power of studies selected for significance. Here I applied the method to automatically extracted test statistics from social psychology journals. I computed z-curves for 70+ eminent social psychologists (H-index > 35).
The results can be used to evaluate the published results reported by individual researchers. The main information provided in the table are (a) the replicability of all published p-values, (b) the replicability of just significant p-values (defined as p-values greater than pnorm(2.5) = .0124, and (c) the replicability of p-values with moderate evidence against the null-hypothesis (.0124 > p > .0027). More detailed information is provided in the z-curve plots (powergraphs) that are linked to researchers’ names. An index less than 50% would suggest that these p-values are no longer significant after adjusting for selection for significance. As can be seen in the table, most just significant results are no longer significant after correction for bias.
Caveat: Interpret with Care
The results should not be overinterpreted. They are estimates based on an objective statistical procedure, but no statistical method can compensate perfectly for the various practices that led to the observed distribution of p-values (transformed into z-scores). However, in the absence of any information which results can be trusted, these graphs provide some information. How this information is used by consumers depends ultimately on consumers’ subjective beliefs. Information about the average replicability of researchers’ published results may influence these beliefs.
It is also important to point out that a low replicability index does not mean researchers were committing scientific misconduct. There are no clear guidelines about acceptable and unacceptable statistical practices in psychology. Zcurve is not designed to detect scientific fraud. In fact, it assumes that researcher collect data, but conduct analyses in a way that increases the chances of producing a significant result. The bias introduced by selection for significance is well known and considered acceptable in psychological science.
There are also many factors that can bias results in favor of researchers’ hypotheses without researchers’ awareness. Thus, the bias evident in many graphs does not imply that researchers intentionally manipulated data to support their claims. Thus, I attribute the bias to unidentified researcher influences. It is not important to know how bias occurred. It is only important to detect biases and to correct for them.
It is necessary to do so for individual researchers because bias varies across researchers. For example, the R-Index for all results ranges from 22% to 81%. It would be unfair to treat all social psychologists alike when their research practices are a reliable moderator of replicability. Providing personalized information about replicability allows consumers of social psychological research to avoid stereotyping social psychologists and to take individual differences in research practices into account.
Finally, it should be said that producing replicabilty estimates is subject to biases and errors. Researchers may differ in their selection of hypotheses that they are reporting. A more informative analysis would require hand-coding of researchers’ focal hypothesis tests. At the moment, R-Index does not have the resources to code all published results in social psychology, let alone other areas of psychology. This is an important task for the future. At the moment, automatically extracted results have some heuristic value.
One unintended and unfortunate consequence of making this information available is that some researchers’ reputation might be negatively effected by a low replicability score. This cost has be be weighted against the benefit to the public and the scientific community of obtaining information about the robustness of published results. In this regard, the replicability rankings are no different from actual replication studies that fail to replicate an original finding. The only difference is that replicability rankings use all published results, whereas actual replication studies are often limited to a single or a few studies. While replication failures in a single study are ambiguous, replicability esitmates based on hundreds of published results are more diagnostic of researchers’ practices.
Nevertheless, statistical estimates provide no definitive answer about the reproducibility of a published result. Ideally, eminent researchers would conduct their own replication studies to demonstrate that their most important findings can be replicated under optimal conditions.
It is also important to point out that researchers have responded differently to the replication crisis that became apparent in 2011. It may be unfair to generalize from past practices to new findings for researchers who changed their practices. If researchers preregistered their studies and followed a well-designed registered research protocol, new results may be more robust than a researchers’ past record suggests.
Finally, the results show evidence of good replicability for some social psychologists. Thus, the rankings avoid the problem of selectively targeting researchers with low replicability, which can lead to a negative bias in evaluations of social psychology. The focus on researchers with a high H-index means that the results are representative of the field.
If you believe that you should not be listed as an eminent social psychologists, please contact me so that I can remove you from the list.
If you think you are an eminent social psychologists and you want to be included in the ranking, please contact me so that I can add you to the list.
If you have any suggestions or comments how I can make these rankings more informative, please let me know in the comments section.
*** *** *** *** ***
REPLICABILITY RANKING OF EMINENT SOCIAL PSYCHOLOGISTS
[sorted by R-Index for all tests from highest to lowest rank]
Rank | Journal | R-Index (all) | R-Index (2.0-2.5) | R-Index (2.5-3.0) | #P-vals | H-Index | #Pub | #cit(*1000) |
1 | Steven J. Heine | 81 | 44 | 55 | 197 | 41 | 83 | 10 |
2 | James J. Gross | 80 | 35 | 58 | 360 | 82 | 413 | 34 |
3 | Constantine Sedikides | 76 | 45 | 53 | 884 | 52 | 263 | 10 |
4 | Bertram Gawronski | 73 | 29 | 51 | 1717 | 37 | 113 | 6 |
5 | Kathleen D. Vohs | 73 | 36 | 53 | 452 | 49 | 158 | 11 |
6 | Paul Rozin | 73 | 39 | 57 | 155 | 65 | 218 | 16 |
7 | Alice H. Eagly | 72 | 42 | 50 | 384 | 61 | 161 | 18 |
8 | Anthony G. Greenwald | 72 | 28 | 51 | 273 | 64 | 175 | 26 |
9 | David Dunning | 72 | 30 | 50 | 674 | 40 | 105 | 8 |
10 | Richard E. Nisbett | 72 | 57 | 59 | 190 | 62 | 119 | 20 |
11 | Shinobu Kitayama | 70 | 37 | 44 | 545 | 35 | 103 | 15 |
12 | Timothy D. Wilson | 70 | 30 | 52 | 327 | 44 | 85 | 14 |
13 | Mahzarin R. Banaji | 68 | 52 | 53 | 651 | 60 | 133 | 22 |
14 | Marilynn B. Brewer | 68 | 43 | 46 | 193 | 49 | 107 | 15 |
15 | Patricia G. Devine | 67 | 44 | 52 | 1098 | 35 | 83 | 9 |
16 | Susan T. Fiske | 67 | 26 | 41 | 419 | 66 | 213 | 22 |
17 | Thomas Gilovich | 67 | 23 | 44 | 754 | 44 | 104 | 8 |
18 | Daniel T. Gilbert | 66 | 26 | 46 | 357 | 43 | 107 | 8 |
19 | Hazel R. Markus | 66 | 48 | 53 | 348 | 38 | 96 | 12 |
20 | Mark P. Zanna | 66 | 26 | 49 | 565 | 57 | 167 | 11 |
21 | Wendy Wood | 66 | 33 | 40 | 373 | 42 | 112 | 7 |
22 | Brad J. Bushman | 65 | 33 | 50 | 247 | 48 | 227 | 13 |
23 | E. Tory. Higgins | 65 | 27 | 44 | 887 | 72 | 274 | 25 |
24 | Jeff Greenberg | 65 | 30 | 42 | 679 | 60 | 154 | 13 |
25 | Nira Liberman | 64 | 25 | 37 | 1578 | 46 | 115 | 10 |
26 | Caryl E. Rusbult | 63 | 27 | 34 | 171 | 36 | 68 | 9 |
27 | Dacher Keltner | 63 | 38 | 43 | 903 | 60 | 159 | 15 |
28 | Harry Reis | 63 | 21 | 41 | 470 | 37 | 83 | 6 |
29 | John F. Dovidio | 63 | 28 | 37 | 2323 | 56 | 206 | 12 |
30 | John T. Cacioppo | 63 | 23 | 41 | 256 | 101 | 422 | 41 |
31 | Nalini Ambady | 63 | 24 | 42 | 675 | 51 | 171 | 10 |
32 | Philip R. Shaver | 63 | 24 | 42 | 675 | 60 | 184 | 13 |
33 | Richard E. Petty | 63 | 30 | 37 | 1428 | 69 | 190 | 20 |
34 | Robert B. Cialdini | 63 | 45 | 49 | 258 | 51 | 121 | 11 |
35 | Michael Ross | 63 | 38 | 51 | 631 | 42 | 245 | 7 |
36 | Lee Ross | 62 | 23 | 44 | 952 | 49 | 218 | 12 |
37 | Roy F. Baumeister | 62 | 32 | 44 | 1015 | 100 | 363 | 46 |
38 | S. Alexander Haslam | 62 | 34 | 41 | 289 | 52 | 234 | 9 |
39 | Tom Pyszczynski | 62 | 33 | 40 | 1101 | 60 | 149 | 13 |
40 | Philip E. Tetlock | 62 | 29 | 39 | 158 | 58 | 189 | 11 |
41 | Arie W. Kruglanski | 61 | 23 | 42 | 1140 | 50 | 228 | 13 |
42 | Galen V. Bodenhausen | 61 | 18 | 43 | 465 | 39 | 80 | 8 |
43 | Norbert Schwarz | 61 | 36 | 42 | 2524 | 49 | 138 | 13 |
44 | Jonathan Haidt | 60 | 16 | 37 | 98 | 42 | 84 | 14 |
45 | Shelly Chaiken | 60 | 15 | 36 | 288 | 46 | 86 | 10 |
46 | Ap Dijksterhuis | 59 | 17 | 36 | 456 | 42 | 118 | 8 |
47 | Eddie Harmon-Jones | 59 | 25 | 37 | 343 | 57 | 212 | 10 |
48 | Fritz Strack | 59 | 22 | 43 | 588 | 51 | 149 | 11 |
49 | Joseph P. Forgas | 59 | 29 | 43 | 176 | 37 | 142 | 4 |
50 | Yaacov Trope | 59 | 20 | 39 | 1957 | 57 | 135 | 12 |
51 | Charles M. Judd | 59 | 28 | 33 | 666 | 37 | 142 | 6 |
52 | Craig A. Anderson | 59 | 26 | 34 | 265 | 51 | 117 | 11 |
53 | C. Nathan DeWall | 58 | 30 | 41 | 1099 | 37 | 135 | 7 |
54 | Eli J. Finkel | 58 | 24 | 35 | 1921 | 38 | 109 | 4 |
55 | Jeffry A. Simpson | 58 | 18 | 28 | 261 | 42 | 95 | 7 |
56 | Peter M. Gollwitzer | 58 | 29 | 44 | 1711 | 45 | 158 | 11 |
57 | Mario Mikulincer | 58 | 29 | 34 | 104 | 68 | 272 | 15 |
58 | Russell H. Fazio | 57 | 23 | 29 | 464 | 51 | 134 | 14 |
59 | Thomas Mussweiler | 57 | 24 | 38 | 1128 | 35 | 85 | 5 |
60 | Daniel M. Wegner | 56 | 27 | 36 | 699 | 53 | 130 | 14 |
61 | John A. Bargh | 56 | 20 | 36 | 755 | 61 | 140 | 22 |
62 | Robert S. Wyer | 56 | 33 | 44 | 283 | 40 | 203 | 8 |
63 | Carol S. Dweck | 54 | 17 | 29 | 458 | 59 | 147 | 20 |
64 | Michael Inzlicht | 54 | 32 | 36 | 156 | 36 | 124 | 5 |
65 | John T. Jost | 52 | 21 | 33 | 249 | 46 | 132 | 11 |
66 | Shelly E. Taylor | 51 | 15 | 28 | 198 | 73 | 169 | 28 |
67 | Claude M. Steele | 48 | 38 | 42 | 376 | 29 | 49 | 13 |
68 | Gerald L. Clore | 45 | 20 | 35 | 200 | 37 | 84 | 9 |
69 | Adam D. Galinsky | 43 | 24 | 28 | 585 | 57 | 198 | 11 |
70 | Robert Zajonc | 39 | 11 | 26 | 67 | 40 | 114 | 16 |
71 | Jennifer Crocker | 22 | 5 | 5 | 99 | 38 | 68 | 7 |
The table is a little obscure. I think you need to spell things out in some more detail. And a graph that lets us see the results in a visual form might help too.
Could you please add more highly cited researchers on the list? For instance, Ryan and Deci and their self-determination theory seem to be very widely applied. I wonder if you have any data on their replicability?
Thank you for the suggestion. I will work on an update this year.
An example graph with a caption describing how to interpret it would be useful.
Thanks. I am working on a revised post with updated data