Last update 2/24/2021
(the latest updated included articles published in 2020. This produced some changes in the rankings).
Introduction
Since Fisher invented null-hypothesis significance testing, researchers have used p < .05 as a statistical criterion to interpret results as discoveries worthwhile of discussion (i.e., the null-hypothesis is false). Once published, these results are often treated as real findings even though alpha does not control the risk of false discoveries.
Statisticians have warned against the exclusive reliance on p < .05, but nearly 100 years after Fisher popularized this approach, it is still the most common way to interpret data. The main reason is that many attempts to improve on this practice have failed. The main problem is that a single statistical result is difficult to interpret. However, when individual results are interpreted in the context of other results, they become more informative. Based on the distribution of p-values it is possible to estimate the maximum false discovery rate (Bartos & Schimmack, 2020; Jager & Leek, 2014). This approach can be applied to the p-values published by individual authors to adjust p-values to keep the risk of false discoveries at a reasonable level, FDR < .05.
Researchers who mainly test true hypotheses with high power have a high discovery rate (many p-values below .05) and a low false discovery rate (FDR < .05). Figure 1 shows an example of a researcher who followed this strategy (for a detailed description of z-curve plots, see Schimmack, 2021).
We see that out of the 317 test-statistics retrieved from his articles, 246 were significant with alpha = .05. This is an observed discovery rate of 78%. We also see that this discovery rate closely matches the estimated discovery rate based on the distribution of the significant p-values, p < .05. The EDR is 79%. With an EDR of 79%, the maximum false discovery rate is only 1%. However, the 95%CI is wide and the lower bound of the CI for the EDR, 27%, allows for 14% false discoveries.
When the ODR matches the EDR, there is no evidence of publication bias. In this case, we can improve the estimates by fitting all p-values, including the non-significant ones. With a tighter CI for the EDR, we see that the 95%CI for the maximum FDR ranges from 1% to 3%. Thus, we can be confident that no more than 5% of the significant results wit alpha = .05 are false discoveries. Readers can therefore continue to use alpha = .05 to look for interesting discoveries in Matsumoto’s articles.
Figure 3 shows the results for a different type of researcher who took a risk and studied weak effect sizes with small samples. This produces many non-significant results that are often not published. The selection for significance inflates the observed discovery rate, but the z-curve plot and the comparison with the EDR shows the influence of publication bias. Here the ODR is similar to Figure 1, but the EDR is only 11%. An EDR of 11% translates into a large maximum false discovery rate of 41%. In addition, the 95%CI of the EDR includes 5%, which means the risk of false positives could be as high as 100%. In this case, using alpha = .05 to interpret results as discoveries is very risky. Clearly, p < .05 means something very different when reading an article by David Matsumoto or Shelly Chaiken.
Rather than dismissing all of Chaiken’s results, we can try to lower alpha to reduce the false discovery rate. If we set alpha = .01, the FDR is 15%. If we set alpha = .005, the FDR is 8%. To get the FDR below 5%, we need to set alpha to .001.
A uniform criterion of FDR < 5% is applied to all researchers in the rankings below. For some this means no adjustment to the traditional criterion. For others, alpha is lowered to .01, and for a few even lower than that.
The rankings below are based on automatrically extracted test-statistics from 40 journals (List of journals). The results should be interpreted with caution and treated as preliminary. They depend on the specific set of journals that were searched, the way results are being reported, and many other factors. The data are available (data.drop) and researchers can exclude articles or add articles and run their own analyses using the z-curve package in R (https://replicationindex.com/2020/01/10/z-curve-2-0/).
I am also happy to receive feedback about coding errors. I also recommended to hand-code articles to adjust alpha for focal hypothesis tests. This typically lowers the EDR and increases the FDR. For example, the automated method produced an EDR of 31 for Bargh, whereas hand-coding of focal tests produced an EDR of 12 (Bargh-Audit).
And here are the rankings. The results are fully automated and I was not able to cover up the fact that I placed only #139 out of 300 in the rankings. In another post, I will explain how researchers can move up in the rankings. Of course, one way to move up in the rankings is to increase statistical power in future studies. The rankings will be updated again when the 2021 data are available.
Despite the preliminary nature, I am confident that the results provide valuable information. Until know all p-values below .05 have been treated as if they are equally informative. The rankings here show that this is not the case. While p = .02 can be informative for one researcher, p = .002 may still entail a high false discovery risk for another researcher.
| Name | Tests | ODR | EDR | ERR | FDR | Alpha |
| Robert A. Emmons | 58 | 88 | 85 | 88 | 1 | .05 |
| David Matsumoto | 378 | 83 | 79 | 85 | 1 | .05 |
| Linda J. Skitka | 532 | 68 | 75 | 82 | 2 | .05 |
| Jonathan B. Freeman | 274 | 59 | 75 | 81 | 2 | .05 |
| Virgil Zeigler-Hill | 515 | 72 | 74 | 81 | 2 | .05 |
| David P. Schmitt | 207 | 78 | 71 | 77 | 2 | .05 |
| Emily A. Impett | 549 | 77 | 70 | 76 | 2 | .05 |
| Michael E. McCullough | 334 | 69 | 69 | 78 | 2 | .05 |
| Kipling D. Williams | 843 | 75 | 69 | 77 | 2 | .05 |
| John M. Zelenski | 156 | 71 | 69 | 76 | 2 | .05 |
| Kurt Gray | 487 | 79 | 69 | 81 | 2 | .05 |
| Hilary B. Bergsieker | 439 | 67 | 68 | 74 | 2 | .05 |
| Cameron Anderson | 652 | 71 | 67 | 74 | 3 | .05 |
| Jamil Zaki | 430 | 78 | 66 | 76 | 3 | .05 |
| Phoebe C. Ellsworth | 605 | 74 | 65 | 72 | 3 | .05 |
| Jim Sidanius | 487 | 69 | 65 | 72 | 3 | .05 |
| Benjamin R. Karney | 392 | 56 | 65 | 73 | 3 | .05 |
| A. Janet Tomiyama | 76 | 78 | 65 | 76 | 3 | .05 |
| Carol D. Ryff | 280 | 84 | 64 | 76 | 3 | .05 |
| Juliane Degner | 435 | 63 | 64 | 71 | 3 | .05 |
| Thomas N Bradbury | 398 | 61 | 63 | 69 | 3 | .05 |
| Steven J. Heine | 597 | 78 | 63 | 77 | 3 | .05 |
| David M. Amodio | 584 | 66 | 63 | 70 | 3 | .05 |
| Elaine Fox | 472 | 79 | 62 | 78 | 3 | .05 |
| Klaus Fiedler | 1421 | 78 | 60 | 72 | 3 | .05 |
| Richard W. Robins | 270 | 76 | 60 | 70 | 4 | .05 |
| William B. Swann Jr. | 1070 | 78 | 59 | 80 | 4 | .05 |
| Margaret S. Clark | 505 | 75 | 59 | 77 | 4 | .05 |
| Edward P. Lemay | 289 | 87 | 59 | 81 | 4 | .05 |
| Patricia G. Devine | 606 | 71 | 58 | 67 | 4 | .05 |
| B. Keith Payne | 879 | 71 | 58 | 76 | 4 | .05 |
| Ximena B. Arriaga | 284 | 66 | 58 | 69 | 4 | .05 |
| Rainer Reisenzein | 201 | 65 | 57 | 69 | 4 | .05 |
| Jean M. Twenge | 381 | 72 | 56 | 59 | 4 | .05 |
| Barbara A. Mellers | 287 | 80 | 56 | 78 | 4 | .05 |
| Joris Lammers | 705 | 69 | 56 | 69 | 4 | .05 |
| Nicholas Epley | 1504 | 74 | 55 | 72 | 4 | .05 |
| Richard M. Ryan | 998 | 78 | 52 | 69 | 5 | .05 |
| Edward L. Deci | 284 | 79 | 52 | 63 | 5 | .05 |
| Ethan Kross | 614 | 66 | 52 | 67 | 5 | .05 |
| Lee Jussim | 226 | 80 | 52 | 71 | 5 | .05 |
| Samuel D. Gosling | 108 | 58 | 51 | 62 | 5 | .05 |
| Jens B. Asendorpf | 253 | 74 | 51 | 69 | 5 | .05 |
| Roger Giner-Sorolla | 663 | 81 | 51 | 80 | 5 | .05 |
| Tessa V. West | 691 | 71 | 51 | 59 | 5 | .05 |
| James J. Gross | 1104 | 72 | 50 | 77 | 5 | .05 |
| Paul Rozin | 449 | 78 | 50 | 84 | 5 | .05 |
| Shinobu Kitayama | 983 | 76 | 50 | 71 | 5 | .05 |
| Janice R. Kelly | 366 | 75 | 50 | 70 | 5 | .05 |
| Sheena S. Iyengar | 207 | 63 | 50 | 80 | 5 | .05 |
| Paul K. Piff | 166 | 77 | 50 | 63 | 5 | .05 |
| Mina Cikara | 392 | 71 | 49 | 80 | 5 | .05 |
| Bertram Gawronski | 1803 | 72 | 48 | 76 | 6 | .01 |
| Edward R. Hirt | 1042 | 81 | 48 | 65 | 6 | .01 |
| Penelope Lockwood | 458 | 71 | 48 | 70 | 6 | .01 |
| John T. Cacioppo | 438 | 76 | 47 | 69 | 6 | .01 |
| Matthew D. Lieberman | 398 | 72 | 47 | 80 | 6 | .01 |
| Daniel M. Wegner | 602 | 76 | 47 | 65 | 6 | .01 |
| Agneta H. Fischer | 952 | 75 | 47 | 69 | 6 | .01 |
| Leaf van Boven | 711 | 72 | 47 | 67 | 6 | .01 |
| Stephanie A. Fryberg | 248 | 62 | 47 | 66 | 6 | .01 |
| Alice H. Eagly | 330 | 75 | 46 | 71 | 6 | .01 |
| Rainer Banse | 402 | 78 | 46 | 72 | 6 | .01 |
| Jeanne L. Tsai | 1241 | 73 | 46 | 67 | 6 | .01 |
| Jennifer S. Lerner | 181 | 80 | 46 | 61 | 6 | .01 |
| Dacher Keltner | 1233 | 72 | 45 | 64 | 6 | .01 |
| Constantine Sedikides | 2566 | 71 | 45 | 70 | 6 | .01 |
| Andrea L. Meltzer | 549 | 52 | 45 | 72 | 6 | .01 |
| R. Chris Fraley | 642 | 70 | 45 | 72 | 7 | .01 |
| Brian A. Nosek | 816 | 68 | 44 | 81 | 7 | .01 |
| Ursula Hess | 774 | 78 | 44 | 71 | 7 | .01 |
| S. Alexander Haslam | 1198 | 72 | 43 | 64 | 7 | .01 |
| Charles M. Judd | 1054 | 76 | 43 | 68 | 7 | .01 |
| Mark Schaller | 565 | 73 | 43 | 61 | 7 | .01 |
| Jason P. Mitchell | 600 | 73 | 43 | 73 | 7 | .01 |
| Jessica L. Tracy | 632 | 74 | 43 | 71 | 7 | .01 |
| Mario Mikulincer | 901 | 89 | 42 | 64 | 7 | .01 |
| Lisa Feldman Barrett | 644 | 69 | 42 | 70 | 7 | .01 |
| Susan T. Fiske | 911 | 78 | 42 | 74 | 7 | .01 |
| Bernadette Park | 973 | 77 | 42 | 64 | 7 | .01 |
| Paul A. M. Van Lange | 1092 | 70 | 42 | 63 | 7 | .01 |
| Wendi L. Gardner | 798 | 67 | 42 | 63 | 7 | .01 |
| Philip E. Tetlock | 549 | 79 | 41 | 73 | 7 | .01 |
| Jordan B. Peterson | 266 | 60 | 41 | 79 | 7 | .01 |
| Michael Inzlicht | 566 | 64 | 41 | 61 | 8 | .01 |
| Stacey Sinclair | 327 | 70 | 41 | 57 | 8 | .01 |
| Richard E. Petty | 2771 | 69 | 40 | 64 | 8 | .01 |
| Norbert Schwarz | 1337 | 72 | 40 | 63 | 8 | .01 |
| Wendy Wood | 462 | 75 | 40 | 62 | 8 | .01 |
| Tiffany A. Ito | 349 | 80 | 40 | 64 | 8 | .01 |
| Elizabeth Page-Gould | 411 | 57 | 40 | 66 | 8 | .01 |
| Carol S. Dweck | 1028 | 70 | 39 | 63 | 8 | .01 |
| Marcel Zeelenberg | 868 | 76 | 39 | 79 | 8 | .01 |
| Christian S. Crandall | 362 | 75 | 39 | 59 | 8 | .01 |
| Tobias Greitemeyer | 1737 | 72 | 39 | 67 | 8 | .01 |
| Jason E. Plaks | 582 | 70 | 39 | 67 | 8 | .01 |
| Jerry Suls | 413 | 71 | 38 | 68 | 8 | .01 |
| Eric D. Knowles | 384 | 68 | 38 | 64 | 8 | .01 |
| John F. Dovidio | 2019 | 69 | 38 | 62 | 9 | .01 |
| C. Nathan DeWall | 1336 | 73 | 38 | 63 | 9 | .01 |
| Harry T. Reis | 998 | 69 | 38 | 74 | 9 | .01 |
| Joshua Correll | 549 | 61 | 38 | 62 | 9 | .01 |
| Abigail A. Scholer | 556 | 58 | 38 | 62 | 9 | .01 |
| Mahzarin R. Banaji | 880 | 73 | 37 | 78 | 9 | .01 |
| Antony S. R. Manstead | 1656 | 72 | 37 | 62 | 9 | .01 |
| Kevin N. Ochsner | 406 | 79 | 37 | 70 | 9 | .01 |
| Fritz Strack | 607 | 75 | 37 | 56 | 9 | .01 |
| Ayelet Fishbach | 1416 | 78 | 37 | 59 | 9 | .01 |
| Lorne Campbell | 433 | 67 | 37 | 61 | 9 | .01 |
| Geoff MacDonald | 406 | 67 | 37 | 67 | 9 | .01 |
| Mark J. Brandt | 277 | 70 | 37 | 70 | 9 | .01 |
| Craig A. Anderson | 467 | 76 | 36 | 55 | 9 | .01 |
| Barbara L. Fredrickson | 287 | 72 | 36 | 61 | 9 | .01 |
| Nyla R. Branscombe | 1276 | 70 | 36 | 65 | 9 | .01 |
| Niall Bolger | 376 | 67 | 36 | 58 | 9 | .01 |
| D. S. Moskowitz | 3418 | 74 | 36 | 63 | 9 | .01 |
| Duane T. Wegener | 980 | 77 | 36 | 60 | 9 | .01 |
| Joanne V. Wood | 1093 | 74 | 36 | 60 | 9 | .01 |
| Yaacov Schul | 411 | 61 | 36 | 64 | 9 | .01 |
| Jeff T. Larsen | 181 | 74 | 36 | 67 | 10 | .01 |
| Nalini Ambady | 1256 | 62 | 35 | 56 | 10 | .01 |
| John T. Jost | 794 | 70 | 35 | 61 | 10 | .01 |
| Daphna Oyserman | 446 | 55 | 35 | 54 | 10 | .01 |
| Samuel L. Gaertner | 321 | 75 | 35 | 61 | 10 | .01 |
| Michael Harris Bond | 378 | 73 | 35 | 84 | 10 | .01 |
| Michael D. Robinson | 1388 | 78 | 35 | 66 | 10 | .01 |
| Igor Grossmann | 203 | 64 | 35 | 66 | 10 | .01 |
| Azim F. Sharif | 183 | 74 | 35 | 68 | 10 | .01 |
| Eva Walther | 493 | 82 | 35 | 66 | 10 | .01 |
| C. Miguel Brendl | 121 | 76 | 35 | 68 | 10 | .01 |
| Emily Balcetis | 599 | 69 | 35 | 68 | 10 | .01 |
| Diana I. Tamir | 156 | 62 | 35 | 62 | 10 | .01 |
| Thomas Gilovich | 1193 | 80 | 34 | 69 | 10 | .01 |
| Paula M. Niedenthal | 522 | 69 | 34 | 61 | 10 | .01 |
| Ozlem Ayduk | 549 | 62 | 34 | 59 | 10 | .01 |
| Wiebke Bleidorn | 99 | 63 | 34 | 74 | 10 | .01 |
| Alison Ledgerwood | 214 | 75 | 34 | 54 | 10 | .01 |
| Kerry Kawakami | 487 | 68 | 33 | 56 | 10 | .01 |
| Christopher R. Agnew | 325 | 75 | 33 | 76 | 10 | .01 |
| Jennifer A. Richeson | 831 | 67 | 33 | 52 | 11 | .01 |
| Malte Friese | 501 | 61 | 33 | 57 | 11 | .01 |
| Danu Anthony Stinson | 494 | 77 | 33 | 54 | 11 | .01 |
| Mark Snyder | 562 | 72 | 32 | 63 | 11 | .01 |
| Robert B. Cialdini | 379 | 72 | 32 | 56 | 11 | .01 |
| Russell H. Fazio | 1094 | 69 | 32 | 61 | 11 | .01 |
| Eli J. Finkel | 1392 | 62 | 32 | 57 | 11 | .01 |
| Ulrich Schimmack | 318 | 75 | 32 | 63 | 11 | .01 |
| Margo J. Monteith | 773 | 76 | 32 | 77 | 11 | .01 |
| E. Ashby Plant | 831 | 77 | 31 | 51 | 11 | .01 |
| Christopher K. Hsee | 689 | 75 | 31 | 63 | 11 | .01 |
| Yuen J. Huo | 132 | 74 | 31 | 80 | 11 | .01 |
| Roy F. Baumeister | 2442 | 69 | 31 | 52 | 12 | .01 |
| John A. Bargh | 651 | 72 | 31 | 55 | 12 | .01 |
| Tom Pyszczynski | 948 | 69 | 31 | 54 | 12 | .01 |
| Delroy L. Paulhus | 121 | 77 | 31 | 82 | 12 | .01 |
| Kathleen D. Vohs | 944 | 68 | 31 | 51 | 12 | .01 |
| Jamie Arndt | 1318 | 69 | 31 | 50 | 12 | .01 |
| Arthur Aron | 307 | 65 | 30 | 56 | 12 | .01 |
| Anthony G. Greenwald | 357 | 72 | 30 | 83 | 12 | .01 |
| Jennifer Crocker | 515 | 68 | 30 | 67 | 12 | .01 |
| Dale T. Miller | 521 | 71 | 30 | 64 | 12 | .01 |
| Aaron C. Kay | 1320 | 70 | 30 | 51 | 12 | .01 |
| Lauren J. Human | 447 | 59 | 30 | 70 | 12 | .01 |
| Steven W. Gangestad | 198 | 63 | 30 | 41 | 13 | .005 |
| Nicholas O. Rule | 1294 | 68 | 30 | 75 | 13 | .01 |
| Jeff Greenberg | 1358 | 77 | 29 | 54 | 13 | .01 |
| Hazel Rose Markus | 674 | 76 | 29 | 68 | 13 | .01 |
| Russell Spears | 2286 | 73 | 29 | 55 | 13 | .01 |
| Gordon B. Moskowitz | 374 | 72 | 29 | 57 | 13 | .01 |
| Richard E. Nisbett | 319 | 73 | 29 | 69 | 13 | .01 |
| Eliot R. Smith | 445 | 79 | 29 | 73 | 13 | .01 |
| Boris Egloff | 274 | 81 | 29 | 58 | 13 | .01 |
| Caryl E. Rusbult | 218 | 60 | 29 | 54 | 13 | .01 |
| Dirk Wentura | 830 | 65 | 29 | 64 | 13 | .01 |
| Nir Halevy | 262 | 68 | 29 | 72 | 13 | .01 |
| Adam D. Galinsky | 2154 | 70 | 28 | 49 | 13 | .01 |
| Jeffry A. Simpson | 697 | 74 | 28 | 55 | 13 | .01 |
| Yoav Bar-Anan | 525 | 75 | 28 | 76 | 13 | .01 |
| Roland Neumann | 258 | 77 | 28 | 67 | 13 | .01 |
| Richard J. Davidson | 380 | 64 | 28 | 51 | 14 | .01 |
| Eddie Harmon-Jones | 738 | 73 | 28 | 70 | 14 | .01 |
| Brent W. Roberts | 562 | 72 | 28 | 77 | 14 | .01 |
| Naomi I. Eisenberger | 179 | 74 | 28 | 79 | 14 | .01 |
| Sander L. Koole | 767 | 65 | 28 | 52 | 14 | .01 |
| Shelly L. Gable | 364 | 64 | 28 | 50 | 14 | .01 |
| Joshua Aronson | 183 | 85 | 28 | 46 | 14 | .005 |
| Elizabeth W. Dunn | 395 | 75 | 28 | 64 | 14 | .01 |
| Grainne M. Fitzsimons | 585 | 68 | 28 | 49 | 14 | .01 |
| Geoffrey J. Leonardelli | 290 | 68 | 28 | 48 | 14 | .005 |
| Matthew Feinberg | 295 | 77 | 28 | 69 | 14 | .01 |
| Jan De Houwer | 1972 | 70 | 27 | 72 | 14 | .01 |
| Karl Christoph Klauer | 801 | 67 | 27 | 65 | 14 | .01 |
| Guido H. E. Gendolla | 422 | 76 | 27 | 47 | 14 | .005 |
| Jennifer S. Beer | 80 | 56 | 27 | 54 | 14 | .01 |
| Klaus R. Scherer | 467 | 83 | 26 | 78 | 15 | .01 |
| Galen V. Bodenhausen | 585 | 74 | 26 | 61 | 15 | .01 |
| Sonja Lyubomirsky | 531 | 71 | 26 | 59 | 15 | .01 |
| Claude M. Steele | 434 | 73 | 26 | 42 | 15 | .005 |
| William G. Graziano | 532 | 71 | 26 | 66 | 15 | .01 |
| Kristin Laurin | 648 | 63 | 26 | 51 | 15 | .01 |
| Kerri L. Johnson | 532 | 76 | 25 | 76 | 15 | .01 |
| Phillip R. Shaver | 566 | 81 | 25 | 71 | 16 | .01 |
| David Dunning | 818 | 74 | 25 | 70 | 16 | .01 |
| Laurie A. Rudman | 482 | 72 | 25 | 68 | 16 | .01 |
| Joel Cooper | 257 | 72 | 25 | 39 | 16 | .005 |
| Batja Mesquita | 416 | 71 | 25 | 73 | 16 | .01 |
| Ronald S. Friedman | 183 | 79 | 25 | 44 | 16 | .005 |
| Steven J. Sherman | 888 | 74 | 24 | 62 | 16 | .01 |
| Alison L. Chasteen | 223 | 68 | 24 | 69 | 16 | .01 |
| Shigehiro Oishi | 1109 | 64 | 24 | 61 | 17 | .01 |
| Thomas Mussweiler | 604 | 70 | 24 | 43 | 17 | .005 |
| Mark W. Baldwin | 247 | 72 | 24 | 41 | 17 | .005 |
| Jonathan Haidt | 368 | 76 | 23 | 73 | 17 | .01 |
| Brandon J. Schmeichel | 652 | 66 | 23 | 45 | 17 | .005 |
| Jeffrey W Sherman | 992 | 68 | 23 | 71 | 17 | .01 |
| Felicia Pratto | 410 | 73 | 23 | 75 | 18 | .01 |
| Klaus Rothermund | 738 | 71 | 23 | 76 | 18 | .01 |
| Bernard A. Nijstad | 693 | 71 | 23 | 52 | 18 | .005 |
| Roland Imhoff | 365 | 74 | 23 | 73 | 18 | .01 |
| Jennifer L. Eberhardt | 202 | 71 | 23 | 62 | 18 | .005 |
| Michael Ross | 1164 | 70 | 22 | 62 | 18 | .005 |
| Marilynn B. Brewer | 314 | 75 | 22 | 62 | 18 | .005 |
| Dieter Frey | 1538 | 68 | 22 | 58 | 18 | .005 |
| David M. Buss | 461 | 82 | 22 | 80 | 19 | .01 |
| Wendy Berry Mendes | 965 | 68 | 22 | 44 | 19 | .005 |
| Yoel Inbar | 280 | 67 | 22 | 71 | 19 | .01 |
| Sean M. McCrea | 584 | 73 | 22 | 54 | 19 | .005 |
| Spike W. S. Lee | 145 | 68 | 22 | 64 | 19 | .005 |
| Joseph P. Forgas | 888 | 83 | 21 | 59 | 19 | .005 |
| Maya Tamir | 1342 | 80 | 21 | 64 | 19 | .005 |
| Paul W. Eastwick | 583 | 65 | 21 | 69 | 19 | .005 |
| Elizabeth Levy Paluck | 31 | 84 | 21 | 55 | 20 | .005 |
| Andrew J. Elliot | 1018 | 81 | 20 | 67 | 21 | .005 |
| Jay J. van Bavel | 437 | 64 | 20 | 71 | 21 | .005 |
| Tanya L. Chartrand | 424 | 67 | 20 | 33 | 21 | .001 |
| Geoffrey L. Cohen | 1590 | 68 | 20 | 50 | 21 | .005 |
| David A. Pizarro | 227 | 71 | 20 | 69 | 21 | .005 |
| Ana Guinote | 378 | 76 | 20 | 47 | 21 | .005 |
| Kentaro Fujita | 458 | 69 | 20 | 62 | 21 | .005 |
| William A. Cunningham | 238 | 76 | 20 | 64 | 22 | .005 |
| Robert S. Wyer | 871 | 82 | 19 | 63 | 22 | .005 |
| Peter M. Gollwitzer | 1303 | 64 | 19 | 58 | 22 | .005 |
| Gerald L. Clore | 456 | 74 | 19 | 45 | 22 | .001 |
| Amy J. C. Cuddy | 170 | 81 | 19 | 72 | 22 | .005 |
| Nilanjana Dasgupta | 383 | 76 | 19 | 52 | 22 | .005 |
| Travis Proulx | 174 | 63 | 19 | 62 | 22 | .005 |
| James K. McNulty | 1047 | 56 | 19 | 65 | 23 | .005 |
| Dolores Albarracin | 520 | 67 | 19 | 56 | 23 | .005 |
| Richard P. Eibach | 753 | 69 | 19 | 47 | 23 | .001 |
| Kennon M. Sheldon | 698 | 74 | 18 | 66 | 23 | .005 |
| Wilhelm Hofmann | 624 | 67 | 18 | 66 | 23 | .005 |
| Ed Diener | 498 | 64 | 18 | 68 | 24 | .005 |
| Frank D. Fincham | 734 | 69 | 18 | 59 | 24 | .005 |
| Toni Schmader | 546 | 69 | 18 | 61 | 24 | .005 |
| Roland Deutsch | 365 | 78 | 18 | 71 | 24 | .005 |
| Lisa K. Libby | 418 | 65 | 18 | 54 | 24 | .005 |
| James M. Tyler | 130 | 87 | 18 | 74 | 24 | .005 |
| Chen-Bo Zhong | 327 | 68 | 18 | 49 | 25 | .005 |
| Brad J. Bushman | 897 | 74 | 17 | 62 | 25 | .005 |
| Ara Norenzayan | 225 | 72 | 17 | 61 | 25 | .005 |
| Benoit Monin | 635 | 65 | 17 | 56 | 25 | .005 |
| Michel Tuan Pham | 246 | 86 | 17 | 68 | 25 | .005 |
| E. Tory. Higgins | 1920 | 68 | 17 | 54 | 26 | .001 |
| Timothy D. Wilson | 798 | 65 | 17 | 63 | 26 | .005 |
| Ap Dijksterhuis | 750 | 68 | 17 | 54 | 26 | .005 |
| Michael W. Kraus | 617 | 72 | 17 | 55 | 26 | .005 |
| Carey K. Morewedge | 633 | 76 | 17 | 65 | 26 | .005 |
| Leandre R. Fabrigar | 632 | 70 | 17 | 67 | 26 | .005 |
| Joseph Cesario | 146 | 62 | 17 | 45 | 26 | .001 |
| Simone Schnall | 270 | 62 | 17 | 31 | 26 | .001 |
| Daniel T. Gilbert | 724 | 65 | 16 | 65 | 27 | .005 |
| Melissa J. Ferguson | 1163 | 72 | 16 | 69 | 27 | .005 |
| Charles S. Carver | 154 | 82 | 16 | 64 | 28 | .005 |
| Mark P. Zanna | 659 | 64 | 16 | 48 | 28 | .001 |
| Sandra L. Murray | 697 | 60 | 16 | 55 | 28 | .001 |
| Laura A. King | 391 | 76 | 16 | 68 | 29 | .005 |
| Heejung S. Kim | 858 | 59 | 16 | 55 | 29 | .001 |
| Gun R. Semin | 159 | 79 | 15 | 64 | 29 | .005 |
| Nathaniel M Lambert | 456 | 66 | 15 | 59 | 30 | .001 |
| Shelley E. Taylor | 438 | 69 | 15 | 54 | 31 | .001 |
| Nira Liberman | 1304 | 75 | 15 | 65 | 31 | .005 |
| Lee Ross | 349 | 77 | 14 | 63 | 31 | .001 |
| Ziva Kunda | 217 | 67 | 14 | 56 | 31 | .001 |
| Jon K. Maner | 1040 | 65 | 14 | 52 | 32 | .001 |
| Arie W. Kruglanski | 1228 | 78 | 14 | 58 | 33 | .001 |
| Gabriele Oettingen | 1047 | 61 | 14 | 49 | 33 | .001 |
| Gregory M. Walton | 587 | 69 | 14 | 44 | 33 | .001 |
| Sarah E. Hill | 509 | 78 | 13 | 52 | 34 | .001 |
| Fiona Lee | 221 | 67 | 13 | 58 | 34 | .001 |
| Michael A. Olson | 346 | 65 | 13 | 63 | 35 | .001 |
| Michael A. Zarate | 120 | 52 | 13 | 31 | 36 | .001 |
| Daniel M. Oppenheimer | 198 | 80 | 12 | 60 | 37 | .001 |
| Yaacov Trope | 1277 | 73 | 12 | 57 | 38 | .001 |
| Steven J. Spencer | 541 | 67 | 12 | 44 | 38 | .001 |
| Deborah A. Prentice | 89 | 80 | 12 | 57 | 38 | .001 |
| William von Hippel | 398 | 65 | 12 | 48 | 40 | .001 |
| Oscar Ybarra | 305 | 63 | 12 | 55 | 40 | .001 |
| Dov Cohen | 641 | 68 | 11 | 44 | 41 | .001 |
| Ian McGregor | 409 | 66 | 11 | 40 | 41 | .001 |
| Mark Muraven | 496 | 52 | 11 | 44 | 41 | .001 |
| Martie G. Haselton | 186 | 73 | 11 | 54 | 43 | .001 |
| Susan M. Andersen | 361 | 74 | 11 | 48 | 43 | .001 |
| Shelly Chaiken | 360 | 74 | 11 | 52 | 44 | .001 |
| Hans Ijzerman | 214 | 56 | 9 | 46 | 51 | .001 |
Only 801 of the listed 1260 effects were actually taken from research that I was involved in (some seem to stem from articles for which I was editor, others are a mystery to me). On the other hand, the majority of my research is missing. It seems preferable to publish data that is actually based on a more or less representative sample of research actually done by the person with whom that data is associated.
LikeLike
Thank you for the comment. They are valuable to improve the informativeness of the z-curve analyses.
1. only social/personalty journals and general journals like Psych Science were used (I posted a list of the journals).
I will make clear which journals were used.
2. I am trying to screen out mentions of names as editor, but the program is not perfect. I will look into this and update according.
3. I found a way to screen out more articles where your name appeared in footnotes (thank you).
4. I updated the results and they did improve.
5. Please check the new results.
LikeLike
Thank you for the quick response. Some of my research is published in psychophysiology or cognitive journals hence I now understand why so much is missing.
LikeLike
I figure that research practices can vary once physiological measures are taken or in cognitive studies with within-subject designs. I will eventually do similar posts for other areas.
LikeLike
I’m dismayed (and aghast) to see that I’m almost at the bottom of this list. Any advice on how to investigate this further to see where the problem lies?
LikeLike
Thank you for your comment.
You can download a file called “William von Hippel-rindex.csv”
It contains all the articles that were used and computes the R-Index based on the z-scores found for that article. The R-Index is a simple way to estimate replicability that works for small sets of test statistics. An R-Index of 50 would suggest that the replicability is about 50%. The EDR would be lower, but is hard to estimate with a small set of test statistics. The file is sorted by the R-Index. Articles with an R-Index below 50 are probably not robust. This is a good way to start diagnosing the problem.
LikeLike
Hi Uli, that’s very helpful – thanks!
But now I’m confused. To start with the worst offenders on my list, I have four papers with an R-Index of 0. I can’t tell what two of them are, as your identifier doesn’t include the article title or authors, but two of them are clear. The first of those two has large samples, reports a wide variety of large and small correlations, and strikes me as highly replicable. Indeed, study 2 (N=466) is a direct replication of study 1 (N=196) with an even larger sample. Study 3 goes in a slightly different direction, but mostly relies on the data from Study 2. The other paper reports large samples (Ns = 200) but small effects. We submitted it with only one study, the editor asked for replication, we ran a direct replication with the same sample size and found the same effect. Those are both in the paper. Since then we’ve tried to replicate it once and have succeeded (that finding isn’t yet published).
That’s the first issue, and strikes me as the most important. Secondarily, there are at least four or five papers in this list that aren’t my own – perhaps more but it’s hard to tell what some of the papers are – and the resultant list of papers is only about 1/3 of my empirical publications. Thus, setting aside the most important issue above, I don’t have a clear sense of what my actual replicability score would look like with all of my papers.
All the best, Bill
LikeLike
please check the number of results. Many papers with R-Index of 0 have only 1 result which is often just a missing value, meaning no results were found. So, you can ignore those.
LikeLiked by 1 person
I also made clear which journals were searched for these articles. Please see the list on the blog post.
LikeLiked by 1 person
I would also be happy to run an analysis on all of your articles, if you send me the pdfs.
LikeLiked by 1 person
There are numerous correlations reported in both papers, along with various mediational analyses in one of them, so definitely not a single result.
With regard to the second issue, the file lists the journal title and year, but that’s it. Sometimes I haven’t published in that journal in that year, so I know it’s not me. Sometimes I have, but in this particular case the only paper I published in that journal in that year has another one of the R = 0 examples, but includes a sample in the millions and a multiverse analysis. There’s no chance that could have a replicability index of 0.
LikeLiked by 1 person
Thanks Uli, very kind of you to offer to run the analysis for me. I’ve created a dropbox folder with all of my empirical articles in it and shared it with you. Let me know if that doesn’t come through. Best, Bill
LikeLiked by 1 person