Social psychology has a replication problem. The reason is that social psychologists used questionable research practices to increase their chances of reporting significant results. The consequence is that the real risk of a false positive result is higher than the stated 5% level in publications. In other words, p < .05 no longer means that at most 5% of published results are false positives (Sterling, 1959). Another problem is that selection for significance with low power produces inflated effect sizes estimates. Estimates suggests that on average published effect sizes are inflated by 100% (OSC, 2015). These problems have persisted for decades (Sterling, 1959), but only now psychologists are recognizing that published results provide weak evidence and might not be replicable even if the same study were replicated exactly.
How should consumers of empirical social psychology (textbook writers, undergraduate students, policy planners) respond to the fact that published results cannot be trusted at face value? Jerry Brunner and I have been working on ways to correct published results for the inflation introduced by selection for significance and questionable practices. Z-curve estimates the mean power of studies selected for significance. Here I applied the method to automatically extracted test statistics from social psychology journals. I computed z-curves for 70+ eminent social psychologists (H-index > 35).
The results can be used to evaluate the published results reported by individual researchers. The main information provided in the table are (a) the replicability of all published p-values, (b) the replicability of just significant p-values (defined as p-values greater than pnorm(2.5) = .0124, and (c) the replicability of p-values with moderate evidence against the null-hypothesis (.0124 > p > .0027). More detailed information is provided in the z-curve plots (powergraphs) that are linked to researchers’ names. An index less than 50% would suggest that these p-values are no longer significant after adjusting for selection for significance. As can be seen in the table, most just significant results are no longer significant after correction for bias.
Caveat: Interpret with Care
The results should not be overinterpreted. They are estimates based on an objective statistical procedure, but no statistical method can compensate perfectly for the various practices that led to the observed distribution of p-values (transformed into z-scores). However, in the absence of any information which results can be trusted, these graphs provide some information. How this information is used by consumers depends ultimately on consumers’ subjective beliefs. Information about the average replicability of researchers’ published results may influence these beliefs.
It is also important to point out that a low replicability index does not mean researchers were committing scientific misconduct. There are no clear guidelines about acceptable and unacceptable statistical practices in psychology. Zcurve is not designed to detect scientific fraud. In fact, it assumes that researcher collect data, but conduct analyses in a way that increases the chances of producing a significant result. The bias introduced by selection for significance is well known and considered acceptable in psychological science.
There are also many factors that can bias results in favor of researchers’ hypotheses without researchers’ awareness. Thus, the bias evident in many graphs does not imply that researchers intentionally manipulated data to support their claims. Thus, I attribute the bias to unidentified researcher influences. It is not important to know how bias occurred. It is only important to detect biases and to correct for them.
It is necessary to do so for individual researchers because bias varies across researchers. For example, the R-Index for all results ranges from 22% to 81%. It would be unfair to treat all social psychologists alike when their research practices are a reliable moderator of replicability. Providing personalized information about replicability allows consumers of social psychological research to avoid stereotyping social psychologists and to take individual differences in research practices into account.
Finally, it should be said that producing replicabilty estimates is subject to biases and errors. Researchers may differ in their selection of hypotheses that they are reporting. A more informative analysis would require hand-coding of researchers’ focal hypothesis tests. At the moment, R-Index does not have the resources to code all published results in social psychology, let alone other areas of psychology. This is an important task for the future. At the moment, automatically extracted results have some heuristic value.
One unintended and unfortunate consequence of making this information available is that some researchers’ reputation might be negatively effected by a low replicability score. This cost has be be weighted against the benefit to the public and the scientific community of obtaining information about the robustness of published results. In this regard, the replicability rankings are no different from actual replication studies that fail to replicate an original finding. The only difference is that replicability rankings use all published results, whereas actual replication studies are often limited to a single or a few studies. While replication failures in a single study are ambiguous, replicability esitmates based on hundreds of published results are more diagnostic of researchers’ practices.
Nevertheless, statistical estimates provide no definitive answer about the reproducibility of a published result. Ideally, eminent researchers would conduct their own replication studies to demonstrate that their most important findings can be replicated under optimal conditions.
It is also important to point out that researchers have responded differently to the replication crisis that became apparent in 2011. It may be unfair to generalize from past practices to new findings for researchers who changed their practices. If researchers preregistered their studies and followed a well-designed registered research protocol, new results may be more robust than a researchers’ past record suggests.
Finally, the results show evidence of good replicability for some social psychologists. Thus, the rankings avoid the problem of selectively targeting researchers with low replicability, which can lead to a negative bias in evaluations of social psychology. The focus on researchers with a high H-index means that the results are representative of the field.
If you believe that you should not be listed as an eminent social psychologists, please contact me so that I can remove you from the list.
If you think you are an eminent social psychologists and you want to be included in the ranking, please contact me so that I can add you to the list.
If you have any suggestions or comments how I can make these rankings more informative, please let me know in the comments section.
*** *** *** *** ***
REPLICABILITY RANKING OF EMINENT SOCIAL PSYCHOLOGISTS
[sorted by R-Index for all tests from highest to lowest rank]
|Rank||Journal||R-Index (all)||R-Index (2.0-2.5)||R-Index (2.5-3.0)||#P-vals||H-Index||#Pub||#cit(*1000)|
|1||Steven J. Heine||81||44||55||197||41||83||10|
|2||James J. Gross||80||35||58||360||82||413||34|
|5||Kathleen D. Vohs||73||36||53||452||49||158||11|
|7||Alice H. Eagly||72||42||50||384||61||161||18|
|8||Anthony G. Greenwald||72||28||51||273||64||175||26|
|10||Richard E. Nisbett||72||57||59||190||62||119||20|
|12||Timothy D. Wilson||70||30||52||327||44||85||14|
|13||Mahzarin R. Banaji||68||52||53||651||60||133||22|
|14||Marilynn B. Brewer||68||43||46||193||49||107||15|
|15||Patricia G. Devine||67||44||52||1098||35||83||9|
|16||Susan T. Fiske||67||26||41||419||66||213||22|
|18||Daniel T. Gilbert||66||26||46||357||43||107||8|
|19||Hazel R. Markus||66||48||53||348||38||96||12|
|20||Mark P. Zanna||66||26||49||565||57||167||11|
|22||Brad J. Bushman||65||33||50||247||48||227||13|
|23||E. Tory. Higgins||65||27||44||887||72||274||25|
|26||Caryl E. Rusbult||63||27||34||171||36||68||9|
|29||John F. Dovidio||63||28||37||2323||56||206||12|
|30||John T. Cacioppo||63||23||41||256||101||422||41|
|32||Philip R. Shaver||63||24||42||675||60||184||13|
|33||Richard E. Petty||63||30||37||1428||69||190||20|
|34||Robert B. Cialdini||63||45||49||258||51||121||11|
|37||Roy F. Baumeister||62||32||44||1015||100||363||46|
|38||S. Alexander Haslam||62||34||41||289||52||234||9|
|40||Philip E. Tetlock||62||29||39||158||58||189||11|
|41||Arie W. Kruglanski||61||23||42||1140||50||228||13|
|42||Galen V. Bodenhausen||61||18||43||465||39||80||8|
|49||Joseph P. Forgas||59||29||43||176||37||142||4|
|51||Charles M. Judd||59||28||33||666||37||142||6|
|52||Craig A. Anderson||59||26||34||265||51||117||11|
|53||C. Nathan DeWall||58||30||41||1099||37||135||7|
|54||Eli J. Finkel||58||24||35||1921||38||109||4|
|55||Jeffry A. Simpson||58||18||28||261||42||95||7|
|56||Peter M. Gollwitzer||58||29||44||1711||45||158||11|
|58||Russell H. Fazio||57||23||29||464||51||134||14|
|60||Daniel M. Wegner||56||27||36||699||53||130||14|
|61||John A. Bargh||56||20||36||755||61||140||22|
|62||Robert S. Wyer||56||33||44||283||40||203||8|
|63||Carol S. Dweck||54||17||29||458||59||147||20|
|65||John T. Jost||52||21||33||249||46||132||11|
|66||Shelly E. Taylor||51||15||28||198||73||169||28|
|67||Claude M. Steele||48||38||42||376||29||49||13|
|68||Gerald L. Clore||45||20||35||200||37||84||9|
|69||Adam D. Galinsky||43||24||28||585||57||198||11|