Update: 2/27/2021
This post has been replaced by a new post. The rankings here are only shown for the sake of transparency, but scores have been replaced and the new scores should be used (https://replicationindex.com/2021/01/19/personalized-p-values/). The correlation between these scores and the new scores is r ~ .5. This is fairly low and I have been trying to figure out the reason for the discrepancy. However, I am not able to reproduce the results posted here. The problem is that I did not document the criteria for journal selection and I did not store the files with the list of articles for each author. Thus, these results are not reproducible. In contrast, the new results are reproducible and the data are openly shared to allow others to reproduce the results.

Original Post from November 8, 2018

Social psychology has a replication problem.  The reason is that social psychologists used questionable research practices to increase their chances of reporting significant results. The consequence is that the real risk of a false positive result is higher than the stated 5% level in publications. In other words, p < .05 no longer means that at most 5% of published results are false positives (Sterling, 1959). Another problem is that selection for significance with low power produces inflated effect sizes estimates. Estimates suggests that on average published effect sizes are inflated by 100% (OSC, 2015). These problems have persisted for decades (Sterling, 1959), but only now psychologists are recognizing that published results provide weak evidence and might not be replicable even if the same study were replicated exactly.

How should consumers of empirical social psychology (textbook writers, undergraduate students, policy planners) respond to the fact that published results cannot be trusted at face value? Jerry Brunner and I have been working on ways to correct published results for the inflation introduced by selection for significance and questionable practices.  Z-curve estimates the mean power of studies selected for significance.  Here I applied the method to automatically extracted test statistics from social psychology journals.  I computed z-curves for 70+ eminent social psychologists (H-index > 35).

The results can be used to evaluate the published results reported by individual researchers.  The main information provided in the table are (a) the replicability of all published p-values, (b) the replicability of just significant p-values (defined as p-values greater than pnorm(2.5) = .0124, and (c) the replicability of p-values with moderate evidence against the null-hypothesis (.0124 > p > .0027). More detailed information is provided in the z-curve plots (powergraphs) that are linked to researchers’ names. An index less than 50% would suggest that these p-values are no longer significant after adjusting for selection for significance.  As can be seen in the table, most just significant results are no longer significant after correction for bias.

Caveat: Interpret with Care

The results should not be overinterpreted. They are estimates based on an objective statistical procedure, but no statistical method can compensate perfectly for the various practices that led to the observed distribution of p-values (transformed into z-scores).  However, in the absence of any information which results can be trusted, these graphs provide some information.  How this information is used by consumers depends ultimately on consumers’ subjective beliefs.  Information about the average replicability of researchers’ published results may influence these beliefs.

It is also important to point out that a low replicability index does not mean researchers were committing scientific misconduct.  There are no clear guidelines about acceptable and unacceptable statistical practices in psychology.  Zcurve is not designed to detect scientific fraud. In fact, it assumes that researcher collect data, but conduct analyses in a way that increases the chances of producing a significant result.  The bias introduced by selection for significance is well known and considered acceptable in psychological science.

There are also many factors that can bias results in favor of researchers’ hypotheses without researchers’ awareness. Thus, the bias evident in many graphs does not imply that researchers intentionally manipulated data to support their claims. Thus, I attribute the bias to unidentified researcher influences.  It is not important to know how bias occurred. It is only important to detect biases and to correct for them.

It is necessary to do so for individual researchers because bias varies across researchers.  For example, the R-Index for all results ranges from 22% to 81%.  It would be unfair to treat all social psychologists alike when their research practices are a reliable moderator of replicability.  Providing personalized information about replicability allows consumers of social psychological research to avoid stereotyping social psychologists and to take individual differences in research practices into account.

Finally, it should be said that producing replicabilty estimates is subject to biases and errors.  Researchers may differ in their selection of hypotheses that they are reporting. A more informative analysis would require hand-coding of researchers’ focal hypothesis tests.  At the moment, R-Index does not have the resources to code all published results in social psychology, let alone other areas of psychology.  This is an important task for the future.  At the moment, automatically extracted results have some heuristic value.

One unintended and unfortunate consequence of making this information available is that some researchers’ reputation might be negatively effected by a low replicability score.  This cost has be be weighted against the benefit to the public and the scientific community of obtaining information about the robustness of published results.  In this regard, the replicability rankings are no different from actual replication studies that fail to replicate an original finding.  The only difference is that replicability rankings use all published results, whereas actual replication studies are often limited to a single or a few studies.  While replication failures in a single study are ambiguous, replicability esitmates based on hundreds of published results are more diagnostic of researchers’ practices.

Nevertheless, statistical estimates provide no definitive answer about the reproducibility of a published result.  Ideally, eminent researchers would conduct their own replication studies to demonstrate that their most important findings can be replicated under optimal conditions.

It is also important to point out that researchers have responded differently to the replication crisis that became apparent in 2011.   It may be unfair to generalize from past practices to new findings for researchers who changed their practices.  If researchers preregistered their studies and followed a well-designed registered research protocol, new results may be more robust than a researchers’ past record suggests.

Finally, the results show evidence of good replicability for some social psychologists.  Thus, the rankings avoid the problem of selectively targeting researchers with low replicability, which can lead to a negative bias in evaluations of social psychology.  The focus on researchers with a high H-index means that the results are representative of the field.

If you believe that you should not be listed as an eminent social psychologists, please contact me so that I can remove you from the list.

If you think you are an eminent social psychologists and you want to be included in the ranking, please contact me so that I can add you to the list.

If you have any suggestions or comments how I can make these rankings more informative, please let me know in the comments section.

***   ***   ***    ***    ***

REPLICABILITY RANKING OF EMINENT SOCIAL PSYCHOLOGISTS

[sorted by R-Index for all tests from highest to lowest rank]

Rank	Journal	R-Index (all)	R-Index (2.0-2.5)	R-Index (2.5-3.0)	#P-vals	H-Index	#Pub	#cit(*1000)
1	Steven J. Heine	81	44	55	197	41	83	10
2	James J. Gross	80	35	58	360	82	413	34
3	Constantine Sedikides	76	45	53	884	52	263	10
4	Bertram Gawronski	73	29	51	1717	37	113	6
5	Kathleen D. Vohs	73	36	53	452	49	158	11
6	Paul Rozin	73	39	57	155	65	218	16
7	Alice H. Eagly	72	42	50	384	61	161	18
8	Anthony G. Greenwald	72	28	51	273	64	175	26
9	David Dunning	72	30	50	674	40	105	8
10	Richard E. Nisbett	72	57	59	190	62	119	20
11	Shinobu Kitayama	70	37	44	545	35	103	15
12	Timothy D. Wilson	70	30	52	327	44	85	14
13	Mahzarin R. Banaji	68	52	53	651	60	133	22
14	Marilynn B. Brewer	68	43	46	193	49	107	15
15	Patricia G. Devine	67	44	52	1098	35	83	9
16	Susan T. Fiske	67	26	41	419	66	213	22
17	Thomas Gilovich	67	23	44	754	44	104	8
18	Daniel T. Gilbert	66	26	46	357	43	107	8
19	Hazel R. Markus	66	48	53	348	38	96	12
20	Mark P. Zanna	66	26	49	565	57	167	11
21	Wendy Wood	66	33	40	373	42	112	7
22	Brad J. Bushman	65	33	50	247	48	227	13
23	E. Tory. Higgins	65	27	44	887	72	274	25
24	Jeff Greenberg	65	30	42	679	60	154	13
25	Nira Liberman	64	25	37	1578	46	115	10
26	Caryl E. Rusbult	63	27	34	171	36	68	9
27	Dacher Keltner	63	38	43	903	60	159	15
28	Harry Reis	63	21	41	470	37	83	6
29	John F. Dovidio	63	28	37	2323	56	206	12
30	John T. Cacioppo	63	23	41	256	101	422	41
31	Nalini Ambady	63	24	42	675	51	171	10
32	Philip R. Shaver	63	24	42	675	60	184	13
33	Richard E. Petty	63	30	37	1428	69	190	20
34	Robert B. Cialdini	63	45	49	258	51	121	11
35	Michael Ross	63	38	51	631	42	245	7
36	Lee Ross	62	23	44	952	49	218	12
37	Roy F. Baumeister	62	32	44	1015	100	363	46
38	S. Alexander Haslam	62	34	41	289	52	234	9
39	Tom Pyszczynski	62	33	40	1101	60	149	13
40	Philip E. Tetlock	62	29	39	158	58	189	11
41	Arie W. Kruglanski	61	23	42	1140	50	228	13
42	Galen V. Bodenhausen	61	18	43	465	39	80	8
43	Norbert Schwarz	61	36	42	2524	49	138	13
44	Jonathan Haidt	60	16	37	98	42	84	14
45	Shelly Chaiken	60	15	36	288	46	86	10
46	Ap Dijksterhuis	59	17	36	456	42	118	8
47	Eddie Harmon-Jones	59	25	37	343	57	212	10
48	Fritz Strack	59	22	43	588	51	149	11
49	Joseph P. Forgas	59	29	43	176	37	142	4
50	Yaacov Trope	59	20	39	1957	57	135	12
51	Charles M. Judd	59	28	33	666	37	142	6
52	Craig A. Anderson	59	26	34	265	51	117	11
53	C. Nathan DeWall	58	30	41	1099	37	135	7
54	Eli J. Finkel	58	24	35	1921	38	109	4
55	Jeffry A. Simpson	58	18	28	261	42	95	7
56	Peter M. Gollwitzer	58	29	44	1711	45	158	11
57	Mario Mikulincer	58	29	34	104	68	272	15
58	Russell H. Fazio	57	23	29	464	51	134	14
59	Thomas Mussweiler	57	24	38	1128	35	85	5
60	Daniel M. Wegner	56	27	36	699	53	130	14
61	John A. Bargh	56	20	36	755	61	140	22
62	Robert S. Wyer	56	33	44	283	40	203	8
63	Carol S. Dweck	54	17	29	458	59	147	20
64	Michael Inzlicht	54	32	36	156	36	124	5
65	John T. Jost	52	21	33	249	46	132	11
66	Shelly E. Taylor	51	15	28	198	73	169	28
67	Claude M. Steele	48	38	42	376	29	49	13
68	Gerald L. Clore	45	20	35	200	37	84	9
69	Adam D. Galinsky	43	24	28	585	57	198	11
70	Robert Zajonc	39	11	26	67	40	114	16
71	Jennifer Crocker	22	5	5	99	38	68	7