Personalized P-Values for Social/Personality Psychologists

January 19, 2021Alpha Wars, False Discovery Rate, Replicability, Replicability Ranking, Science-Wise False Discovery Rate, Social PsychologyUlrich Schimmack

Last update 8/25/2021
(expanded to 410 social/personality psychologists; included Dan Ariely)

Introduction

Since Fisher invented null-hypothesis significance testing, researchers have used p < .05 as a statistical criterion to interpret results as discoveries worthwhile of discussion (i.e., the null-hypothesis is false). Once published, these results are often treated as real findings even though alpha does not control the risk of false discoveries.

Statisticians have warned against the exclusive reliance on p < .05, but nearly 100 years after Fisher popularized this approach, it is still the most common way to interpret data. The main reason is that many attempts to improve on this practice have failed. The main problem is that a single statistical result is difficult to interpret. However, when individual results are interpreted in the context of other results, they become more informative. Based on the distribution of p-values it is possible to estimate the maximum false discovery rate (Bartos & Schimmack, 2020; Jager & Leek, 2014). This approach can be applied to the p-values published by individual authors to adjust p-values to keep the risk of false discoveries at a reasonable level, FDR < .05.

Researchers who mainly test true hypotheses with high power have a high discovery rate (many p-values below .05) and a low false discovery rate (FDR < .05). Figure 1 shows an example of a researcher who followed this strategy (for a detailed description of z-curve plots, see Schimmack, 2021).

We see that out of the 317 test-statistics retrieved from his articles, 246 were significant with alpha = .05. This is an observed discovery rate of 78%. We also see that this discovery rate closely matches the estimated discovery rate based on the distribution of the significant p-values, p < .05. The EDR is 79%. With an EDR of 79%, the maximum false discovery rate is only 1%. However, the 95%CI is wide and the lower bound of the CI for the EDR, 27%, allows for 14% false discoveries.

When the ODR matches the EDR, there is no evidence of publication bias. In this case, we can improve the estimates by fitting all p-values, including the non-significant ones. With a tighter CI for the EDR, we see that the 95%CI for the maximum FDR ranges from 1% to 3%. Thus, we can be confident that no more than 5% of the significant results wit alpha = .05 are false discoveries. Readers can therefore continue to use alpha = .05 to look for interesting discoveries in Matsumoto’s articles.

Figure 3 shows the results for a different type of researcher who took a risk and studied weak effect sizes with small samples. This produces many non-significant results that are often not published. The selection for significance inflates the observed discovery rate, but the z-curve plot and the comparison with the EDR shows the influence of publication bias. Here the ODR is similar to Figure 1, but the EDR is only 11%. An EDR of 11% translates into a large maximum false discovery rate of 41%. In addition, the 95%CI of the EDR includes 5%, which means the risk of false positives could be as high as 100%. In this case, using alpha = .05 to interpret results as discoveries is very risky. Clearly, p < .05 means something very different when reading an article by David Matsumoto or Shelly Chaiken.

Rather than dismissing all of Chaiken’s results, we can try to lower alpha to reduce the false discovery rate. If we set alpha = .01, the FDR is 15%. If we set alpha = .005, the FDR is 8%. To get the FDR below 5%, we need to set alpha to .001.

A uniform criterion of FDR < 5% is applied to all researchers in the rankings below. For some this means no adjustment to the traditional criterion. For others, alpha is lowered to .01, and for a few even lower than that.

The rankings below are based on automatrically extracted test-statistics from 40 journals (List of journals). The results should be interpreted with caution and treated as preliminary. They depend on the specific set of journals that were searched, the way results are being reported, and many other factors. The data are available (data.drop) and researchers can exclude articles or add articles and run their own analyses using the z-curve package in R (https://replicationindex.com/2020/01/10/z-curve-2-0/).

I am also happy to receive feedback about coding errors. I also recommended to hand-code articles to adjust alpha for focal hypothesis tests. This typically lowers the EDR and increases the FDR. For example, the automated method produced an EDR of 31 for Bargh, whereas hand-coding of focal tests produced an EDR of 12 (Bargh-Audit).

And here are the rankings. The results are fully automated and I was not able to cover up the fact that I placed only #188 out of 400 in the rankings. In another post, I will explain how researchers can move up in the rankings. Of course, one way to move up in the rankings is to increase statistical power in future studies. The rankings will be updated again when the 2021 data are available.

Despite the preliminary nature, I am confident that the results provide valuable information. Until know all p-values below .05 have been treated as if they are equally informative. The rankings here show that this is not the case. While p = .02 can be informative for one researcher, p = .002 may still entail a high false discovery risk for another researcher.

Good science requires not only open and objective reporting of new data; it also requires unbiased review of the literature. However, there are no rules and regulations regarding citations, and many authors cherry-pick citations that are consistent with their claims. Even when studies have failed to replicate, original studies are cited without citing the replication failures. In some cases, authors even cite original articles that have been retracted. Fortunately, it is easy to spot these acts of unscientific behavior. Here I am starting a project to list examples of bad scientific behaviors. Hopefully, more scientists will take the time to hold their colleagues accountable for ethical behavior in citations. They can even do so by posting anonymously on the PubPeer comment site.

Rank	Name	Tests	ODR	EDR	ERR	FDR	Alpha
1	Robert A. Emmons	53	87	89	90	1	.05
2	Allison L. Skinner	229	59	81	85	1	.05
3	David Matsumoto	378	83	79	85	1	.05
4	Linda J. Skitka	532	68	75	82	2	.05
5	Todd K. Shackelford	305	77	75	82	2	.05
6	Jonathan B. Freeman	274	59	75	81	2	.05
7	Virgil Zeigler-Hill	515	72	74	81	2	.05
8	Arthur A. Stone	310	75	73	81	2	.05
9	David P. Schmitt	207	78	71	77	2	.05
10	Emily A. Impett	549	77	70	76	2	.05
11	Paula Bressan	62	82	70	76	2	.05
12	Kurt Gray	487	79	69	81	2	.05
13	Michael E. McCullough	334	69	69	78	2	.05
14	Kipling D. Williams	843	75	69	77	2	.05
15	John M. Zelenski	156	71	69	76	2	.05
16	Elke U. Weber	312	69	68	77	0	.05
17	Hilary B. Bergsieker	439	67	68	74	2	.05
18	Cameron Anderson	652	71	67	74	3	.05
19	Rachael E. Jack	249	70	66	80	3	.05
20	Jamil Zaki	430	78	66	76	3	.05
21	A. Janet Tomiyama	76	78	65	76	3	.05
22	Benjamin R. Karney	392	56	65	73	3	.05
23	Phoebe C. Ellsworth	605	74	65	72	3	.05
24	Jim Sidanius	487	69	65	72	3	.05
25	Amelie Mummendey	461	70	65	72	3	.05
26	Carol D. Ryff	280	84	64	76	3	.05
27	Juliane Degner	435	63	64	71	3	.05
28	Steven J. Heine	597	78	63	77	3	.05
29	David M. Amodio	584	66	63	70	3	.05
30	Thomas N Bradbury	398	61	63	69	3	.05
31	Elaine Fox	472	79	62	78	3	.05
32	Miles Hewstone	1427	70	62	73	3	.05
33	Linda R. Tropp	344	65	61	80	3	.05
34	Rainer Greifeneder	944	75	61	77	3	.05
35	Klaus Fiedler	1950	77	61	74	3	.05
36	Jesse Graham	377	70	60	76	3	.05
37	Richard W. Robins	270	76	60	70	4	.05
38	Simine Vazire	137	66	60	64	4	.05
39	On Amir	267	67	59	88	4	.05
40	Edward P. Lemay	289	87	59	81	4	.05
41	William B. Swann Jr.	1070	78	59	80	4	.05
42	Margaret S. Clark	505	75	59	77	4	.05
43	Bernhard Leidner	724	64	59	65	4	.05
44	B. Keith Payne	879	71	58	76	4	.05
45	Ximena B. Arriaga	284	66	58	69	4	.05
46	Joris Lammers	728	69	58	69	4	.05
47	Patricia G. Devine	606	71	58	67	4	.05
48	Rainer Reisenzein	201	65	57	69	4	.05
49	Barbara A. Mellers	287	80	56	78	4	.05
50	Joris Lammers	705	69	56	69	4	.05
51	Jean M. Twenge	381	72	56	59	4	.05
52	Nicholas Epley	1504	74	55	72	4	.05
53	Kaiping Peng	566	77	54	75	4	.05
54	Krishna Savani	638	71	53	69	5	.05
55	Leslie Ashburn-Nardo	109	80	52	83	5	.05
56	Lee Jussim	226	80	52	71	5	.05
57	Richard M. Ryan	998	78	52	69	5	.05
58	Ethan Kross	614	66	52	67	5	.05
59	Edward L. Deci	284	79	52	63	5	.05
60	Roger Giner-Sorolla	663	81	51	80	5	.05
61	Bertram F. Malle	422	73	51	75	5	.05
62	Jens B. Asendorpf	253	74	51	69	5	.05
63	Samuel D. Gosling	108	58	51	62	5	.05
64	Tessa V. West	691	71	51	59	5	.05
65	Paul Rozin	449	78	50	84	5	.05
66	Joachim I. Krueger	436	78	50	81	5	.05
67	Sheena S. Iyengar	207	63	50	80	5	.05
68	James J. Gross	1104	72	50	77	5	.05
69	Mark Rubin	306	68	50	75	5	.05
70	Pieter Van Dessel	578	70	50	75	5	.05
71	Shinobu Kitayama	983	76	50	71	5	.05
72	Matthew J. Hornsey	1656	74	50	71	5	.05
73	Janice R. Kelly	366	75	50	70	5	.05
74	Antonio L. Freitas	247	79	50	64	5	.05
75	Paul K. Piff	166	77	50	63	5	.05
76	Mina Cikara	392	71	49	80	5	.05
77	Beate Seibt	379	72	49	62	6	.01
78	Ludwin E. Molina	163	69	49	61	5	.05
79	Bertram Gawronski	1803	72	48	76	6	.01
80	Penelope Lockwood	458	71	48	70	6	.01
81	Edward R. Hirt	1042	81	48	65	6	.01
82	Matthew D. Lieberman	398	72	47	80	6	.01
83	John T. Cacioppo	438	76	47	69	6	.01
84	Agneta H. Fischer	952	75	47	69	6	.01
85	Leaf van Boven	711	72	47	67	6	.01
86	Stephanie A. Fryberg	248	62	47	66	6	.01
87	Daniel M. Wegner	602	76	47	65	6	.01
88	Anne E. Wilson	785	71	47	64	6	.01
89	Rainer Banse	402	78	46	72	6	.01
90	Alice H. Eagly	330	75	46	71	6	.01
91	Jeanne L. Tsai	1241	73	46	67	6	.01
92	Jennifer S. Lerner	181	80	46	61	6	.01
93	Andrea L. Meltzer	549	52	45	72	6	.01
94	R. Chris Fraley	642	70	45	72	7	.01
95	Constantine Sedikides	2566	71	45	70	6	.01
96	Paul Slovic	377	74	45	70	6	.01
97	Dacher Keltner	1233	72	45	64	6	.01
98	Brian A. Nosek	816	68	44	81	7	.01
99	George Loewenstein	752	71	44	72	7	.01
100	Ursula Hess	774	78	44	71	7	.01
101	Jason P. Mitchell	600	73	43	73	7	.01
102	Jessica L. Tracy	632	74	43	71	7	.01
103	Charles M. Judd	1054	76	43	68	7	.01
104	S. Alexander Haslam	1198	72	43	64	7	.01
105	Mark Schaller	565	73	43	61	7	.01
106	Susan T. Fiske	911	78	42	74	7	.01
107	Lisa Feldman Barrett	644	69	42	70	7	.01
108	Jolanda Jetten	1956	73	42	67	7	.01
109	Mario Mikulincer	901	89	42	64	7	.01
110	Bernadette Park	973	77	42	64	7	.01
111	Paul A. M. Van Lange	1092	70	42	63	7	.01
112	Wendi L. Gardner	798	67	42	63	7	.01
113	Will M. Gervais	110	69	42	59	7	.01
114	Jordan B. Peterson	266	60	41	79	7	.01
115	Philip E. Tetlock	549	79	41	73	7	.01
116	Amanda B. Diekman	438	83	41	70	7	.01
117	Daniel H. J. Wigboldus	492	76	41	67	8	.01
118	Michael Inzlicht	686	66	41	63	8	.01
119	Naomi Ellemers	2388	74	41	63	8	.01
120	Phillip Atiba Goff	299	68	41	62	7	.01
121	Stacey Sinclair	327	70	41	57	8	.01
122	Francesca Gino	2521	75	40	69	8	.01
123	Michael I. Norton	1136	71	40	69	8	.01
124	David J. Hauser	156	74	40	68	8	.01
125	Elizabeth Page-Gould	411	57	40	66	8	.01
126	Tiffany A. Ito	349	80	40	64	8	.01
127	Richard E. Petty	2771	69	40	64	8	.01
128	Tim Wildschut	1374	73	40	64	8	.01
129	Norbert Schwarz	1337	72	40	63	8	.01
130	Veronika Job	362	70	40	63	8	.01
131	Wendy Wood	462	75	40	62	8	.01
132	Minah H. Jung	156	83	39	83	8	.01
133	Marcel Zeelenberg	868	76	39	79	8	.01
134	Tobias Greitemeyer	1737	72	39	67	8	.01
135	Jason E. Plaks	582	70	39	67	8	.01
136	Carol S. Dweck	1028	70	39	63	8	.01
137	Christian S. Crandall	362	75	39	59	8	.01
138	Harry T. Reis	998	69	38	74	9	.01
139	Vanessa K. Bohns	420	77	38	74	8	.01
140	Jerry Suls	413	71	38	68	8	.01
141	Eric D. Knowles	384	68	38	64	8	.01
142	C. Nathan DeWall	1336	73	38	63	9	.01
143	Clayton R. Critcher	697	82	38	63	9	.01
144	John F. Dovidio	2019	69	38	62	9	.01
145	Joshua Correll	549	61	38	62	9	.01
146	Abigail A. Scholer	556	58	38	62	9	.01
147	Chris Janiszewski	107	81	38	58	9	.01
148	Herbert Bless	586	73	38	57	9	.01
149	Mahzarin R. Banaji	880	73	37	78	9	.01
150	Rolf Reber	280	64	37	72	9	.01
151	Kevin N. Ochsner	406	79	37	70	9	.01
152	Mark J. Brandt	277	70	37	70	9	.01
153	Geoff MacDonald	406	67	37	67	9	.01
154	Mara Mather	1038	78	37	67	9	.01
155	Antony S. R. Manstead	1656	72	37	62	9	.01
156	Lorne Campbell	433	67	37	61	9	.01
157	Sanford E. DeVoe	236	71	37	61	9	.01
158	Ayelet Fishbach	1416	78	37	59	9	.01
159	Fritz Strack	607	75	37	56	9	.01
160	Jeff T. Larsen	181	74	36	67	10	.01
161	Nyla R. Branscombe	1276	70	36	65	9	.01
162	Yaacov Schul	411	61	36	64	9	.01
163	D. S. Moskowitz	3418	74	36	63	9	.01
164	Pablo Brinol	1356	67	36	62	9	.01
165	Todd B. Kashdan	377	73	36	61	9	.01
166	Barbara L. Fredrickson	287	72	36	61	9	.01
167	Duane T. Wegener	980	77	36	60	9	.01
168	Joanne V. Wood	1093	74	36	60	9	.01
169	Daniel A. Effron	484	66	36	60	9	.01
170	Niall Bolger	376	67	36	58	9	.01
171	Craig A. Anderson	467	76	36	55	9	.01
172	Michael Harris Bond	378	73	35	84	10	.01
173	Glenn Adams	270	71	35	73	10	.01
174	Daniel M. Bernstein	404	73	35	70	10	.01
175	C. Miguel Brendl	121	76	35	68	10	.01
176	Azim F. Sharif	183	74	35	68	10	.01
177	Emily Balcetis	599	69	35	68	10	.01
178	Eva Walther	493	82	35	66	10	.01
179	Michael D. Robinson	1388	78	35	66	10	.01
180	Igor Grossmann	203	64	35	66	10	.01
181	Diana I. Tamir	156	62	35	62	10	.01
182	Samuel L. Gaertner	321	75	35	61	10	.01
183	John T. Jost	794	70	35	61	10	.01
184	Eric L. Uhlmann	457	67	35	61	10	.01
185	Nalini Ambady	1256	62	35	56	10	.01
186	Daphna Oyserman	446	55	35	54	10	.01
187	Victoria M. Esses	295	75	35	53	10	.01
188	Linda J. Levine	495	74	34	78	10	.01
189	Wiebke Bleidorn	99	63	34	74	10	.01
190	Thomas Gilovich	1193	80	34	69	10	.01
191	Alexander J. Rothman	133	69	34	65	10	.01
192	Francis J. Flynn	378	72	34	63	10	.01
193	Paula M. Niedenthal	522	69	34	61	10	.01
194	Ozlem Ayduk	549	62	34	59	10	.01
195	Paul Ekman	88	70	34	55	10	.01
196	Alison Ledgerwood	214	75	34	54	10	.01
197	Christopher R. Agnew	325	75	33	76	10	.01
198	Michelle N. Shiota	242	60	33	63	11	.01
199	Malte Friese	501	61	33	57	11	.01
200	Kerry Kawakami	487	68	33	56	10	.01
201	Danu Anthony Stinson	494	77	33	54	11	.01
202	Jennifer A. Richeson	831	67	33	52	11	.01
203	Margo J. Monteith	773	76	32	77	11	.01
204	Ulrich Schimmack	318	75	32	63	11	.01
205	Mark Snyder	562	72	32	63	11	.01
206	Russell H. Fazio	1094	69	32	61	11	.01
207	Eric van Dijk	238	67	32	60	11	.01
208	Tom Meyvis	377	77	32	60	11	.01
209	Eli J. Finkel	1392	62	32	57	11	.01
210	Robert B. Cialdini	379	72	32	56	11	.01
211	Jonathan W. Kunstman	430	66	32	53	11	.01
212	Delroy L. Paulhus	121	77	31	82	12	.01
213	Yuen J. Huo	132	74	31	80	11	.01
214	Gerd Bohner	513	71	31	70	11	.01
215	Christopher K. Hsee	689	75	31	63	11	.01
216	Vivian Zayas	251	71	31	60	12	.01
217	John A. Bargh	651	72	31	55	12	.01
218	Tom Pyszczynski	948	69	31	54	12	.01
219	Roy F. Baumeister	2442	69	31	52	12	.01
220	E. Ashby Plant	831	77	31	51	11	.01
221	Kathleen D. Vohs	944	68	31	51	12	.01
222	Jamie Arndt	1318	69	31	50	12	.01
223	Anthony G. Greenwald	357	72	30	83	12	.01
224	Nicholas O. Rule	1294	68	30	75	13	.01
225	Lauren J. Human	447	59	30	70	12	.01
226	Jennifer Crocker	515	68	30	67	12	.01
227	Dale T. Miller	521	71	30	64	12	.01
228	Thomas W. Schubert	353	70	30	60	12	.01
229	W. Keith Campbell	528	70	30	58	12	.01
230	Arthur Aron	307	65	30	56	12	.01
231	Pamela K. Smith	149	66	30	52	12	.01
232	Aaron C. Kay	1320	70	30	51	12	.01
233	Steven W. Gangestad	198	63	30	41	13	.005
234	Eliot R. Smith	445	79	29	73	13	.01
235	Nir Halevy	262	68	29	72	13	.01
236	E. Allan Lind	370	82	29	72	13	.01
237	Richard E. Nisbett	319	73	29	69	13	.01
238	Hazel Rose Markus	674	76	29	68	13	.01
239	Emanuele Castano	445	69	29	65	13	.01
240	Dirk Wentura	830	65	29	64	13	.01
241	Boris Egloff	274	81	29	58	13	.01
242	Monica Biernat	813	77	29	57	13	.01
243	Gordon B. Moskowitz	374	72	29	57	13	.01
244	Russell Spears	2286	73	29	55	13	.01
245	Jeff Greenberg	1358	77	29	54	13	.01
246	Caryl E. Rusbult	218	60	29	54	13	.01
247	Naomi I. Eisenberger	179	74	28	79	14	.01
248	Brent W. Roberts	562	72	28	77	14	.01
249	Yoav Bar-Anan	525	75	28	76	13	.01
250	Eddie Harmon-Jones	738	73	28	70	14	.01
251	Matthew Feinberg	295	77	28	69	14	.01
252	Roland Neumann	258	77	28	67	13	.01
253	Eugene M. Caruso	822	75	28	64	13	.01
254	Ulrich Kuehnen	822	75	28	64	13	.01
255	Elizabeth W. Dunn	395	75	28	64	14	.01
256	Jeffry A. Simpson	697	74	28	55	13	.01
257	Sander L. Koole	767	65	28	52	14	.01
258	Richard J. Davidson	380	64	28	51	14	.01
259	Shelly L. Gable	364	64	28	50	14	.01
260	Adam D. Galinsky	2154	70	28	49	13	.01
261	Grainne M. Fitzsimons	585	68	28	49	14	.01
262	Geoffrey J. Leonardelli	290	68	28	48	14	.005
263	Joshua Aronson	183	85	28	46	14	.005
264	Henk Aarts	1003	67	28	45	14	.005
265	Vanessa K. Bohns	422	76	27	74	15	.01
266	Jan De Houwer	1972	70	27	72	14	.01
267	Dan Ariely	600	70	27	69	14	.01
268	Charles Stangor	185	81	27	68	15	.01
269	Karl Christoph Klauer	801	67	27	65	14	.01
270	Mario Gollwitzer	500	58	27	62	14	.01
271	Jennifer S. Beer	80	56	27	54	14	.01
272	Eldar Shafir	107	78	27	51	14	.01
273	Guido H. E. Gendolla	422	76	27	47	14	.005
274	Klaus R. Scherer	467	83	26	78	15	.01
275	William G. Graziano	532	71	26	66	15	.01
276	Galen V. Bodenhausen	585	74	26	61	15	.01
277	Sonja Lyubomirsky	530	71	26	59	15	.01
278	Kai Sassenberg	872	71	26	56	15	.01
279	Kristin Laurin	648	63	26	51	15	.01
280	Claude M. Steele	434	73	26	42	15	.005
281	David G. Rand	392	70	25	81	15	.01
282	Paul Bloom	502	72	25	79	16	.01
283	Kerri L. Johnson	532	76	25	76	15	.01
284	Batja Mesquita	416	71	25	73	16	.01
285	Rebecca J. Schlegel	261	67	25	71	15	.01
286	Phillip R. Shaver	566	81	25	71	16	.01
287	David Dunning	818	74	25	70	16	.01
288	Laurie A. Rudman	482	72	25	68	16	.01
289	David A. Lishner	105	65	25	63	16	.01
290	Mark J. Landau	950	78	25	45	16	.005
291	Ronald S. Friedman	183	79	25	44	16	.005
292	Joel Cooper	257	72	25	39	16	.005
293	Alison L. Chasteen	223	68	24	69	16	.01
294	Jeff Galak	313	73	24	68	17	.01
295	Steven J. Sherman	888	74	24	62	16	.01
296	Shigehiro Oishi	1109	64	24	61	17	.01
297	Thomas Mussweiler	604	70	24	43	17	.005
298	Mark W. Baldwin	247	72	24	41	17	.005
299	Evan P. Apfelbaum	256	62	24	41	17	.005
300	Nurit Shnabel	564	76	23	78	18	.01
301	Klaus Rothermund	738	71	23	76	18	.01
302	Felicia Pratto	410	73	23	75	18	.01
303	Jonathan Haidt	368	76	23	73	17	.01
304	Roland Imhoff	365	74	23	73	18	.01
305	Jeffrey W Sherman	992	68	23	71	17	.01
306	Jennifer L. Eberhardt	202	71	23	62	18	.005
307	Bernard A. Nijstad	693	71	23	52	18	.005
308	Brandon J. Schmeichel	652	66	23	45	17	.005
309	Sam J. Maglio	325	72	23	42	17	.005
310	David M. Buss	461	82	22	80	19	.01
311	Yoel Inbar	280	67	22	71	19	.01
312	Serena Chen	865	72	22	67	19	.005
313	Spike W. S. Lee	145	68	22	64	19	.005
314	Marilynn B. Brewer	314	75	22	62	18	.005
315	Michael Ross	1164	70	22	62	18	.005
316	Dieter Frey	1538	68	22	58	18	.005
317	G. Daniel Lassiter	189	82	22	55	19	.01
318	Sean M. McCrea	584	73	22	54	19	.005
319	Wendy Berry Mendes	965	68	22	44	19	.005
320	Paul W. Eastwick	583	65	21	69	19	.005
321	Kees van den Bos	1150	84	21	69	20	.005
322	Maya Tamir	1342	80	21	64	19	.005
323	Joseph P. Forgas	888	83	21	59	19	.005
324	Michaela Wanke	362	74	21	59	19	.005
325	Dolores Albarracin	540	66	21	56	20	.005
326	Elizabeth Levy Paluck	31	84	21	55	20	.005
327	Vanessa LoBue	299	68	20	76	21	.01
328	Christopher J. Armitage	160	62	20	73	21	.005
329	Elizabeth A. Phelps	686	78	20	72	21	.005
330	Jay J. van Bavel	437	64	20	71	21	.005
331	David A. Pizarro	227	71	20	69	21	.005
332	Andrew J. Elliot	1018	81	20	67	21	.005
333	William A. Cunningham	238	76	20	64	22	.005
334	Kentaro Fujita	458	69	20	62	21	.005
335	Geoffrey L. Cohen	1590	68	20	50	21	.005
336	Ana Guinote	378	76	20	47	21	.005
337	Tanya L. Chartrand	424	67	20	33	21	.001
338	Selin Kesebir	328	66	19	73	22	.005
339	Vincent Y. Yzerbyt	1412	73	19	73	22	.01
340	Amy J. C. Cuddy	170	81	19	72	22	.005
341	James K. McNulty	1047	56	19	65	23	.005
342	Robert S. Wyer	871	82	19	63	22	.005
343	Travis Proulx	174	63	19	62	22	.005
344	Peter M. Gollwitzer	1303	64	19	58	22	.005
345	Nilanjana Dasgupta	383	76	19	52	22	.005
346	Richard P. Eibach	753	69	19	47	23	.001
347	Gerald L. Clore	456	74	19	45	22	.001
348	James M. Tyler	130	87	18	74	24	.005
349	Roland Deutsch	365	78	18	71	24	.005
350	Ed Diener	498	64	18	68	24	.005
351	Kennon M. Sheldon	698	74	18	66	23	.005
352	Wilhelm Hofmann	624	67	18	66	23	.005
353	Laura L. Carstensen	723	77	18	64	24	.005
354	Toni Schmader	546	69	18	61	24	.005
355	Frank D. Fincham	734	69	18	59	24	.005
356	David K. Sherman	1128	61	18	57	24	.005
357	Lisa K. Libby	418	65	18	54	24	.005
358	Chen-Bo Zhong	327	68	18	49	25	.005
359	Stefan C. Schmukle	114	62	17	71	26	.005
360	Michel Tuan Pham	246	86	17	68	25	.005
361	Leandre R. Fabrigar	632	70	17	67	26	.005
362	Neal J. Roese	368	64	17	65	25	.005
363	Carey K. Morewedge	633	76	17	65	26	.005
364	Timothy D. Wilson	798	65	17	63	26	.005
365	Brad J. Bushman	897	74	17	62	25	.005
366	Ara Norenzayan	225	72	17	61	25	.005
367	Benoit Monin	635	65	17	56	25	.005
368	Michael W. Kraus	617	72	17	55	26	.005
369	Ad van Knippenberg	683	72	17	55	26	.001
370	E. Tory. Higgins	1868	68	17	54	25	.001
371	Ap Dijksterhuis	750	68	17	54	26	.005
372	Joseph Cesario	146	62	17	45	26	.001
373	Simone Schnall	270	62	17	31	26	.001
374	Joshua M. Ackerman	380	53	16	70	13	.01
375	Melissa J. Ferguson	1163	72	16	69	27	.005
376	Laura A. King	391	76	16	68	29	.005
377	Daniel T. Gilbert	724	65	16	65	27	.005
378	Charles S. Carver	154	82	16	64	28	.005
379	Leif D. Nelson	409	74	16	64	28	.005
380	David DeSteno	201	83	16	57	28	.005
381	Sandra L. Murray	697	60	16	55	28	.001
382	Heejung S. Kim	858	59	16	55	29	.001
383	Mark P. Zanna	659	64	16	48	28	.001
384	Nira Liberman	1304	75	15	65	31	.005
385	Gun R. Semin	159	79	15	64	29	.005
386	Tal Eyal	439	62	15	62	29	.005
387	Nathaniel M Lambert	456	66	15	59	30	.001
388	Angela L. Duckworth	122	61	15	55	30	.005
389	Dana R. Carney	200	60	15	53	30	.001
390	Lee Ross	349	77	14	63	31	.001
391	Arie W. Kruglanski	1228	78	14	58	33	.001
392	Ziva Kunda	217	67	14	56	31	.001
393	Shelley E. Taylor	427	69	14	52	31	.001
394	Jon K. Maner	1040	65	14	52	32	.001
395	Gabriele Oettingen	1047	61	14	49	33	.001
396	Nicole L. Mead	240	70	14	46	33	.01
397	Gregory M. Walton	587	69	14	44	33	.001
398	Michael A. Olson	346	65	13	63	35	.001
399	Fiona Lee	221	67	13	58	34	.001
400	Melody M. Chao	237	57	13	58	36	.001
401	Adam L. Alter	314	78	13	54	36	.001
402	Sarah E. Hill	509	78	13	52	34	.001
403	Jaime L. Kurtz	91	55	13	38	37	.001
404	Michael A. Zarate	120	52	13	31	36	.001
405	Jennifer K. Bosson	659	76	12	64	40	.001
406	Daniel M. Oppenheimer	198	80	12	60	37	.001
407	Deborah A. Prentice	89	80	12	57	38	.001
408	Yaacov Trope	1277	73	12	57	38	.001
409	Oscar Ybarra	305	63	12	55	40	.001
410	William von Hippel	398	65	12	48	40	.001
411	Steven J. Spencer	541	67	12	44	38	.001
412	Martie G. Haselton	186	73	11	54	43	.001
413	Shelly Chaiken	360	74	11	52	44	.001
414	Susan M. Andersen	361	74	11	48	43	.001
415	Dov Cohen	641	68	11	44	41	.001
416	Mark Muraven	496	52	11	44	41	.001
417	Ian McGregor	409	66	11	40	41	.001
418	Hans Ijzerman	214	56	9	46	51	.001
419	Linda M. Isbell	115	64	9	41	50	.001
420	Cheryl J. Wakslak	278	73	8	35	59	.001

33 thoughts on “Personalized P-Values for Social/Personality Psychologists”

Ursula Hess says:

January 20, 2021 at 6:39 am

Only 801 of the listed 1260 effects were actually taken from research that I was involved in (some seem to stem from articles for which I was editor, others are a mystery to me). On the other hand, the majority of my research is missing. It seems preferable to publish data that is actually based on a more or less representative sample of research actually done by the person with whom that data is associated.

Loading...

Reply
1. Ulrich Schimmack says:
  
  January 20, 2021 at 7:35 am
  
  Thank you for the comment. They are valuable to improve the informativeness of the z-curve analyses.
  1. only social/personalty journals and general journals like Psych Science were used (I posted a list of the journals).
  I will make clear which journals were used.
  2. I am trying to screen out mentions of names as editor, but the program is not perfect. I will look into this and update according.
  3. I found a way to screen out more articles where your name appeared in footnotes (thank you).
  4. I updated the results and they did improve.
  5. Please check the new results.
  
  Loading...
  
  Reply
  1. Ursula Hess says:
    
    January 20, 2021 at 10:37 am
    
    Thank you for the quick response. Some of my research is published in psychophysiology or cognitive journals hence I now understand why so much is missing.
    
    Loading...
  2. Ulrich Schimmack says:
    
    January 20, 2021 at 11:35 am
    
    I figure that research practices can vary once physiological measures are taken or in cognitive studies with within-subject designs. I will eventually do similar posts for other areas.
    
    Loading...
Bill von Hippel says:

January 23, 2021 at 12:26 am

I’m dismayed (and aghast) to see that I’m almost at the bottom of this list. Any advice on how to investigate this further to see where the problem lies?

Loading...

Reply
1. Ulrich Schimmack says:
  
  January 23, 2021 at 8:51 am
  
  Thank you for your comment.
  You can download a file called “William von Hippel-rindex.csv”
  It contains all the articles that were used and computes the R-Index based on the z-scores found for that article. The R-Index is a simple way to estimate replicability that works for small sets of test statistics. An R-Index of 50 would suggest that the replicability is about 50%. The EDR would be lower, but is hard to estimate with a small set of test statistics. The file is sorted by the R-Index. Articles with an R-Index below 50 are probably not robust. This is a good way to start diagnosing the problem.
  
  Loading...
  
  Reply
  1. Bill von Hippel says:
    
    January 23, 2021 at 4:51 pm
    
    Hi Uli, that’s very helpful – thanks!
    
    But now I’m confused. To start with the worst offenders on my list, I have four papers with an R-Index of 0. I can’t tell what two of them are, as your identifier doesn’t include the article title or authors, but two of them are clear. The first of those two has large samples, reports a wide variety of large and small correlations, and strikes me as highly replicable. Indeed, study 2 (N=466) is a direct replication of study 1 (N=196) with an even larger sample. Study 3 goes in a slightly different direction, but mostly relies on the data from Study 2. The other paper reports large samples (Ns = 200) but small effects. We submitted it with only one study, the editor asked for replication, we ran a direct replication with the same sample size and found the same effect. Those are both in the paper. Since then we’ve tried to replicate it once and have succeeded (that finding isn’t yet published).
    
    That’s the first issue, and strikes me as the most important. Secondarily, there are at least four or five papers in this list that aren’t my own – perhaps more but it’s hard to tell what some of the papers are – and the resultant list of papers is only about 1/3 of my empirical publications. Thus, setting aside the most important issue above, I don’t have a clear sense of what my actual replicability score would look like with all of my papers.
    
    All the best, Bill
    
    Loading...
  2. Ulrich Schimmack says:
    
    January 23, 2021 at 5:19 pm
    
    please check the number of results. Many papers with R-Index of 0 have only 1 result which is often just a missing value, meaning no results were found. So, you can ignore those.
    
    Loading...
  3. Ulrich Schimmack says:
    
    January 23, 2021 at 5:20 pm
    
    I also made clear which journals were searched for these articles. Please see the list on the blog post.
    
    Loading...
  4. Ulrich Schimmack says:
    
    January 23, 2021 at 5:21 pm
    
    I would also be happy to run an analysis on all of your articles, if you send me the pdfs.
    
    Loading...
Pingback: Jens Forster and the Credibility Crisis in Social Psychology | Replicability-Index
Bill von Hippel says:

January 23, 2021 at 5:26 pm

There are numerous correlations reported in both papers, along with various mediational analyses in one of them, so definitely not a single result.

With regard to the second issue, the file lists the journal title and year, but that’s it. Sometimes I haven’t published in that journal in that year, so I know it’s not me. Sometimes I have, but in this particular case the only paper I published in that journal in that year has another one of the R = 0 examples, but includes a sample in the millions and a multiverse analysis. There’s no chance that could have a replicability index of 0.

Loading...

Reply
Bill von Hippel says:

January 23, 2021 at 5:29 pm

Thanks Uli, very kind of you to offer to run the analysis for me. I’ve created a dropbox folder with all of my empirical articles in it and shared it with you. Let me know if that doesn’t come through. Best, Bill

Loading...

Reply
Pingback: The Prevalence of Questionable Research Practices in Social Psychology | Replicability-Index
Pingback: A tutorial about effect sizes, power, z-curve analysis, and personalized p-values. | Replicability-Index
Pingback: Replicability Rankings of Eminent Social Psychologists | Replicability-Index
Pingback: Power and Success: When the R-Index meets the H-Index | Replicability-Index
Dolores Albarracin says:

April 5, 2021 at 11:38 am

Hi Uli,

I am surprised that the work you are analyzing for my index contains only 36 entries (when Web of Science retrieves 155). Two of the entries you list are not mine.

Please make sure you include all empirical publications for an author, which in my case span a variety of areas, methods, and journals. Excluding the papers that are not mine would also be methodologically sound and make your own work more authoritative before you disseminate it.

Thanks,
Dolores

Loading...

Reply
1. Ulrich Schimmack says:
  
  April 5, 2021 at 12:48 pm
  
  The method uses a sampling approach. It is based on the journals that are tracked in the replicability rankings of 120 psychology journals, although it may expand as the rankings expand. The list of 120 journals includes the major social psychology journals. So, it is possible that your results might differ for other areas, but as this list focusses on social psychology, it also makes sense to focus on these journals.
  
  Loading...
  
  Reply
2. Ulrich Schimmack says:
  
  April 5, 2021 at 12:49 pm
  
  If you point out the two articles that are not yours, I will exclude them.
  
  Loading...
  
  Reply
3. Ulrich Schimmack says:
  
  April 5, 2021 at 12:52 pm
  
  P.S. We all learned after 2011 that the way we collected and analyzed data was wrong and resulted in inflated estimates of replicability and effect sizes. These results mainly reflect this. How have your research practices changed over the past 10 years in response to the replication crisis?
  
  Loading...
  
  Reply
  1. Dolores Albarracin says:
    
    April 9, 2021 at 9:01 am
    
    There are two articles of which I am not an author. They are not too easy to find in the very small sample you have. I have to find your data again to point them out but they are pretty obvious from the authors and should be checked. Thanks
    
    Loading...
  2. Ulrich Schimmack says:
    
    April 9, 2021 at 9:15 am
    
    It is not that easy. I would have to open all 36 articles to check the full author list. As you already did the work, it would be nice to share the information with me so that I can remove them.
    
    Loading...
  3. Dolores Albarracin says:
    
    April 9, 2021 at 11:01 am
    
    These two are not mine, Uli:
    
    entry 26, The impression management of intelligence
    entry 5, US southern and northern differences in perception
    
    You should add an index of how much of an author’s work you’re tracking. It is likely not a valid representation of the author per se unless you select papers at random from the person’s record.
    
    The replication crisis was fully established towards the mid-2010s right? so the last 10 years do not reflect the changes in response to it.
    
    Loading...
  4. Ulrich Schimmack says:
    
    April 9, 2021 at 3:05 pm
    
    I found the problem why those two studies were included and ran the search again. The main results showed a change in the ODR from 67 to 66 percent and in the EDR from 19 to 21 percent.
    
    Not all of your other publications are original studies (e.g. meta-analysis). Others are in years that are not covered or journals that are not covered. If you send me a folder with pdf files of these articles, I am happy to include them.
    
    Loading...
  5. Dolores Albarracin says:
    
    April 9, 2021 at 6:30 pm
    
    Thanks, Uli. I will get you the PDFs. Meta-analyses are still quantitative research, so you should find a way of including them. The statistics are all comparable and power matters too.
    
    Loading...
  6. Ulrich Schimmack says:
    
    April 10, 2021 at 9:27 am
    
    Meta-analysis are different from original articles in important ways.
    1. Authors have no influence on the quality of the studies they meta-analyze.
    2. The focus is on effect size estimation and not on hypothesis testing.
    Thus, it makes no sense to include them in an investigation of the robustness of original research.
    
    Loading...
  7. Dolores Albarracin says:
    
    April 10, 2021 at 12:24 pm
    
    Well, they don’t control the quality but they can select for quality and they definitely have statistical power considerations. In many cases, the focus IS actually on hypothesis testing and testing new hypotheses, and the method does signal an interest in reproducibility for sure.
    
    Loading...
  8. Ulrich Schimmack says:
    
    April 10, 2021 at 12:30 pm
    
    I am not saying meta-analysis are not important and cannot be evaluated, but my method cannot do this. This also means that the results here are not an overall evaluation of a researcher, which no single index can (including H-Index).
    
    Loading...
Pingback: Replicability Audit of Ap Dijksterhuis | Replicability-Index
Pingback: Bill von Hippel and Ulrich Schimmack discuss Bill’s Replicability Index | Replicability-Index
Pingback: Dr. Ulrich Schimmack Blogs about Replicability | Replicability-Index
Pingback: Bill von Hippel and Ulrich Schimmack discuss Bill’s Replicability Index: Part 2 | Replicability-Index