Personalized P-Values for Social/Personality Psychologists

January 19, 2021Alpha Wars, False Discovery Rate, Replicability, Replicability Ranking, Science-Wise False Discovery Rate, Social PsychologyUlrich Schimmack

Last update 4/9/2021
(includes 2020, expanded to 353 social/personality psychologists, minor corrections, added rank numbers for easy comparison)

Introduction

Since Fisher invented null-hypothesis significance testing, researchers have used p < .05 as a statistical criterion to interpret results as discoveries worthwhile of discussion (i.e., the null-hypothesis is false). Once published, these results are often treated as real findings even though alpha does not control the risk of false discoveries.

Statisticians have warned against the exclusive reliance on p < .05, but nearly 100 years after Fisher popularized this approach, it is still the most common way to interpret data. The main reason is that many attempts to improve on this practice have failed. The main problem is that a single statistical result is difficult to interpret. However, when individual results are interpreted in the context of other results, they become more informative. Based on the distribution of p-values it is possible to estimate the maximum false discovery rate (Bartos & Schimmack, 2020; Jager & Leek, 2014). This approach can be applied to the p-values published by individual authors to adjust p-values to keep the risk of false discoveries at a reasonable level, FDR < .05.

Researchers who mainly test true hypotheses with high power have a high discovery rate (many p-values below .05) and a low false discovery rate (FDR < .05). Figure 1 shows an example of a researcher who followed this strategy (for a detailed description of z-curve plots, see Schimmack, 2021).

We see that out of the 317 test-statistics retrieved from his articles, 246 were significant with alpha = .05. This is an observed discovery rate of 78%. We also see that this discovery rate closely matches the estimated discovery rate based on the distribution of the significant p-values, p < .05. The EDR is 79%. With an EDR of 79%, the maximum false discovery rate is only 1%. However, the 95%CI is wide and the lower bound of the CI for the EDR, 27%, allows for 14% false discoveries.

When the ODR matches the EDR, there is no evidence of publication bias. In this case, we can improve the estimates by fitting all p-values, including the non-significant ones. With a tighter CI for the EDR, we see that the 95%CI for the maximum FDR ranges from 1% to 3%. Thus, we can be confident that no more than 5% of the significant results wit alpha = .05 are false discoveries. Readers can therefore continue to use alpha = .05 to look for interesting discoveries in Matsumoto’s articles.

Figure 3 shows the results for a different type of researcher who took a risk and studied weak effect sizes with small samples. This produces many non-significant results that are often not published. The selection for significance inflates the observed discovery rate, but the z-curve plot and the comparison with the EDR shows the influence of publication bias. Here the ODR is similar to Figure 1, but the EDR is only 11%. An EDR of 11% translates into a large maximum false discovery rate of 41%. In addition, the 95%CI of the EDR includes 5%, which means the risk of false positives could be as high as 100%. In this case, using alpha = .05 to interpret results as discoveries is very risky. Clearly, p < .05 means something very different when reading an article by David Matsumoto or Shelly Chaiken.

Rather than dismissing all of Chaiken’s results, we can try to lower alpha to reduce the false discovery rate. If we set alpha = .01, the FDR is 15%. If we set alpha = .005, the FDR is 8%. To get the FDR below 5%, we need to set alpha to .001.

A uniform criterion of FDR < 5% is applied to all researchers in the rankings below. For some this means no adjustment to the traditional criterion. For others, alpha is lowered to .01, and for a few even lower than that.

The rankings below are based on automatrically extracted test-statistics from 40 journals (List of journals). The results should be interpreted with caution and treated as preliminary. They depend on the specific set of journals that were searched, the way results are being reported, and many other factors. The data are available (data.drop) and researchers can exclude articles or add articles and run their own analyses using the z-curve package in R (https://replicationindex.com/2020/01/10/z-curve-2-0/).

I am also happy to receive feedback about coding errors. I also recommended to hand-code articles to adjust alpha for focal hypothesis tests. This typically lowers the EDR and increases the FDR. For example, the automated method produced an EDR of 31 for Bargh, whereas hand-coding of focal tests produced an EDR of 12 (Bargh-Audit).

And here are the rankings. The results are fully automated and I was not able to cover up the fact that I placed only #139 out of 300 in the rankings. In another post, I will explain how researchers can move up in the rankings. Of course, one way to move up in the rankings is to increase statistical power in future studies. The rankings will be updated again when the 2021 data are available.

Despite the preliminary nature, I am confident that the results provide valuable information. Until know all p-values below .05 have been treated as if they are equally informative. The rankings here show that this is not the case. While p = .02 can be informative for one researcher, p = .002 may still entail a high false discovery risk for another researcher.

Rank	Name	Tests	ODR	EDR	ERR	FDR	Alpha
1	Robert A. Emmons	58	88	85	88	1	.05
2	David Matsumoto	378	83	79	85	1	.05
3	Linda J. Skitka	532	68	75	82	2	.05
4	Jonathan B. Freeman	274	59	75	81	2	.05
5	Virgil Zeigler-Hill	515	72	74	81	2	.05
6	Arthur A. Stone	310	75	73	81	2	.05
7	David P. Schmitt	207	78	71	77	2	.05
8	Emily A. Impett	549	77	70	76	2	.05
9	Kurt Gray	487	79	69	81	2	.05
10	Kipling D. Williams	843	75	69	77	2	.05
11	John M. Zelenski	156	71	69	76	2	.05
12	Michael E. McCullough	334	69	69	78	2	.05
13	Hilary B. Bergsieker	439	67	68	74	2	.05
14	Cameron Anderson	652	71	67	74	3	.05
15	Jamil Zaki	430	78	66	76	3	.05
16	Rachel E. Jack	249	70	66	80	3	.05
17	A. Janet Tomiyama	76	78	65	76	3	.05
18	Phoebe C. Ellsworth	605	74	65	72	3	.05
19	Jim Sidanius	487	69	65	72	3	.05
20	Benjamin R. Karney	392	56	65	73	3	.05
21	Carol D. Ryff	280	84	64	76	3	.05
22	Juliane Degner	435	63	64	71	3	.05
23	Steven J. Heine	597	78	63	77	3	.05
24	David M. Amodio	584	66	63	70	3	.05
25	Thomas N Bradbury	398	61	63	69	3	.05
26	Elaine Fox	472	79	62	78	3	.05
27	Klaus Fiedler	1950	77	61	74	3	.05
28	Linda R. Tropp	344	65	61	80	3	.05
29	Richard W. Robins	270	76	60	70	4	.05
30	Simine Vazir	137	66	60	64	4	.05
31	Edward P. Lemay	289	87	59	81	4	.05
32	William B. Swann Jr.	1070	78	59	80	4	.05
33	Margaret S. Clark	505	75	59	77	4	.05
34	Bernhard Leidner	724	64	59	65	4	.05
35	Patricia G. Devine	606	71	58	67	4	.05
36	B. Keith Payne	879	71	58	76	4	.05
37	Ximena B. Arriaga	284	66	58	69	4	.05
38	Rainer Reisenzein	201	65	57	69	4	.05
39	Barbara A. Mellers	287	80	56	78	4	.05
40	Jean M. Twenge	381	72	56	59	4	.05
41	Joris Lammers	705	69	56	69	4	.05
42	Nicholas Epley	1504	74	55	72	4	.05
43	Krishna Savani	638	71	53	69	5	.05
44	Lee Jussim	226	80	52	71	5	.05
45	Edward L. Deci	284	79	52	63	5	.05
46	Richard M. Ryan	998	78	52	69	5	.05
47	Ethan Kross	614	66	52	67	5	.05
48	Roger Giner-Sorolla	663	81	51	80	5	.05
49	Jens B. Asendorpf	253	74	51	69	5	.05
50	Bertram F. Malle	422	73	51	75	5	.05
51	Tessa V. West	691	71	51	59	5	.05
52	Samuel D. Gosling	108	58	51	62	5	.05
53	Stefan Schmukle	436	78	50	81	5	.05
54	Paul Rozin	449	78	50	84	5	.05
55	Joachim I. Krueger	436	78	50	81	5	.05
56	Paul K. Piff	166	77	50	63	5	.05
57	Shinobu Kitayama	983	76	50	71	5	.05
58	Janice R. Kelly	366	75	50	70	5	.05
59	Matthew J. Hornsey	1656	74	50	71	5	.05
60	James J. Gross	1104	72	50	77	5	.05
61	Mark Rubin	306	68	50	75	5	.05
62	Sheena S. Iyengar	207	63	50	80	5	.05
63	Antonio L. Freitas	247	79	50	64	5	.05
64	Mina Cikara	392	71	49	80	5	.05
65	Ludwin E. Molina	163	69	49	61	5	.05
66	Edward R. Hirt	1042	81	48	65	6	.01
67	Bertram Gawronski	1803	72	48	76	6	.01
68	Penelope Lockwood	458	71	48	70	6	.01
69	John T. Cacioppo	438	76	47	69	6	.01
70	Daniel M. Wegner	602	76	47	65	6	.01
71	Agneta H. Fischer	952	75	47	69	6	.01
72	Matthew D. Lieberman	398	72	47	80	6	.01
73	Leaf van Boven	711	72	47	67	6	.01
74	Stephanie A. Fryberg	248	62	47	66	6	.01
75	Jennifer S. Lerner	181	80	46	61	6	.01
76	Rainer Banse	402	78	46	72	6	.01
77	Alice H. Eagly	330	75	46	71	6	.01
78	Jeanne L. Tsai	1241	73	46	67	6	.01
79	Dacher Keltner	1233	72	45	64	6	.01
80	Constantine Sedikides	2566	71	45	70	6	.01
81	Andrea L. Meltzer	549	52	45	72	6	.01
82	R. Chris Fraley	642	70	45	72	7	.01
83	Ursula Hess	774	78	44	71	7	.01
84	Brian A. Nosek	816	68	44	81	7	.01
85	Charles M. Judd	1054	76	43	68	7	.01
86	Jessica L. Tracy	632	74	43	71	7	.01
87	Mark Schaller	565	73	43	61	7	.01
88	Jason P. Mitchell	600	73	43	73	7	.01
89	S. Alexander Haslam	1198	72	43	64	7	.01
90	Mario Mikulincer	901	89	42	64	7	.01
91	Susan T. Fiske	911	78	42	74	7	.01
92	Bernadette Park	973	77	42	64	7	.01
93	Jolanda Jetten	1956	73	42	67	7	.01
94	Paul A. M. Van Lange	1092	70	42	63	7	.01
95	Lisa Feldman Barrett	644	69	42	70	7	.01
96	Wendi L. Gardner	798	67	42	63	7	.01
97	Philip E. Tetlock	549	79	41	73	7	.01
98	Phillip Atiba Goff	299	68	41	62	7	.01
99	Jordan B. Peterson	266	60	41	79	7	.01
100	Amanda B. Diekman	438	83	41	70	7	.01
101	Stacey Sinclair	327	70	41	57	8	.01
102	Michael Inzlicht	686	66	41	63	8	.01
103	Tiffany A. Ito	349	80	40	64	8	.01
104	Wendy Wood	462	75	40	62	8	.01
105	Norbert Schwarz	1337	72	40	63	8	.01
106	Richard E. Petty	2771	69	40	64	8	.01
107	Elizabeth Page-Gould	411	57	40	66	8	.01
108	Tim Wildschut	1374	73	40	64	8	.01
109	Veronika Job	362	70	40	63	8	.01
110	Marcel Zeelenberg	868	76	39	79	8	.01
111	Christian S. Crandall	362	75	39	59	8	.01
112	Tobias Greitemeyer	1737	72	39	67	8	.01
113	Carol S. Dweck	1028	70	39	63	8	.01
114	Jason E. Plaks	582	70	39	67	8	.01
115	Jerry Suls	413	71	38	68	8	.01
116	Eric D. Knowles	384	68	38	64	8	.01
117	C. Nathan DeWall	1336	73	38	63	9	.01
118	John F. Dovidio	2019	69	38	62	9	.01
119	Harry T. Reis	998	69	38	74	9	.01
120	Joshua Correll	549	61	38	62	9	.01
121	Abigail A. Scholer	556	58	38	62	9	.01
122	Clayton R. Critcher	697	82	38	63	9	.01
123	Kevin N. Ochsner	406	79	37	70	9	.01
124	Ayelet Fishbach	1416	78	37	59	9	.01
125	Fritz Strack	607	75	37	56	9	.01
126	Mahzarin R. Banaji	880	73	37	78	9	.01
127	Antony S. R. Manstead	1656	72	37	62	9	.01
128	Mark J. Brandt	277	70	37	70	9	.01
129	Lorne Campbell	433	67	37	61	9	.01
130	Geoff MacDonald	406	67	37	67	9	.01
131	Sanford E. DeVoe	236	71	37	61	9	.01
132	Duane T. Wegener	980	77	36	60	9	.01
133	Craig A. Anderson	467	76	36	55	9	.01
134	D. S. Moskowitz	3418	74	36	63	9	.01
135	Joanne V. Wood	1093	74	36	60	9	.01
136	Todd B. Kashdan	377	73	36	61	9	.01
137	Barbara L. Fredrickson	287	72	36	61	9	.01
138	Nyla R. Branscombe	1276	70	36	65	9	.01
139	Niall Bolger	376	67	36	58	9	.01
140	Yaacov Schul	411	61	36	64	9	.01
141	Jeff T. Larsen	181	74	36	67	10	.01
142	Eva Walther	493	82	35	66	10	.01
143	Michael D. Robinson	1388	78	35	66	10	.01
144	C. Miguel Brendl	121	76	35	68	10	.01
145	Samuel L. Gaertner	321	75	35	61	10	.01
146	Victoria M. Esses	295	75	35	53	10	.01
147	Azim F. Sharif	183	74	35	68	10	.01
148	Michael Harris Bond	378	73	35	84	10	.01
149	Glenn Adams	270	71	35	73	10	.01
150	John T. Jost	794	70	35	61	10	.01
151	Emily Balcetis	599	69	35	68	10	.01
152	Eric L. Uhlmann	457	67	35	61	10	.01
153	Igor Grossmann	203	64	35	66	10	.01
154	Nalini Ambady	1256	62	35	56	10	.01
155	Diana I. Tamir	156	62	35	62	10	.01
156	Daphna Oyserman	446	55	35	54	10	.01
157	Thomas Gilovich	1193	80	34	69	10	.01
158	Alison Ledgerwood	214	75	34	54	10	.01
159	Linda J. Levine	495	74	34	78	10	.01
160	Paula M. Niedenthal	522	69	34	61	10	.01
161	Wiebke Bleidorn	99	63	34	74	10	.01
162	Ozlem Ayduk	549	62	34	59	10	.01
163	Christopher R. Agnew	325	75	33	76	10	.01
164	Kerry Kawakami	487	68	33	56	10	.01
165	Danu Anthony Stinson	494	77	33	54	11	.01
166	Jennifer A. Richeson	831	67	33	52	11	.01
167	Malte Friese	501	61	33	57	11	.01
168	Michelle N. Shiota	242	60	33	63	11	.01
169	Margo J. Monteith	773	76	32	77	11	.01
170	Ulrich Schimmack	318	75	32	63	11	.01
171	Mark Snyder	562	72	32	63	11	.01
172	Robert B. Cialdini	379	72	32	56	11	.01
173	Russell H. Fazio	1094	69	32	61	11	.01
174	Eric van Dijk	238	67	32	60	11	.01
175	Eli J. Finkel	1392	62	32	57	11	.01
176	E. Ashby Plant	831	77	31	51	11	.01
177	Christopher K. Hsee	689	75	31	63	11	.01
178	Yuen J. Huo	132	74	31	80	11	.01
179	Delroy L. Paulhus	121	77	31	82	12	.01
180	John A. Bargh	651	72	31	55	12	.01
181	Roy F. Baumeister	2442	69	31	52	12	.01
182	Tom Pyszczynski	948	69	31	54	12	.01
183	Jamie Arndt	1318	69	31	50	12	.01
184	Kathleen D. Vohs	944	68	31	51	12	.01
185	Vivian Zayas	251	71	31	60	12	.01
186	Anthony G. Greenwald	357	72	30	83	12	.01
187	Dale T. Miller	521	71	30	64	12	.01
188	Aaron C. Kay	1320	70	30	51	12	.01
189	Jennifer Crocker	515	68	30	67	12	.01
190	Arthur Aron	307	65	30	56	12	.01
191	Arthur Aron	307	65	30	56	12	.01
192	Lauren J. Human	447	59	30	70	12	.01
193	Nicholas O. Rule	1294	68	30	75	13	.01
194	Steven W. Gangestad	198	63	30	41	13	.005
195	Boris Egloff	274	81	29	58	13	.01
196	Eliot R. Smith	445	79	29	73	13	.01
197	Jeff Greenberg	1358	77	29	54	13	.01
198	Monica Biernat	813	77	29	57	13	.01
199	Hazel Rose Markus	674	76	29	68	13	.01
200	Russell Spears	2286	73	29	55	13	.01
201	Richard E. Nisbett	319	73	29	69	13	.01
202	Gordon B. Moskowitz	374	72	29	57	13	.01
203	Nir Halevy	262	68	29	72	13	.01
204	Dirk Wentura	830	65	29	64	13	.01
205	Caryl E. Rusbult	218	60	29	54	13	.01
206	E. Allan Lind	370	82	29	72	13	.01
207	Roland Neumann	258	77	28	67	13	.01
208	Yoav Bar-Anan	525	75	28	76	13	.01
209	Jeffry A. Simpson	697	74	28	55	13	.01
210	Adam D. Galinsky	2154	70	28	49	13	.01
211	Joshua Aronson	183	85	28	46	14	.005
212	Matthew Feinberg	295	77	28	69	14	.01
213	Elizabeth W. Dunn	395	75	28	64	14	.01
214	Naomi I. Eisenberger	179	74	28	79	14	.01
215	Eddie Harmon-Jones	738	73	28	70	14	.01
216	Brent W. Roberts	562	72	28	77	14	.01
217	Grainne M. Fitzsimons	585	68	28	49	14	.01
218	Geoffrey J. Leonardelli	290	68	28	48	14	.005
219	Sander L. Koole	767	65	28	52	14	.01
220	Richard J. Davidson	380	64	28	51	14	.01
221	Shelly L. Gable	364	64	28	50	14	.01
222	Guido H. E. Gendolla	422	76	27	47	14	.005
223	Jan De Houwer	1972	70	27	72	14	.01
224	Karl Christoph Klauer	801	67	27	65	14	.01
225	Jennifer S. Beer	80	56	27	54	14	.01
226	Vanessa K. Bohns	422	76	27	74	15	.01
227	Charles Stangor	185	81	27	68	15	.01
228	Klaus R. Scherer	467	83	26	78	15	.01
229	Galen V. Bodenhausen	585	74	26	61	15	.01
230	Claude M. Steele	434	73	26	42	15	.005
231	Sonja Lyubomirsky	531	71	26	59	15	.01
232	William G. Graziano	532	71	26	66	15	.01
233	Kristin Laurin	648	63	26	51	15	.01
234	Kerri L. Johnson	532	76	25	76	15	.01
235	Phillip R. Shaver	566	81	25	71	16	.01
236	Ronald S. Friedman	183	79	25	44	16	.005
237	Mark J. Landau	950	78	25	45	16	.005
238	Nurit Shnabel	564	76	25	79	16	.01
239	David Dunning	818	74	25	70	16	.01
240	Laurie A. Rudman	482	72	25	68	16	.01
241	Joel Cooper	257	72	25	39	16	.005
242	Batja Mesquita	416	71	25	73	16	.01
243	David A. Lishner	105	65	25	63	16	.01
244	Steven J. Sherman	888	74	24	62	16	.01
245	Alison L. Chasteen	223	68	24	69	16	.01
246	Mark W. Baldwin	247	72	24	41	17	.005
247	Thomas Mussweiler	604	70	24	43	17	.005
248	Shigehiro Oishi	1109	64	24	61	17	.01
249	Evan P. Apfelbaum	256	62	24	41	17	.005
250	Jonathan Haidt	368	76	23	73	17	.01
251	Jeffrey W Sherman	992	68	23	71	17	.01
252	Brandon J. Schmeichel	652	66	23	45	17	.005
253	Sam J. Maglio	325	72	23	42	17	.005
254	Roland Imhoff	365	74	23	73	18	.01
255	Felicia Pratto	410	73	23	75	18	.01
256	Klaus Rothermund	738	71	23	76	18	.01
257	Bernard A. Nijstad	693	71	23	52	18	.005
258	Jennifer L. Eberhardt	202	71	23	62	18	.005
259	Marilynn B. Brewer	314	75	22	62	18	.005
260	Michael Ross	1164	70	22	62	18	.005
261	Dieter Frey	1538	68	22	58	18	.005
262	David M. Buss	461	82	22	80	19	.01
263	Sean M. McCrea	584	73	22	54	19	.005
264	Wendy Berry Mendes	965	68	22	44	19	.005
265	Spike W. S. Lee	145	68	22	64	19	.005
266	Yoel Inbar	280	67	22	71	19	.01
267	Serena Chen	865	72	22	67	19	.005
268	Joseph P. Forgas	888	83	21	59	19	.005
269	Maya Tamir	1342	80	21	64	19	.005
270	Paul W. Eastwick	583	65	21	69	19	.005
271	Elizabeth Levy Paluck	31	84	21	55	20	.005
272	Kees van den Bos	1150	84	21	69	20	.005
273	Dolores Albarracin	540	66	21	56	20	.005
274	Andrew J. Elliot	1018	81	20	67	21	.005
275	Ana Guinote	378	76	20	47	21	.005
276	David A. Pizarro	227	71	20	69	21	.005
277	Kentaro Fujita	458	69	20	62	21	.005
278	Geoffrey L. Cohen	1590	68	20	50	21	.005
279	Tanya L. Chartrand	424	67	20	33	21	.001
280	Jay J. van Bavel	437	64	20	71	21	.005
281	William A. Cunningham	238	76	20	64	22	.005
282	Robert S. Wyer	871	82	19	63	22	.005
283	Amy J. C. Cuddy	170	81	19	72	22	.005
284	Nilanjana Dasgupta	383	76	19	52	22	.005
285	Gerald L. Clore	456	74	19	45	22	.001
286	Peter M. Gollwitzer	1303	64	19	58	22	.005
287	Travis Proulx	174	63	19	62	22	.005
288	Selin Kesebir	328	66	19	73	22	.005
289	Richard P. Eibach	753	69	19	47	23	.001
290	James K. McNulty	1047	56	19	65	23	.005
291	Kennon M. Sheldon	698	74	18	66	23	.005
292	Wilhelm Hofmann	624	67	18	66	23	.005
293	James M. Tyler	130	87	18	74	24	.005
294	Roland Deutsch	365	78	18	71	24	.005
295	Laura L. Carstensen	723	77	18	64	24	.005
296	Frank D. Fincham	734	69	18	59	24	.005
297	Toni Schmader	546	69	18	61	24	.005
298	Lisa K. Libby	418	65	18	54	24	.005
299	Ed Diener	498	64	18	68	24	.005
300	Chen-Bo Zhong	327	68	18	49	25	.005
301	Michel Tuan Pham	246	86	17	68	25	.005
302	Brad J. Bushman	897	74	17	62	25	.005
303	Ara Norenzayan	225	72	17	61	25	.005
304	E. Tory. Higgins	1868	68	17	54	25	.001
305	Benoit Monin	635	65	17	56	25	.005
306	Carey K. Morewedge	633	76	17	65	26	.005
307	Michael W. Kraus	617	72	17	55	26	.005
308	Leandre R. Fabrigar	632	70	17	67	26	.005
309	Ap Dijksterhuis	750	68	17	54	26	.005
310	Timothy D. Wilson	798	65	17	63	26	.005
311	Joseph Cesario	146	62	17	45	26	.001
312	Simone Schnall	270	62	17	31	26	.001
313	Melissa J. Ferguson	1163	72	16	69	27	.005
314	Daniel T. Gilbert	724	65	16	65	27	.005
315	Charles S. Carver	154	82	16	64	28	.005
316	Leif D. Nelson	409	74	16	64	28	.005
317	Mark P. Zanna	659	64	16	48	28	.001
318	Sandra L. Murray	697	60	16	55	28	.001
319	Laura A. King	391	76	16	68	29	.005
320	Heejung S. Kim	858	59	16	55	29	.001
321	Gun R. Semin	159	79	15	64	29	.005
322	Tal Eyal	439	62	15	62	29	.005
323	Nathaniel M Lambert	456	66	15	59	30	.001
324	Dana R. Carney	200	60	15	53	30	.001
325	Nira Liberman	1304	75	15	65	31	.005
326	Lee Ross	349	77	14	63	31	.001
327	Shelley E. Taylor	427	69	14	52	31	.001
328	Ziva Kunda	217	67	14	56	31	.001
329	Jon K. Maner	1040	65	14	52	32	.001
330	Arie W. Kruglanski	1228	78	14	58	33	.001
331	Gregory M. Walton	587	69	14	44	33	.001
332	Gabriele Oettingen	1047	61	14	49	33	.001
333	Sarah E. Hill	509	78	13	52	34	.001
334	Fiona Lee	221	67	13	58	34	.001
335	Michael A. Olson	346	65	13	63	35	.001
336	Michael A. Zarate	120	52	13	31	36	.001
337	Melody M. Chao	237	57	13	58	36	.001
338	Jamie L. Kurtz	91	55	13	38	37	.001
339	Daniel M. Oppenheimer	198	80	12	60	37	.001
340	Deborah A. Prentice	89	80	12	57	38	.001
341	Yaacov Trope	1277	73	12	57	38	.001
342	Steven J. Spencer	541	67	12	44	38	.001
343	William von Hippel	398	65	12	48	40	.001
344	Oscar Ybarra	305	63	12	55	40	.001
345	Dov Cohen	641	68	11	44	41	.001
346	Ian McGregor	409	66	11	40	41	.001
347	Mark Muraven	496	52	11	44	41	.001
348	Susan M. Andersen	361	74	11	48	43	.001
349	Martie G. Haselton	186	73	11	54	43	.001
350	Shelly Chaiken	360	74	11	52	44	.001
351	Linda M. Isbell	115	64	9	41	50	.001
352	Hans Ijzerman	214	56	9	46	51	.001
353	Cheryl J. Wakslak	278	73	8	35	59	.001

28 thoughts on “Personalized P-Values for Social/Personality Psychologists”

Ursula Hess says:

January 20, 2021 at 6:39 am

Only 801 of the listed 1260 effects were actually taken from research that I was involved in (some seem to stem from articles for which I was editor, others are a mystery to me). On the other hand, the majority of my research is missing. It seems preferable to publish data that is actually based on a more or less representative sample of research actually done by the person with whom that data is associated.

LikeLike

Reply
1. Ulrich Schimmack says:
  
  January 20, 2021 at 7:35 am
  
  Thank you for the comment. They are valuable to improve the informativeness of the z-curve analyses.
  1. only social/personalty journals and general journals like Psych Science were used (I posted a list of the journals).
  I will make clear which journals were used.
  2. I am trying to screen out mentions of names as editor, but the program is not perfect. I will look into this and update according.
  3. I found a way to screen out more articles where your name appeared in footnotes (thank you).
  4. I updated the results and they did improve.
  5. Please check the new results.
  
  LikeLike
  
  Reply
  1. Ursula Hess says:
    
    January 20, 2021 at 10:37 am
    
    Thank you for the quick response. Some of my research is published in psychophysiology or cognitive journals hence I now understand why so much is missing.
    
    LikeLike
  2. Ulrich Schimmack says:
    
    January 20, 2021 at 11:35 am
    
    I figure that research practices can vary once physiological measures are taken or in cognitive studies with within-subject designs. I will eventually do similar posts for other areas.
    
    LikeLike
Bill von Hippel says:

January 23, 2021 at 12:26 am

I’m dismayed (and aghast) to see that I’m almost at the bottom of this list. Any advice on how to investigate this further to see where the problem lies?

LikeLike

Reply
1. Ulrich Schimmack says:
  
  January 23, 2021 at 8:51 am
  
  Thank you for your comment.
  You can download a file called “William von Hippel-rindex.csv”
  It contains all the articles that were used and computes the R-Index based on the z-scores found for that article. The R-Index is a simple way to estimate replicability that works for small sets of test statistics. An R-Index of 50 would suggest that the replicability is about 50%. The EDR would be lower, but is hard to estimate with a small set of test statistics. The file is sorted by the R-Index. Articles with an R-Index below 50 are probably not robust. This is a good way to start diagnosing the problem.
  
  LikeLike
  
  Reply
  1. Bill von Hippel says:
    
    January 23, 2021 at 4:51 pm
    
    Hi Uli, that’s very helpful – thanks!
    
    But now I’m confused. To start with the worst offenders on my list, I have four papers with an R-Index of 0. I can’t tell what two of them are, as your identifier doesn’t include the article title or authors, but two of them are clear. The first of those two has large samples, reports a wide variety of large and small correlations, and strikes me as highly replicable. Indeed, study 2 (N=466) is a direct replication of study 1 (N=196) with an even larger sample. Study 3 goes in a slightly different direction, but mostly relies on the data from Study 2. The other paper reports large samples (Ns = 200) but small effects. We submitted it with only one study, the editor asked for replication, we ran a direct replication with the same sample size and found the same effect. Those are both in the paper. Since then we’ve tried to replicate it once and have succeeded (that finding isn’t yet published).
    
    That’s the first issue, and strikes me as the most important. Secondarily, there are at least four or five papers in this list that aren’t my own – perhaps more but it’s hard to tell what some of the papers are – and the resultant list of papers is only about 1/3 of my empirical publications. Thus, setting aside the most important issue above, I don’t have a clear sense of what my actual replicability score would look like with all of my papers.
    
    All the best, Bill
    
    LikeLike
  2. Ulrich Schimmack says:
    
    January 23, 2021 at 5:19 pm
    
    please check the number of results. Many papers with R-Index of 0 have only 1 result which is often just a missing value, meaning no results were found. So, you can ignore those.
    
    LikeLiked by 1 person
  3. Ulrich Schimmack says:
    
    January 23, 2021 at 5:20 pm
    
    I also made clear which journals were searched for these articles. Please see the list on the blog post.
    
    LikeLiked by 1 person
  4. Ulrich Schimmack says:
    
    January 23, 2021 at 5:21 pm
    
    I would also be happy to run an analysis on all of your articles, if you send me the pdfs.
    
    LikeLiked by 1 person
Pingback: Jens Forster and the Credibility Crisis in Social Psychology | Replicability-Index
Bill von Hippel says:

January 23, 2021 at 5:26 pm

There are numerous correlations reported in both papers, along with various mediational analyses in one of them, so definitely not a single result.

With regard to the second issue, the file lists the journal title and year, but that’s it. Sometimes I haven’t published in that journal in that year, so I know it’s not me. Sometimes I have, but in this particular case the only paper I published in that journal in that year has another one of the R = 0 examples, but includes a sample in the millions and a multiverse analysis. There’s no chance that could have a replicability index of 0.

LikeLiked by 1 person

Reply
Bill von Hippel says:

January 23, 2021 at 5:29 pm

Thanks Uli, very kind of you to offer to run the analysis for me. I’ve created a dropbox folder with all of my empirical articles in it and shared it with you. Let me know if that doesn’t come through. Best, Bill

LikeLiked by 1 person

Reply
Pingback: The Prevalence of Questionable Research Practices in Social Psychology | Replicability-Index
Pingback: A tutorial about effect sizes, power, z-curve analysis, and personalized p-values. | Replicability-Index
Pingback: Replicability Rankings of Eminent Social Psychologists | Replicability-Index
Dolores Albarracin says:

April 5, 2021 at 11:38 am

Hi Uli,

I am surprised that the work you are analyzing for my index contains only 36 entries (when Web of Science retrieves 155). Two of the entries you list are not mine.

Please make sure you include all empirical publications for an author, which in my case span a variety of areas, methods, and journals. Excluding the papers that are not mine would also be methodologically sound and make your own work more authoritative before you disseminate it.

Thanks,
Dolores

LikeLike

Reply
1. Ulrich Schimmack says:
  
  April 5, 2021 at 12:48 pm
  
  The method uses a sampling approach. It is based on the journals that are tracked in the replicability rankings of 120 psychology journals, although it may expand as the rankings expand. The list of 120 journals includes the major social psychology journals. So, it is possible that your results might differ for other areas, but as this list focusses on social psychology, it also makes sense to focus on these journals.
  
  LikeLike
  
  Reply
2. Ulrich Schimmack says:
  
  April 5, 2021 at 12:49 pm
  
  If you point out the two articles that are not yours, I will exclude them.
  
  LikeLike
  
  Reply
3. Ulrich Schimmack says:
  
  April 5, 2021 at 12:52 pm
  
  P.S. We all learned after 2011 that the way we collected and analyzed data was wrong and resulted in inflated estimates of replicability and effect sizes. These results mainly reflect this. How have your research practices changed over the past 10 years in response to the replication crisis?
  
  LikeLike
  
  Reply
  1. Dolores Albarracin says:
    
    April 9, 2021 at 9:01 am
    
    There are two articles of which I am not an author. They are not too easy to find in the very small sample you have. I have to find your data again to point them out but they are pretty obvious from the authors and should be checked. Thanks
    
    LikeLike
  2. Ulrich Schimmack says:
    
    April 9, 2021 at 9:15 am
    
    It is not that easy. I would have to open all 36 articles to check the full author list. As you already did the work, it would be nice to share the information with me so that I can remove them.
    
    LikeLike
  3. Dolores Albarracin says:
    
    April 9, 2021 at 11:01 am
    
    These two are not mine, Uli:
    
    entry 26, The impression management of intelligence
    entry 5, US southern and northern differences in perception
    
    You should add an index of how much of an author’s work you’re tracking. It is likely not a valid representation of the author per se unless you select papers at random from the person’s record.
    
    The replication crisis was fully established towards the mid-2010s right? so the last 10 years do not reflect the changes in response to it.
    
    LikeLiked by 1 person
  4. Ulrich Schimmack says:
    
    April 9, 2021 at 3:05 pm
    
    I found the problem why those two studies were included and ran the search again. The main results showed a change in the ODR from 67 to 66 percent and in the EDR from 19 to 21 percent.
    
    Not all of your other publications are original studies (e.g. meta-analysis). Others are in years that are not covered or journals that are not covered. If you send me a folder with pdf files of these articles, I am happy to include them.
    
    LikeLike
  5. Dolores Albarracin says:
    
    April 9, 2021 at 6:30 pm
    
    Thanks, Uli. I will get you the PDFs. Meta-analyses are still quantitative research, so you should find a way of including them. The statistics are all comparable and power matters too.
    
    LikeLike
  6. Ulrich Schimmack says:
    
    April 10, 2021 at 9:27 am
    
    Meta-analysis are different from original articles in important ways.
    1. Authors have no influence on the quality of the studies they meta-analyze.
    2. The focus is on effect size estimation and not on hypothesis testing.
    Thus, it makes no sense to include them in an investigation of the robustness of original research.
    
    LikeLike
  7. Dolores Albarracin says:
    
    April 10, 2021 at 12:24 pm
    
    Well, they don’t control the quality but they can select for quality and they definitely have statistical power considerations. In many cases, the focus IS actually on hypothesis testing and testing new hypotheses, and the method does signal an interest in reproducibility for sure.
    
    LikeLike
  8. Ulrich Schimmack says:
    
    April 10, 2021 at 12:30 pm
    
    I am not saying meta-analysis are not important and cannot be evaluated, but my method cannot do this. This also means that the results here are not an overall evaluation of a researcher, which no single index can (including H-Index).
    
    LikeLike

Replicability-Index

Improving the replicability of empirical research

28 thoughts on “Personalized P-Values for Social/Personality Psychologists”

Leave a Reply Cancel reply

Introduction

Rate this:

Share this:

Like this:

28 thoughts on “Personalized P-Values for Social/Personality Psychologists”

Leave a Reply Cancel reply