2022 Replicability Rankings of Psychology Journals

A major replication project of 100 studies published in 2008 showed that only 37% of published significant results could be replicated (i.e., the replication study also produced a significant result). This finding has raised concerns about the replicability of psychological science (OSC, 2015).

Conducting actual replication studies to examine the credibility of published results is effortful and costly. Therefore, my colleagues and I developed a cheaper and faster method to estimate the replicability of published results on the basis of published test statistics (Schimmack, 2022). The method produces two estimates of replicability that represent the best possible and the worst possible scenario. The expected replication rate (ERR) assumes that it is possible to replicate studies exactly. When this is not the case, selection for significance and regression to the mean will lead to lower success rates in actual replication studies. The expected discovery rate (EDR) is an estimate of the actual number of statistical results that researchers obtain in their laboratories. If selection for significance is ineffective in reducing the risk of false positive results or in selecting studies with more power, replication studies are expected to be no more successful than original studies (Brunner & Schimmack, 2020). In the absence of any further information, I am using the average of the EDR and ERR as the best prediction of the outcome of actual replication studies. I call this index the Actual Replicability Prediction (ARP). Whereas previous rankings relied exclusively on the ERR, the 2021 rankings start using the ARP to rank the replicability of journals.

Figure 1 shows the average ARP, ERR, and EDR for a broad range of psychology journals. Given the large number of test statistics in each year (k > 100,000), the estimates are very precise. The dashed lines show the 95%CI around the linear trend line. The results show that the ARP has increased from around 50% to slightly under 60%. This finding shows that results published in psychological journals have become a bit more replicable, although this prediction needs to be verified with actual replication studies.

However, the increase is not uniform across journals. Whereas some journals in social psychology showed some big increases, other journals show no changes. The big increases in social psychology are in part due to very low replication rates in this field before 2015 (OSC, 2015). For readers of journals changes are less important than actual replication rates. Table 1 shows the rankings of journals. Predicted replication rates range from an astonishing 97% in the Journal of Individual Differences to a disappointing 37% for Annals of Behavioural Medicine. Of course, results for 2021 are influenced by sampling error. More detailed information about previous years and trends can be found by clicking on the Journal Name.

For now, you can compare these results to previous results using prior rankings from 2020, 2019, or 2018 (these posts only report the ERR).

Rank   JournalARP 2021EDR 2021ERR 2021
1Journal of Individual Differences979697
2Journal of Occupational Health Psychology919191
3JPSP-Personality Processes and Individual Differences817686
4Archives of Sexual Behavior787481
5JPSP-Interpersonal Relationships and Group Processes787582
6Sex Roles767379
7Aggressive Behaviours746682
8Evolutionary Psychology747078
9Social Psychology746979
10Journal of Memory and Language736779
12Self and Identity736977
13Journal of Organizational Psychology715686
14Attention, Perception and Psychophysics706277
15European Journal of Developmental Psychology706774
16Journal of Child Psychology and Psychiatry and Allied Disciplines705882
17Law and Human Behavior706278
18Psychology of Religion and Spirituality706773
20Judgment and Decision Making695683
21Political Psychology695880
22Journal of Family Psychology686372
23Journal of Abnormal Psychology665875
24J. of Exp. Psychology – Learning, Memory & Cognition665774
25J. of Exp. Psychology – Human Perception and Performance655377
26Journal of Comparative Psychology644485
27Behavior Therapy635077
28Journal of Research on Adolescence635669
29Quarterly Journal of Experimental Psychology635176
30British Journal of Developmental Psychology624876
31Personality and Individual Differences624678
32Psychology and Aging624975
33Psychonomic Bulletin and Review624975
34Psychological Science624479
35Acta Psychologica614676
36Cognition and Emotion614478
37Journal of Sex Research613884
38Child Development604675
39Cognitive Development604871
40European Journal of Social Psychology604576
41Evolution & Human Behavior593681
42Journal of Experimental Psychology – Applied594473
43Journal of Experimental Child Psychology594671
44JPSP-Attitudes & Social Cognition594375
45Memory and Cognition593880
46Cognitive Therapy and Research584769
47Journal of Experimental Psychology – General584076
48Journal of Experimental Social Psychology583978
49Journal of Health Psychology584274
50Personality and Social Psychology Bulletin583878
51Social Development584670
52Journal of Nonverbal Behavior573678
53Motivation and Emotion574469
55Social Psychological and Personality Science573678
56Cognitive Behaviour Therapy564072
57Developmental Psychology564369
58Frontiers in Psychology564270
59Consciousness and Cognition553773
60Journal of Applied Psychology553970
61Journal of Behavioral Decision Making553079
62Journal of Cross-Cultural Psychology553970
63Journal of Cognition and Development553673
65Addictive Behaviors543376
66Asian Journal of Social Psychology543672
67International Journal of Psychophysiology544068
68British Journal of Psychology533868
70Frontiers in Behavioral Neuroscience534363
71Journal of Affective Disorders533175
72Journal of Child and Family Studies533967
73Journal of Cognitive Psychology533175
74Journal of Research in Personality533174
76Psychology of Men and Masculinity533472
77Psychology of Music534166
79British Journal of Social Psychology523667
80Journal of Applied Social Psychology523966
81Journal of Business and Psychology523271
82Organizational Behavior and Human Decision Processes522580
83Psychology and Marketing523667
84Psychological Medicine524263
85Animal Behavior512875
86Behavioural Brain Research513073
87Canadian Journal of Experimental Psychology512577
88Journal of Personality512380
89Journal of Religion and Health513467
91Group Processes & Intergroup Relations502772
92Journal of Social and Personal Relationships502773
93Biological Psychology493267
94Depression & Anxiety493365
95Experimental Psychology492672
96Journal of Consumer Research493761
97Journal of Educational Psychology492376
98Journal of Youth and Adolescence492772
99Behaviour Research and Therapy483165
101Journal of Consumer Behaviour482274
102Developmental Psychobiology473459
103Frontiers in Human Neuroscience473064
104Journal of Consulting and Clinical Psychology473361
105Hormones and Behavior462863
106Journal of Anxiety Disorders462073
107Journal of Positive Psychology462467
108Cognitive Psychology452267
109European Journal of Personality453259
110Psychology Crime and Law451971
111Developmental Science442563
112Journal of Consumer Psychology441870
113Journal of Social Psychology431770
114Behavioral Neuroscience421667
115Journal of Happiness Studies421667
116Journal of Occupational and Organizational Psychology422361
117Personal Relationships421768
118Journal of Vocational Behavior411765
119Health Psychology391959
120Journal of Counseling Psychology381661
121Annals of Behavioral Medicine371954
> >

If you liked this post, you might also be interested in “Estimating the False Discovery Risk in Psychological Science‘.”

6 thoughts on “2022 Replicability Rankings of Psychology Journals

Leave a Reply