In the late 1980s, experimental social psychology rediscovered the unconscious. One reason might be that psychological laboratories started to use personal computers to conduct studies. This made it possible to present subliminal stimuli or measure reaction times cheaply. Another reason might have been that conscious social cognitive processes are relatively boring and easily accessible to introspection. It was difficult to find novel and eye-catching results with self-reports. The so called implicit revolution (Greenwald & Banaji, 2017) is still going strong, but first signs of problems are visible everywhere. An article by Ap Dijksterhuis (2004) proposed that unconscious process are better than conscious deliberations in making complex choices. This article stimulated research into unconscious thought theory.

Figure 1 shows publication and citation rates in Web of Science. Notably, the publication rate increased steeply until 2011, the year the replication crisis in social psychology started. Afterwards, publications show a slowly decreasing trend. However, citations continue to increase, suggesting that concerns about the robustness of published results has not reduced trust in this literature.

A meta-analysis and failed replication study raised concerns that many findings in this literature may be false positive results (Nieuwenstein et al., 2017). To further examine the credibility of this literature, I subjected the 220 articles in the Web of Science topic search to a z-curve analysis. I first looked for matching articles in a database of articles from 121 psychology journals that includes all major social psychology journals (Schimmack, 2022). This search retrieved 44 articles. An automatic search of these 44 articles produced 534 test statistics. A z-curve analysis of these test statistics showed 64% significant results (not counting marginally significant results, z > 1.65), but the z-curve estimate of power was only 30%. The 95% confidence interval ranges from 10% to 50% and does not include the observed discovery rate of 64%. Thus, there is clear evidence that the published rate of significant results is inflated by unscientific research practices.

An EDR of 30% implies that up to 12% of significant results could be false positive results (Soric, 1989). However, due to uncertainty in the estimate of the EDR, the upper limit of false positive results could be as high as 49%. The main problem is that it is unclear which of the published results are false positives and which ones are real effects. Another problem is that selection for significance inflates effect size estimates and that actual effects are likely to be smaller than published effect size estimates.

One solution to this problem is to focus on results with stronger evidence against the null-hypothesis by lowering the criterion for statistical significance. Some researchers have proposed setting alpha to .005. Figure 3 shows the implications of this criterion value.

The observed discovery rate is now only 34% because many results that were significant with alpha = .05 are no longer significant with alpha = .005. The expected discovery rate also decreases, but the more stringent criterion for significance lowers the false discovery risk to 3% and even the upper limit of the 95% confidence interval is only 20%. This suggests that most of the results with p-values below .005 reported a real effect. However, automatic extraction of test statistics does not distinguish between focal tests of unconscious thought theory and incidental tests of other hypotheses. Thus, it is unclear how many and which of these 184 significant results provide support for unconscious thought theory. The failed replication study by Nieuwenstein et al. (2017) suggests that it is not easy to find conditions under which unconscious thought is superior. In conclusion, there is presently little to no empirical support for unconscious thought theory, but research articles and literature reviews often cite the existing literature as if these studies can be trusted. The decrease in new studies suggests that it is difficult to find credible evidence.