Good science requires not only open and objective reporting of new data; it also requires unbiased review of the literature. However, there are no rules and regulations regarding citations, and many authors cherry-pick citations that are consistent with their claims. Even when studies have failed to replicate, original studies are cited without citing the replication failures. In some cases, authors even cite original articles that have been retracted. Fortunately, it is easy to spot these acts of unscientific behavior. Here I am starting a project to list examples of bad scientific behaviors. Hopefully, more scientists will take the time to hold their colleagues accountable for ethical behavior in citations. They can even do so by posting anonymously on the PubPeer comment site.
| Entry Date | Table of Incorrect Citations |
|---|---|
| 21/10/27 | Authors: Jürgen Kornmeier ,Kriti Bhatia,Ellen Joos Year: 2021 Citation: In the present paradigm, it is of course not difficult to comprehend the direct influence from past percepts of a disambiguated lattice figure on the perception of a highly similar but ambiguous lattice variant. In other precognition paradigms, such as some of those used in the experiments of the seminal Bem paper [85], the potential role of the perceptual history is not as directly comprehensible as in the present study– which does not necessarily rule it out. DOI: https://doi.org/10.1371/journal.pone.0258667 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/26 | Authors: B. Keith Payne1, Jason W. Hannay Year: 2021 Citation: One of the most important contributions from psychological science is the concept of implicit bias. Implicit bias refers to positive or negative mental associations cued spontaneously by social groups. It is measured using cognitive tasks that test how those associations facilitate, interfere with, or otherwise bias task performance [5,6]. Many studies suggest that implicit bias is widespread, even among people who explicitly endorse egalitarian attitudes [7,8]. Others argue that implicit bias is a stable trait- like construct, and that context effects or temporal fluctuations reflect only measurement error [50,51]. DOI: Correction: This quote and many other citations in this article fail to mention that the concept of implicit bias is controversial and lacks strong empirical support. There are many critical articles to cite, but my own criticism of the construct validity of implicit measures references most of them (https://doi.org/10.1177/1745691619863798). Another article directly criticizes Payne and is not cited (https://journals.sagepub.com/doi/abs/10.1177/1745691620931492). The authors cite my article [51], but fails to mention that it also contains evidence to support the claim that implicit racial bias measurs have only modest convergent validity with explicit racism measures and very little discriminant validity. |
| 21/10/25 | Authors: Cassandra Baldwin, Katie E. Garrison, Roy F. Baumeister & Brandon J. Schmeichel Year: 2021 Citation: Research has found that the capacity for executive control may work as if it depended on a limited resource. Effortful acts of control consume some of this resource, resulting in a state known as ego depletion (Baumeister et al., 1998; Muraven & Baumeister, 2000). DOI: 10.1080/15298868.2021.1888787 Correction: does not cite meta-analysis that shows publication bias and no evidence for the effect (https://doi.org/10.3389/fpsyg.2014.00823). Also does not cite two failed replication attempts in major RRR (https://doi.org/10.1177/1745691616652873, https://doi.org/10.1177/0956797621989733) |
| 21/10/25 | Authors: Liad Uziel, Roy F. Baumeister, and Jessica L. Alquist Year: 2021 Citation: Furthermore, temporary reduction in selfcontrol (following laboratory manipulations or activities such as alcohol consumption) often causes an increase in careless and impulsive acts (Baumeister et al., 2007; Hagger et al., 2010). DOI: https://doi.org/10.1037/mot0000213 Correction: cite an outdated meta-analysis that did not control for publication bias and fail to cite an updated meta-analysis that shows clear evidence of publication bias and no evidence for an effect (https://doi.org/10.3389/fpsyg.2014.00823), see also https://replicationindex.com/2016/04/18/is-replicability-report-ego-depletionreplicability-report-of-165-ego-depletion-articles/ |
| 21/10/20 | Authors: Nicole C. Nelson, Julie Chung, Kelsey Ichikawa, and Momin M. Malik Year: 2021 Citation: The second event Earp and Tramifow point to is the publication of psychologist Daryl Bem’s (2011) paper "Feeling the future,” which presented evidence suggesting hat people could anticipate evocative stimuli before they actually happened (such as the ppearance of an erotic image)." DOI: 10.1177/10892680211046508 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Gregory D. Webster, Val Wongsomboon, Elizabeth A. Mahar Year: 2021 Citation: To be sure, quantity need not reflect quality in published articles (e.g., see Bem’s [2011] nine-study article purporting experimental evidence of precognition). DOI: https://doi.org/10.1177/174569162199753 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Jason Chin, Justin T. Pickett, Simine Vazire, Alex O. Holcombe Year: 2021 Citation: The threat posed by QRPs has been discussed most extensively in the field of psychology, arguably the eye of the storm of the “replication crisis.” In the wake of the “False Positive Psychology” paper (Simmons et al. 2011), Daryl Bem’s paper claiming to find evidence of Extra Sensory Perception (ESP; Bem 2011), and several cases of fraud, the field of psychology entered a period of intense self-examination. DOI: https://doi.org/10.1007/s10940-021-09525-6 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Citation: Daryl Bem became notorious for publication of two articles in high-quality journals claiming the existence of ESP (Bem, 2011; Bem&Honorton, 1994). The experimental design and the statistical power looked persuasive enough to lead the editors and reviewers to a decision to publish despite the lack of a theory to explain the results. DOI: https://doi.org/10.3758/s13420-021-00474-5 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: T. D. Stanley, Hristos Doucouliagos, John P. A. Ioannidis, Evan C. Carter Year: 2021 Citation: Bem conducted some dozen(s) of experiments that asked students to “feel the future” by responding in the present to random future stimulus that was unknown to both subjects and experimenters at the time.46–49 Even though Bem seemed to employ state-of-the-art methods, his findings that students could “feel the future” were implausible to most psychologists. DOI: 10.1002/jrsm.1512 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Author: Guido W. Imbens Year: 2021 Citation: In the Journal of Personality and Social Psychology, Bem (2011) studies whether precognition exists: that is, whether future events retroactively affect people’s responses. Reviewing nine experiments, he finds (from the abstract): “The mean effect size (d) in psi performance across all nine experiments was 0.22, and all but one of the experiments yielded statistically significant results.” This finding sparked considerable controversy, some of it methodological. DOI: https://doi.org/10.1257/jep.35.3.157 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Mariella Paul, Gisela H. Govaart, Antonio Schettino Year: 2001 Citation: "Over the last decade, findings from a number of research disciplines have been under careful scrutiny. Prominent examples of research supporting incredible conclusions (Bem, 2011), DOI: https://doi.org/10.1016/j.ijpsycho.2021.02.016 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Andrew T. Little, Thomas B. Pepinsky Year: 2021 Citation: A prominent example here is Bem (2011) on extrasensory perception, which played a central role in uncovering the problems of p-hacking in psychology. DOI: https://doi-org.myaccess.library.utoronto.ca/10.1086/710088 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Bruno Verschuere, Franziska M. Yasrebi-de Kom, Iza van Zelm, MSc, Scott O. Lilienfeld Year: 2021 Citation: and the spurious “discovery” of precognition (Bem, 2011) DOI: https://doi-org.myaccess.library.utoronto.ca/10.1521/pedi_2019_33_426 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Lincoln J. Colling, Dénes Szűcs Year: 2021 Citation: A series of events in the early 2010s, including the publication of Bem’s (2011) infamous study on extrasensory perception (or PSI), and data fabrication by Diederik Stapel and others (Stroebe et al. 2012), led some prominent researchers to claim that psychological science was suffering a crisis of confidence (Pashler and Wagenmakers 2012). DOI: https://doi.org/10.1007/s13164-018-0421-4 Correction: Does not cite evidence that Bem used questionable research practices to produce these results (https://psycnet.apa.org/record/2012-23130-001, https://doi.org/10.3758/s13423-012-0227-9, https://replicationindex.com/2018/01/05/bem-retraction/) and that the results failed to replicate (https://doi.org/10.1037/a0029709) |
| 21/10/20 | Authors: Mόnika Gergelyfia, Ernesto J. Sanz-Arigita, Oleg Solopchuk, Laurence Dricot, Benvenuto Jacob, Alexandre Zénon Year: 2021 Citation: Theories of MF can be classified in two major groups that assume either: (a) alterations of motivational processes leading to restrictions on the recruitment of cognitive resources for the task at hand (…) or b) progressive functional alteration of cognitive processes through metabolic mechanisms ( Gailliot and Baumeis- ter, 2007 ; Christie and Schrater, 2015 ; Holroyd, 2015 ; Hopstaken et al., 2015 ; Blain et al., 2016 ; Gergelyfiet al., 2015 ). DOI: https://doi.org/10.1016/j.neuroimage.2021.118532 Correction: do not cite meta-analysis that shows publication bias and no evidence for glucose effects on willpower (https://doi.org/10.1177/0956797616654911) |
| 21/10/20 | Authors: Alexandra Touroutoglou, Joseph Andreano, Bradford C. Dickerson, Lisa Feldman Barrett Year: 2020 Citation: Some accounts hold that effort serves to manage intrinsic costs to finite resources such as metabolic resources (Gailliot and Baumeister, 2007; Gailliot et al., 2007; Holroyd, 2016), DOI: https://doi.org/10.1016/j.cortex.2019.09.011 Correction: do not cite meta-analysis that shows publication bias and no evidence for glucose effects on willpower (https://doi.org/10.1177/0956797616654911) |
| 21/10/10 | Authors: Scott W. Phillips; Dae-Young Kim Year: 2021 Citation: Johnson et al. (2019) found no evidence for disparity in the shooting deaths of Black or Hispanic people. Rather, their data indicated an anti-White disparity in OIS deaths. DOI: https://journals.sagepub.com/doi/10.1177/0093854821997529 Correction: Retraction (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Richard Stansfield, Ethan Aaronson, Adam Okulicz-Kozaryn Year: 2021 Citation: While recent studies increasingly control for officer and incident characteristics (e.g., Fridell & Lim, 2016; Johnson et al., 2019; Ridgeway et al., 2020) DOI: https://doi.org/10.1016/j.jcrimjus.2021.101828 Correction: Retraction of Johnson et al. (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: P. A. Hancock; John D. Lee; John W. Senders Citation: Misattributions involved in such processes of assessment can, as we have seen, lead to adverse consequences (e.g., Johnson et al., 2019). DOI: DOI: 10. 1177/ 0018 7208 2110 36323 Correction: Retraction (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Desmond Ang Citation: While empirical evidence of racial bias is mixed (Nix et al. 2017; Fryer 2019; Johnson et al. 2019; Knox, Lowe, and Mummolo 2020; Knox and Mummolo 2020) DOI: doi:10.1093/qje/qjaa027 Correction: Retraction of Johnson et al. (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Lara Vomfell; Neil Stewart Year: 2021 Citation: Some studies have argued that the general population in an area is not the appropriate comparison: instead one should compare rates of use of force to how often Black and White people come into contact with police [59–61] DOI: https://www.nature.com/articles/s41562-020-01029-w Correction: [60] Johnson et al. Retracted (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Jordan R. Riddell; John L. Worrall Year: 2021 Citation: Recent years have also seen improvements in benchmarking-related research, that is, in formulating methods to more accurately analyze whether bias (implicit or explicit) or racial disparities exist in both UoF and OIS. Recent examples include Cesario, Johnson, and Terrill (2019), Johnson, Tress, Burkel, Taylor, and Cesario (2019), Shjarback and Nix (2020), and Tregle, Nix, and Alpert (2019). DOI: https://doi.org/10.1016/j.jcrimjus.2020.101775 Correction: Retraction of Johnson et al. (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Dean Knox, Will Lowe, Jonathan Mummolo Year: 2021 Citation: A related study, Johnson et al. (2019), attempts to estimate racial bias in police shootings. Examining only positive cases in which fatal shootings occurred, they find that the majority of shooting victims are white and conclude from this that no antiminority bias exists DOI: https://doi.org/10.1017/S0003055420000039 Correction: Retraction of Johnson et al. (https://www.pnas.org/content/117/30/18130) |
| 21/10/10 | Authors: Ming-Hui Li, Pei-Wei Li ,Li-Lin Rao Year: 2021 Citation: The IAT has been utilized in diverse areas and has proven to have good construct validity and reliability (Gawronski et al., 2020). DOI: https://doi.org/10.1016/j.paid.2021.111107 Correction: does not cite critique of the construct validity of IATs (https://doi.org/10.1177/1745691619863798) |
| 21/10/10 | Authors: Chew Wei Ong, Kenichi Ito Year: 2021 Citation: This penalty treatment of error trials has been shown to improve the correlations between the IAT and explicit measures, indicating a greater construct validity of the IAT. DOI: 10.1111/bjso.12503 Correction: higher correlations do not imply higher construct validity of IATs as measures of implicit attitudes (https://doi.org/10.1177/1745691619863798) |
| 21/10/10 | Authors: Sara Costa, Viviana Langher, Sabine Pirchio Year: 2021 Citation: The most used method to assess implicit attitudes is the “Implicit Association Test” (IAT; Greenwald et al., 1998), which presents a good reliability (Schnabel et al., 2008) and validity (Nosek et al., 2005; Greenwald et al., 2009). DOI: doi: 10.3389/fpsyg.2021.712356 Correction: does not cite critique of the construct validity of IATs (https://doi.org/10.1177/1745691619863798) |
| 21/10/10 | Authors: Christoph Bühren, Julija Michailova Year: 2021 Citation: not available, behind paywall DOI: DOI: 10.4018/IJABE.2021100105 Correction: cite Caruso et al. (2013), but do not cite replication failure by Caruso et al. (2017) (https://journals.sagepub.com/doi/abs/10.1177/0956797617706161) |
| 21/10/10 | Authors: Yang, Gengfeng, Zhenzhen, Dongjing Year: 2021 Citation: "Studies have found that merely activating the concept of money can increase egocentrism, which can further redirect people's attention toward their inner motivations and needs (Zaleskiewicz et al., 2018) and reduce their sense of connectedness with others (Caruso et al., 2013). DOI: 10.1002/cb.1973 Correction: cite Caruso et al. (2013), but do not cite replication failure by Caruso et al. (2017) (https://journals.sagepub.com/doi/abs/10.1177/0956797617706161) |
| 21/10/10 | Authors: Garriy Shteynberg, Theresa A. Kwon, Seong-Jae Yoo, Heather Smith, Jessica Apostle, Dipal Mistry, Kristin Houser Year: 2021 Citation: Money is often described as profane, vulgar, and filthy (Belk & Wallendorf, 1990), yet incidental exposure to money increases the endorsement of the very social systems that render such money meaningful (Caruso et al., 2013). DOI: 10.1002/jts5.95 Correction: cite Caruso et al. (2013), but do not cite replication failure by Caruso et al. (2017) (https://journals.sagepub.com/doi/abs/10.1177/0956797617706161) |
| 21/10/10 | Author: Arden Rowell Year: 2021 Citation: In particular, some studies show that encouraging people to think about things in terms of money may measurably change people's thoughts, feelings, motivations, and behaviors. See Eugene M. Caruso, Kathleen D. Vohs, Brittani Baxter & Adam Waytz, Exposure to Money Increases Endorsement of Free-Market Systems and Social Inequality, 142 J. EXPERIMENTAL PSYCH. 301, 301-02, 305 (2013) DOI: https://scholarship.law.nd.edu/ndlr/vol96/iss4/9 Correction: cite Caruso et al. (2013), but do not cite replication failure by Caruso et al. (2017) (https://journals.sagepub.com/doi/abs/10.1177/0956797617706161) |
| 21/10/10 | Authors: Anna Jasinenkoa, Fabian Christandl, Timo Meynhardt Year: 2020 Citation: Caruso et al. (2013) find that exposure to money (which is prevalent in most shopping situations) activates personal tendencies to justify the market system. Furthermore, they find that money exposure also activates general system justification; however, he effect was far smaller than for the activation of MSJ. DOI: https://doi.org/10.1016/j.jbusres.2020.04.006 Correction: cite Caruso et al. (2013), but do not cite replication failure by Caruso et al. (2017) (https://journals.sagepub.com/doi/abs/10.1177/0956797617706161) |
Some of these critiques of citations appear to me to be pretty superficial. For example Knox/Lowe/Mummolo APSR webpage says it was published online May-2020 (before the Johnson retraction in July 2020).
You think people should be on the hook to issue corrections for *citing* papers that are subsequently retracted? That seems to me to be quite excessive. (The Johnson paper was retracted *because* of the work by Knox/Lowe/Mummolo, so they would ultimately need to include a citation to the original article anyway.)
Thank you for pointing out that the article was published online before the retraction was published. While this means that the authors are not at fault, it is still useful to alert readers of the retraction.
It seems that all of the papers that cite Bem here are using it as an example of a study which was an impetus for the credibility crisis and some implicitly or near explicitly point towards it as an example of QRP, for example, Chin et al which feature the QRP in the paper title even. No-one is trying to claim that ESP is real here (not to say some parapsychologists might).
So these are very different cases to the ego depletion ones.
It does seem quite bizarre to go to this effort, since the “bad scientific behaviors” you are referring to are that whenever anyone cites Bem for any reason, they are doing bad science if they don’t cite Schimmack (2012) as well at the same time.
Should articles that are critical of Bem’s (2011) article cite articles that pointed out flaws in Bem (2011) such as Schimmack (2012)? I think so. After all, how can you cite Bem (2011) to question the claims in the article. The article is not self-critical, it has not been retracted and we should treat it like any other article in scientific journals. The claims in the article are based on solid empirical foundations. If you want to imply that this is not the case, it is not good enough to say “but we know ESP” does not exist. That is not a scientific argument. So, to criticize it you should cite replication failures or evidence of QRPs.
I think you do understand how writing works, and how people cite work in particular contexts for particular purposes and audiences as part of a paragraph structure.
ok, do you think this quote here is an example of unethical and unscientific behaviour (unfortunately a blog post so you cannot add to pubpeer if you did):
“However, in 2011 an article by a well-respected social psychologists changed all this. Daryl Bem published an article that showed time-reversed causal processes. Seemingly, people were able to feel the future (Bem, 2011). This article shock the foundations of social psychology because most social psychologists did not believe in paranormal phenomena. Yet, Bem presented evidence for his crazy claim in 8 out of 9 studies. The only study that did not work was with supraliminal stimuli. The other studies used subliminal stimuli, suggesting that only our unconscious self can feel the future.”
source: https://replicationindex.com/2020/12/05/implicit-love/
I suspect you do not think so, and you could explain why in this particular case you did not also cite those two papers you mention.
If you can perform such an explanation, I would imagine you could also use some theory of mind, logic and reason to explain for these other paragraphs why they did not also cite those two papers given their narrative goals in the context of the paper and paragraph, e.g.:
“Daryl Bem’s paper claiming to find evidence of Extra Sensory Perception (ESP; Bem 2011), and several cases of fraud, the field of psychology entered a period of intense self-examination…”
I think people can see the point you are making with papers such as those with Baumeister uncritically citing evidence for ego-depletion without referring to some of the disputed evidence claims. But by including them along with all these examples with Bem about the replication crisis and tarring with the same brush you devalue your more legitimate criticisms, and make it easier for you to be dismissed. Particularly when you take issue with them not citing you.