5 thoughts on “Social psychology textbook audit: Something smells fishy

  1. Quote from above: “It is time to get rid of the fishy smell and let the fresh wind of open science clean up social psychology.”

    “Open Science” is all fine and good, but i reason it is not a solution to a lot of problems in Science, possibly including cleaning up Social Psychology.

    To try and illustrate my point, here is the data from the “famous” paper by Simmons, Nelson, & Simonsohn (2011) “False positive Psychology” paper https://openpsychologydata.metajnl.com/articles/10.5334/jopd.aa/.

    Now imagine they didn’t write their “False positive” paper, but p-hacked away and presented their finding that listening to The Beatles “When i’m 64” makes you younger (or whatever they found), and made sure their data were “open” (just like in the link above). Would that mean that their findings are “trustworthy” and/or “correct”? I would reason not.

    Also, i reason that Science should work so that the best and brightest people are working on/in it. I view it as making sure planet Earth selects the best people to run in the 4 x 100 meter relay of the “Intergalactic Olympics”. Just like doping controls make it possible to not select (or reward) doping users, Open Science could make sure that people who p-hack, selectively report things, etc. are not selected. However, that’s just one part of trying to select the best people (runners or scientists). Or to use a different example, if we were to let 7 year old kids perform Social Psychological studies (or other studies from different fields) in an “open” and “transparten” manner, would that lead to “good” science. I would reason not.

    In other words:

    1) I agree with Gelman who wrote that honesty and transparency is not enough https://statmodeling.stat.columbia.edu/2017/05/09/honesty-transparency-not-enough/

    2) I also worry that (some? many?) people will only think something is or is not “good” science because (some ? many?) people associate it with “Open Science”. This could lead to not really thinking about matters anymore, or why something is or is not “open science”, and if that is important and why.

    For instance, can you point me to (some? many?) “Open Science” people who spotted, and ringed the alarm bell, when many “Registered Reports” dit not even provide the reader access to the crucial “pre-registration” in their paper? (also see “Mapping the universe of Registered Reports” by Hardwicke & Ioannidis https://osf.io/preprints/metaarxiv/fzpcy/)

    Like

    1. Quote from above: “This could lead to not really thinking about matters anymore, or why something is or is not “open science”, and if that is important and why.”

      Oh, and can you point me to “Registered Replication Reports” that pre-registered the exact labs that will perform the replication study?

      That seems very important (and transparent) to me, because otherwise people could just leave out the results of (some? many?) labs if they do not like the results of those labs.

      To me, pre-registering the exact labs that will perform the study seems even way more important to do then pre-registering the analyses concerning “Registered Replication Reports”. This is because i reason that the analysis is already “registered” in a way in the “original” paper that is being replicated.

      Can you point me to (some? many?) “Open Science” people who spotted, and ringed, the alarm bell about that?

      Like

      1. You wrote: “I don’t know sources but many lab replication projects are often preregistered and well documented on OSF. ”

        It was sort of a rhetorical question, as my investigation of these “Registered Replication Reports” have lead to me not being able to find any pre-registration information about the exact labs that will participate. But please correct me if i am wrong.

        The “Registered Replication Reports” are also pre-registered and well documented on the OSF, but the point i am trying to make is that that doesn’t (apparently) mean it is “open” or “transparent” concerning (what i think are) crucial things.

        For example, here is the “pen-in-mouth” Strack et al. “Registered Replication Report” (we all know and love :P). On page 5 of the paper there is a link depicted to the pre-registration https://osf.io/h2f98/. Now, that leads to a page where you can download a few things, but the thing that (i assume) is the actual pre-registration seems to me to only talk about the analyses, and does not mention the exact labs who will participate https://osf.io/4bnxm/

        As i tried to make clear above with the 2 posts is that 1) leaving out the crucial information in, and to, the -pre-registration information and, 2) simply assuming, and not even thinking about, why something is or is not “open science” seems highly problematic.

        Just because (some? many?) people associate something with “open science” doesn’t mean it’s “good” science, nor does it (apparently) mean it’s even “open” and “transparent”.

        Like

      2. Interesting observation. You could do a bias analysis of the results. I did so for facial feedback to investigate Strack’s accusation of reverse p-hacking and found no bias one way or the other.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s