Brian Nosek has been one of the most influential figures in the open science movement. As co-founder of the Center for Open Science and a leading voice in efforts to reform psychological research, he has played a central role in changing how scientists think about transparency, reproducibility, and methodological rigor.
1. Nosek’s leadership in the open science movement
Nosek was a key organizer of the Reproducibility Project: Psychology (Science, 2015), a landmark collaboration that revealed how difficult it was to replicate many published findings in psychology. The largest replication failures were concentrated in his own field of social psychology. This project helped catalyze sweeping reforms, including preregistration, open data, open materials, and improved statistical practices.
2. Reforms have addressed the replication crisis—but not the validation crisis
While these reforms have improved replicability, psychology also suffers from a validation crisis (Schimmack, 2021): many widely used measures have never been rigorously validated. Some lack evidence for reliability or construct validity. Others are used in contexts where accuracy is crucial—such as providing personal feedback or informing high-stakes decisions—despite insufficient psychometric support.
Replication ensures that results can be reproduced across studies. Validation ensures that a measure actually captures the construct it claims to measure. The second problem is deeper, and in many ways more fundamental. What is the point of replicating a result with an invalid measure?
3. Project Implicit and the IAT as an example of the validation crisis
Nosek co-founded Project Implicit, a public-facing website that provides users with interpretations of their supposed “implicit attitudes” using the Implicit Association Test (IAT). These interpretations include highly self-relevant domains such as racial prejudice, depression, or suicidality.
4. What the IAT can measure — and what it has never been shown to measure
To be clear, this critique is not about whether the IAT has some validity. It does. The IAT correlates modestly with self-report measures of the same constructs, and for statistical reasons it can show small increments in predictive validity when both measures are combined. This is consistent with treating the IAT as an alternative, less reliable way of measuring the same underlying attitudes that people can report about themselves.
The real issue is different:
The IAT has been widely promoted and interpreted as measuring a distinct “implicit” construct that is not accessible to introspection. That claim has never been scientifically validated.
After more than 20 years of research, there is no evidence that the IAT reveals hidden attitudes or unconscious biases that individuals cannot detect through honest self-reflection or awareness of their own thoughts, feelings, and behavior.
Many uses of the IAT—including individualized feedback on Project Implicit—depend on the assumption that the test measures something qualitatively different from self-report. Without evidence for such a construct, these uses have no scientific justification.
5. A blind spot: inconsistent standards between open-science ideals and IAT feedback
In his open-science work, Nosek strongly advocates for:
- transparency
- rigorous validation
- honest communication of uncertainty
- avoidance of overstated claims
- reliable measurement
- accurate interpretation of data
Yet Project Implicit continues to provide categorical, diagnostic-like feedback (“slight,” “moderate,” “strong” bias) without reporting confidence intervals, measurement error, or test–retest variability. Users often interpret these labels as precise and meaningful, even though the underlying measurement error is large enough to make the point estimate uninformative (Schimmack, 2025).
When concerns about validity have been raised—including the absence of evidence that the IAT measures anything inaccessible to introspection—Nosek has not addressed them directly. Instead, he notes that the IAT should not be used for high-stakes decisions and that Project Implicit includes disclaimers. However, these disclaimers appear on secondary pages, are not shown alongside the feedback, and use technical language that lay users are unlikely to understand (Schimmack, 2025).
There is also an important ethical dimension. In psychological research, any study that provides participants with false or misleading feedback requires explicit IRB approval for deception and a thorough debriefing that explains the deception clearly. The feedback practices used by Project Implicit would not meet these ethical standards: users are led to believe that their scores meaningfully reflect “implicit attitudes,” yet they are never fully debriefed that individual IAT results are too unreliable and invalid to support such interpretations. Instead of a clear debriefing, users receive vague disclaimers that fall short of what would ordinarily be required in research involving misleading information.
This is another example of the blind spot: practices that would be flagged as ethically problematic in research settings are treated as acceptable, when they require reforms to communicate the limitations of psychological science openly, transparently, and with humility.
6. Neglect of scientific criticism conflicts with open-science reforms
Open science demands:
- transparent reporting
- clear communication of uncertainty
- avoidance of claims unsupported by evidence
- consistent standards across all methods
However, scientific criticisms of the IAT—especially concerns about its lack of construct validity as an individual diagnostic tool—have not been addressed with the same rigor and candor that Nosek applies to other areas of psychological science.
This inconsistency creates a tension between open-science ideals and the continued public presentation of the IAT.
7. The psychological irony: a blind spot about implicit bias
This post highlights a tension that is obvious to outside observers: there is a conflict between championing transparency and honesty in psychological science and overlooking substantial, long-standing criticism of the IAT as a valid measure of implicit attitudes.
The replication crisis has shown that such contradictions are often difficult to recognize from within a research tradition, and only a few scientists have openly acknowledged being wrong (Kahneman, 2017).
In this sense, Brian Nosek’s response is a classic example of what psychologists call a “blind spot.” This does not imply anything about intent; blind spots operate outside of conscious awareness. A direct response from Brian Nosek would help clarify whether he sees this contradiction himself. Interestingly, empirical research suggests that participants are often aware of the biases that the IAT claims to reveal. It would therefore be informative to learn whether Nosek is aware of the discrepancy between advocating for open-science reforms and defending the continued use of the IAT feedback on Project Implicit.
8. The way forward: applying open science to measurement validity
For psychology to continue improving as a science, the standards of open science—transparency, empirical humility, rigorous validation—must apply equally to all psychological measures, including influential legacy instruments like the IAT.
For Project Implicit, this would require:
- avoiding categorical labels without uncertainty metrics
- clear communication of measurement error
- confidence intervals around individual scores
- transparent statements about what the IAT can and cannot measure
- refraining from implying that the test reveals hidden mental content
Open science has pushed the field forward by demanding uncomfortable honesty about methods. Addressing the blind spot around IAT validity would extend those principles where they are most needed.