Shaheed Azaad ·
The Systematizing Confidence in Open Research and Evidence (SCORE) project recently published a large-scale replication project within the social and behavioural sciences. The authors replicated only 151 of 274 (55%) claims, a finding covered by major outlets such as the New York Times and Forbes. Here, I audited SCORE replications of six studies where at least one claim surprisingly failed to replicate. These failures were surprising because the original study produced a p < .01 while the replication yielded a significant effect in the opposite direction. I show that, of the 12 claims tested across these six replications, five were miscategorised as failed replications because of data analysis and interpretation problems, and six replications were invalid because of poor data quality and an error in the replication’s experiment code. I conclude that individual replications within replicability projects may benefit from more thorough peer review, and that a more comprehensive audit is necessary before we can be confident in the project’s overall replicability estimate.