Statistical Review of Stage 1 and Stage 2 Registered Reports

As part of our ongoing efforts to strengthen the rigor of the Registered Reports pilot, COS and Meta have introduced an additional layer of statistical review across all participating studies, including those that have already received in-principle acceptance (IPA) and were published at Stage 1. We believe having a review focused on the statistical approach described in the Stage 1 manuscript will add additional benefit by mitigating any potential biases or ambiguity in the analysis plan before data collection. Additionally, this will improve the reproducibility and interpretability of the Stage 2 manuscript when results are known.

This additional review is aimed at evaluating both the Stage 1 and forthcoming Stage 2 submissions, with the latter including a reproducibility check of the analyses. The reviewer is an independent statistician contracted by COS, with no involvement or oversight from Meta. Like a typical peer reviewer, they provide feedback to the editor, who then works with authors to determine any needed revisions. For previously published studies, any updates will be reflected in an updated manuscript.

The purpose of this review is:

To ensure that the proposed analyses are clearly specified and appropriately aligned with each study’s hypotheses and design. The statistical reviewer helps confirm that each planned analysis maps to a research question, that the interpretation of results is appropriate given the methods, and that no important elements are missing or underspecified.

Even though this step is happening after IPA for some papers, we believe it is a constructive way to help position each project for long-term success. Defining robust and transparent methods in Stage 1 Registered Reports can be challenging, especially with teams that bring diverse disciplinary backgrounds. This additional review is meant to provide extra guidance to support that aim.

We recognize that the timing of this intervention, post-publication in some cases, isn’t ideal and represents one of several lessons we’re taking from this pilot. We’ll continue to share updates as the pilot progresses and as we learn more about how to best support this model of research transparency and rigor.