Open science is a movement about improving the strength and reliability of our science through uncovering both the results of science and the process underlying it.
Our ongoing Registered Revisions project pioneers both a new policy and a new multi-trial meta design. While the end designs were ultimately fairly streamlined, we needed to do quite a bit of inventing, developing, testing, and iteration. Rather than simply describe that final design, this report traces the paths we explored to get there, and what didn’t quite work like expected. The Registered Revisions Meta Trial Pilot Report was released on MetaArXiv on September 8, 2026, and we’re excited to show you a preview of what you can find in the report.
Registered Revisions as a peer review policy
This exchange between Jeremy Freese and Julia Rohrer nicely describes the policy problem we are trying to address, why it is important, and how Registered Revisions works to address it. When reviewers and editors imply a need for new analysis, authors are put in a high stakes position with relatively little information on how to proceed or what might result in an acceptance. The Registered Revisions policy borrows the central premise of Registered Reports: make an acceptance decision based on the soundness of the new analysis plan to address these comments, regardless of the results.
The impact evaluation problem and the meta trial design
Obtaining randomized trial evidence for Registered Revisions shares a problem that is common for policy impact trials: A single randomized trial with many idiosyncratic policy units (journals in our case) is difficult at best. The first part of our report describes why our initial single trial attempt did not ultimately launch. In practice, the incentives structure, inflexibility inherent in a single trial design, and coordination issues proved too large a hurdle for journals to overcome. We needed to design something else.
The meta trial idea came from going all the way back to the drawing board (in this case very literally; the first meta trial proposal was drawn on a whiteboard when COS still had a physical office). Rather than one big trial, we fostered journals’ own individual trials under a centralized infrastructure, designed to all work together. The meta trial design draws on multi-center clinical trials and prospective meta analyses, allowing individual journal trials to contribute to a shared body of evidence while retaining flexibility in implementation.
Theory to practice: the Pathfinder pilot
The initial ideas for the Registered Revisions policy and meta trial are workable in theory, but we needed to make them work in practice. The Pathfinder Pilot consisted of seven journals whose editorial teams helped us co-design, refine, and test the pieces needed to turn these ideas into functional implementations. At the end of the pilot, we went back to all the editors and asked them how things went, and compiled and reported it here. What went smoothly, what was frustrating, and what do we think about the current designs?
Policy implementation design
How would the Registered Revisions policy work with editors, authors, and reviewers? After much iteration and refining, we landed on a system of templates that mirrors the same steps as in standard peer review, but with embedded instructions and specific prompts for authors, editors, and reviewers.
Trial Workflow Design
Shared infrastructure and design
One of the key benefits of the meta trial design is shared resources among the individual trials. We developed a “study-in-a-kit” approach, where everything from ethics/institutional review board approval to participant and data management systems to analysis code was shared amongst all participating journals. Our report describes how we were able to balance centralized resources and design while building in flexibility for journals’ own preferences and styles where they matter most.
Editorial workflow
Just as with the policy itself, we needed a system by which editors could administer informed consent, track participants, record data, and generally manage their individual trials. We developed a webapp to handle participant management, designed so that it did not rely on or need integration with existing journal editorial management software. As in the policy itself, the editorial workflow was designed to align with existing editorial touchpoints, such as sending the paper out to reviewers or when authors return a revision. All trial-related steps for authors fit within existing workflows.
A number of unusual design features in this project are described in the report, such as why and how most journals ask for informed consent before screening. While our editor team appreciates the system now, the road to getting there was not all smooth sailing. In the report, we describe some of the biggest hurdles we encountered, including software bugs - both ours and from external sources - that created substantial implementation difficulties.
... and more
This report is for anyone who thinks deeply about journal policy or study design. We are showing our work so that everyone can learn from our successes and failures. That includes topics like:
Why did we choose time-to-event process based outcomes as our primary outcomes over research based outcomes like statistical significance?
How did we develop an ethics/institutional review board system that precluded the need for every journal trial to have their own individual applications for each of their studies?
What do editors think about the editorial workloads from both the policy and the trial?
What did a simulation-based engine add to both design and planning for the project?
What mistakes did we make along the way that you can avoid?
How did this trial impact the way editors think about their role in the peer review process?
Let us know what you think! Would you like to see more in-depth reports like this one? Have more questions about our process? Send us an email at noah@cos.io or join the conversation on Bluesky.