Overview: A New Approach to Evaluating Peer Review Practices
This multi-journal initiative tests whether a structured editorial policy—in-principle acceptance for revision plans—can reduce questionable research practices during peer review.
Journals retain full editorial control and decision-making. The Center for Open Science provides protocol templates, IRB documentation, onboarding, and support.
What are Registered Revisions?
Pre-commitment tools like Preregistration and Registered Reports may help mitigate publication bias and questionable research practices, but their adoption has been slow, and their impact remains difficult to assess. Registered Revisions aim to introduce pre-commitment practices into the peer review process and to assess their effects.
Registered Revisions is a policy that activates when peer reviewers request new data or analyses. Instead of ad hoc revisions, authors draft a revision plan which, if approved, grants in-principle acceptance (IPA). This approach may promote transparency, reduce uncertainty and multiple review rounds, and help limit questionable research practices and publication bias during peer review.
How it Works
COS is testing this approach through a collaborative, multi-journal set of randomized experiments on Registered Revisions. Participating journals implement a ready-to-use study design within their existing editorial workflows. COS provides infrastructure, support, and IRB pathways. Journals retain full ownership of their trial and data, and results across journals will be aggregated in a meta-analysis.
Get Involved
We're currently inviting journal editors and publishers to join this initiative and help build the evidence base for scalable, transparent improvements to the peer review process.
Additional details, including detailed protocols and the data and code repository will be made available on our OSF page here.
This research is funded by the NSF (grant #2152424) and was was approved by the University of Virginia Institutional Review Board (Protocol #6358).
Each participating journal runs its own randomized controlled trial (RCT) to evaluate the Registered Revisions policy. The study is designed to fit within existing editorial workflows, using a common protocol developed by the Center for Open Science (COS).
A Shared Design, Implemented Individually
Journals use a boilerplate study protocol that standardizes core elements of the trial while allowing for local adaptation. COS provides all the materials needed to launch the study, including trial templates, author instructions, and editorial onboarding.
Support at Every Step
COS offers technical infrastructure, IRB documentation, randomization tools, and hands-on support to help journals implement the trial with minimal disruption. Editorial teams retain control of the peer review process. A detailed protocol and implementation materials are available on our OSF project.
How the Trial Works
When peer reviewers request new data or analyses, eligible manuscripts are randomized to one of two arms:
Ownership and Data Use
Journals maintain full ownership of their trial data and any resulting publications. COS coordinates the aggregation of data across participating sites to conduct a meta-analysis of policy effects.
This model allows journals to evaluate a promising policy intervention while contributing to a larger effort to explore whether embedded experimentation can support evidence-based improvements in peer review.
Project materials and documentation on OSF → https://osf.io/tshqu/
This research is funded by the NSF (grant #2152424) and was approved by the University of Virginia Institutional Review Board (Protocol #6358).
Participating in the Registered Revisions project gives journals the opportunity to contribute to evidence-based improvements in peer review, while maintaining control over their editorial processes and data.
Advance transparency without overhauling your workflow
This study allows journals to pilot a promising policy, Registered Revisions, without needing to adopt it permanently or system-wide. The study design is embedded in your current workflow, and COS provides the tools and support to make implementation straightforward.
Retain full control over your process and outcomes
Editors continue to manage review and make editorial decisions. Journals retain ownership of their data and any resulting publications.
Flexibility and opportunities for expansion
The project is designed to work with your journal's needs to produce the most realistic evidence possible. We are also happy to add on to questionnaires and processes to integrate new questionnaires or other studies.
Contribute to a broader initiative with local impact
While each trial is individually run, the collective data will be pooled in a meta-analysis to assess policy effects across contexts. Participating journals also help build the foundation for future policy experimentation in scholarly publishing, an area where rigorous testing has been rare but increasingly necessary.
Advance a new generation of evidence generation and practice
Large scale experimental evidence for research policy is difficult to come by, due in large part due problems in coordination, logistics, and incentives. This project represents a new way forward for simultaneously implementing experimental policies while developing strong experimental evidence. You’ll join a growing group of journals committed to transparency, innovation, and collaboration. COS works closely with each journal to ensure the study reflects your goals and capacities. Participating editors often tell us they appreciate the opportunity to try something new while knowing they’re helping knowledge generation in the field.
Project materials and documentation on OSF → https://osf.io/tshqu/
This research is funded by the NSF (grant #2152424) and was approved by the University of Virginia Institutional Review Board (Protocol #6358).
The Registered Revisions project builds on a successful pilot phase completed in 2025. In partnership with a group of seven journals, we tested the full study design, from editorial onboarding and author workflows to technical infrastructure and data aggregation.
As of May 31, 2025, 141 authors signed consent to participate, 24 manuscripts were randomized, and 5 reached a final editorial decision within the study period. Editors provided broadly positive feedback on the study design, infrastructure, support materials, and overall feasibility.
Participating pilot journals included:
The pilot demonstrated that Registered Revisions can be implemented within diverse editorial systems and that embedded experimentation is feasible with minimal disruption. We refined materials and clarified decision points for editors to improve the experience. Pilot materials are available on our OSF project.
With the core protocol in place and lessons from the pilot integrated, we’re now expanding to a broader set of journals.
Project materials and documentation on OSF → https://osf.io/tshqu/
This research is funded by the NSF (grant #2152424) and was approved by the University of Virginia Institutional Review Board (Protocol #6358).
We’re currently inviting academic journals to join the next phase of the Registered Revisions project.
Participation involves running a randomized trial using a shared protocol, with support from the Center for Open Science throughout. Journals retain full ownership of their study and data and contribute to a larger effort to evaluate this policy and the potential of embedded experimentation in publishing more broadly. Explore full documentation of the project on our OSF project.
Whether you’re ready to join or just want to learn more, we’re happy to connect.
Contact:
Noah Haber – noah@cos.io
Macie Daley – macie@cos.io
Project materials and documentation on OSF → https://osf.io/tshqu/
This research is funded by the NSF (grant #2152424) and was approved by the University of Virginia Institutional Review Board (Protocol #6358).
What outcomes are you measuring in the study?
We’re primarily interested in how the Registered Revisions policy affects the peer review process. The main outcome is time from submission to final decision. We’re also tracking how many participating manuscripts reach a final decision, how often null results are reported, and how authors and editors experience the process. We ask about things like whether the policy was helpful, burdensome, or changed how participants think about transparency in peer review.
What’s in it for me (as an editor or journal)?
This is a chance to contribute to growing knowledge about how journal policies affect research transparency and publication outcomes. Editors and journals can publish their results - either independently or in collaboration with COS - and we provide support with study materials, data infrastructure, and even writing if helpful. It’s also a low-burden way to pilot a new editorial policy with a clear structure and defined scope.
My journal rarely asks for new data or analyses during review. Can we still participate?
Yes. We welcome journals with low volumes of eligible manuscripts. The study design allows for variation in how often the policy is triggered, and even small amounts of data help us assess feasibility and variation across journals. We’re also interested in whether this kind of embedded experimentation is workable across different publishing contexts, not just in high-output settings.
What qualifies a manuscript for inclusion in the study?
When a reviewer requests new data or analysis during peer review.
Do we need IRB approval?
COS provides IRB pathways and templates. Most sites rely on COS’s central IRB, but local processes can be accommodated.
How is randomization handled?
COS handles randomization through its infrastructure and provides clear workflows for editorial teams.
How long does it take to get started?
Most journals complete onboarding and launch within 4–6 weeks.
Do we need to implement the policy across all submissions?
No. The policy is only applied to eligible manuscripts within the trial, and it does not require permanent adoption.
Project materials and documentation on OSF → https://osf.io/tshqu/
This research is funded by the NSF (grant #2152424) and was approved by the University of Virginia Institutional Review Board (Protocol #6358).
6218 Georgia Avenue NW, Suite #1, Unit 3189
Washington, DC 20011
Email: contact@cos.io
Unless otherwise noted, this site is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.
Responsible stewards of your support
COS has earned top recognition from Charity Navigator and Candid (formerly GuideStar) for our financial transparency and accountability to our mission. COS and the OSF were also awarded SOC2 accreditation in 2023 after an independent assessment of our security and procedures by the American Institute of CPAs (AICPA).
We invite all of our sponsors, partners, and members of the community to learn more about how our organization operates, our impact, our financial performance, and our nonprofit status.
