The Registered Revisions Project:    
Evaluating Innovations in Peer Review
Questions or partnership opportunities? Contact Noah Haber (noah@cos.io) and Macie Daley (macie@cos.io).

Overview

Testing Innovations in Peer Review

The Registered Revisions project is an academic collaboration testing whether a new journal policy can help reduce questionable research practices (QRPs) and publication bias during peer review.

Pre-commitment tools like Pregistration and Registered Reports may help mitigate publication bias and questionable research practices, but their adoption has been slow, and their impact remains difficult to assess. Registered Revisions is a policy designed to address that: authors respond to requests for new data or new analyses during peer review by drafting a revision plan, and if it's approved, they receive in-principle acceptance (IPA) before doing the additional work. 

We're testing this approach through a collaborative, multi-journal set of randomized experiments on Registered Revisions. Participating journals implement a ready-to-use study design within their existing editorial workflows. The Center for Open Science provides infrastructure, support, and IRB pathways. Journals retain full ownership of their trial and data, and results across journals will be aggregated in a meta-analysis.

Between journals 

rc3t mini 1 crop

 

 

Within journals 

r3ct mini 2 crop

 

We're currently inviting journal editors and publishers to join this initiative and help build the evidence base for scalable, transparent improvements to the peer review process.

Additional details, including detailed protocols and the data and code repository will be made available at our OSF page here: https://osf.io/tshqu/.


This research is funded by the NSF (grant #2152424)

How it Works

A Collaborative Set of  Studies, Built into Existing Editorial Workflows

rct graph 2

Each participating journal runs its own randomized controlled trial (RCT) to evaluate the Registered Revisions policy. The study is designed to fit within existing editorial workflows, using a common protocol developed by the Center for Open Science (COS).

A Shared Design, Implemented Individually

Journals use a boilerplate study protocol that standardizes core elements of the trial while allowing for local adaptation. COS provides all the materials needed to launch the study, including trial templates, author instructions, and editorial onboarding.

Support at Every Step
COS offers technical infrastructure, IRB documentation, randomization tools, and hands-on support to help journals implement the trial with minimal disruption. Editorial teams retain control of the peer review process. A detailed protocol and implementation materials are available on our OSF project.

How the Trial Works

rct chart new
When peer reviewers request new data or analyses, eligible manuscripts are randomized to one of two arms:

  • In the Registered Revisions arm, authors are asked to submit a revision plan describing the additional work. If the plan is approved, the manuscript receives in-principle acceptance (IPA) before the work is carried out.
  • In the standard condition, authors proceed with the review process as usual.

Ownership and Data Use
Journals maintain full ownership of their trial data and any resulting publications. COS coordinates the aggregation of data across participating sites to conduct a meta-analysis of policy effects.

This model allows journals to evaluate a promising policy intervention while contributing to a larger effort to explore whether embedded experimentation can support evidence-based improvements in peer review.


Project materials and documentation on OSF → https://osf.io/tshqu/ 

This research is funded by the NSF (grant #2152424)

Why Participate

Why Partner with Us

Participating in the Registered Revisions project gives journals the opportunity to contribute to evidence-based improvements in peer review, while maintaining control over their editorial processes and data.

Advance transparency without overhauling your workflow
This study allows journals to pilot a promising policy, Registered Revisions, without needing to adopt it permanently or system-wide. The study design is embedded in your current workflow, and COS provides the tools and support to make implementation straightforward.

Retain full control over your process and outcomes
Editors continue to manage review and make editorial decisions. Journals retain ownership of their data and any resulting publications. 

Flexibility and opportunities for expansion
The project is designed to work with your journal's needs to get the most realistic evidence possible. We are also happy to add on to questionnaires and processes to integrate new questionnaires or other studies.

Contribute to a broader initiative with local impact
While each trial is individually run, the collective data will be pooled in a meta-analysis to assess policy effects across contexts. Participating journals also help build the foundation for future policy experimentation in scholarly publishing, an area where rigorous testing has been rare but increasingly necessary.

Advance a new generation of evidence generation and practice
Large scale experimental evidence for research policy is difficult to come by, due in large part due problems in coordination, logistics, and incentives. This project represents a new way forward for simultaneously implementing experimental policies while developing strong experimental evicence. You’ll join a growing group of journals committed to transparency, innovation, and collaboration. COS works closely with each journal to ensure the study reflects your goals and capacities.

Participating editors often tell us they appreciate the opportunity to try something new while knowing they’re helping knowledge generation in the field.

 


Project materials and documentation on OSF → https://osf.io/tshqu/ 

This research is funded by the NSF (grant #2152424)

Progress to Date

Where We're at Now

The Registered Revisions project builds on a successful pilot phase completed in 2025. In partnership with a group of seven journals, we tested the full study design, from editorial onboarding and author workflows to technical infrastructure and data aggregation. 

In total, 141 authors signed consent to participate, 24 manuscripts were randomized, and 5 reached a final editorial decision within the study period. Editors provided broadly positive feedback on the study design, infrastructure, support materials, and overall feasibility.

Participating pilot journals included:

  • Analyses of Social Issues and Public Policy
  • Collabra: Psychology
  • Entrepreneurship Theory and Practice
  • Evidence Based Toxicology
  • Journal of Media Psychology
  • Leadership Quarterly
  • Science & Medicine in Football

The pilot demonstrated that Registered Revisions can be implemented within diverse editorial systems and that embedded experimentation is feasible with minimal disruption. We refined materials and clarified decision points for editors to improve the experience. Pilot materials are available on our OSF project.

With the core protocol in place and lessons from the pilot integrated, we’re now expanding to a broader set of journals.


Project materials and documentation on OSF → https://osf.io/tshqu/ 

This research is funded by the NSF (grant #2152424)

 

Get Involved

Get Involved

We’re currently inviting academic journals to join the next phase of the Registered Revisions project.

Participation involves running a randomized trial using a shared protocol, with support from the Center for Open Science throughout. Journals retain full ownership of their study and data and contribute to a larger effort to evaluate this policy and the potential of embedded experimentation in publishing more broadly. Explore full documentation of the project on our OSF project.

Whether you’re ready to join or just want to learn more, we’re happy to connect.

Contact:
Noah Haber – noah@cos.io
Macie Daley – macie@cos.io

 


Project materials and documentation on OSF → https://osf.io/tshqu/ 

This research is funded by the NSF (grant #2152424)

FAQs

Frequently Asked Questions

What outcomes are you measuring in the study?
We’re primarily interested in how the Registered Revisions policy affects the peer review process. The main outcome is time from submission to final decision. We’re also tracking how many participating manuscripts reach a final decision, how often null results are reported, and how authors and editors experience the process. We ask about things like whether the policy was helpful, burdensome, or changed how participants think about transparency in peer review.


What’s in it for me (as an editor or journal)?
This is a chance to contribute to growing knowledge about how journal policies affect research transparency and publication outcomes. Editors and journals can publish their results - either independently or in collaboration with COS - and we provide support with study materials, data infrastructure, and even writing if helpful. It’s also a low-burden way to pilot a new editorial policy with a clear structure and defined scope.


My journal rarely asks for new data or analyses during review. Can we still participate?
Yes. We welcome journals with low volumes of eligible manuscripts. The study design allows for variation in how often the policy is triggered, and even small amounts of data help us assess feasibility and variation across journals. We’re also interested in whether this kind of embedded experimentation is workable across different publishing contexts, not just in high-output settings.


What qualifies a manuscript for inclusion in the study?
When a reviewer requests new data or analysis during peer review.


Do we need IRB approval?
COS provides IRB pathways and templates. Most sites rely on COS’s central IRB, but local processes can be accommodated.


How is randomization handled?
COS handles randomization through its infrastructure and provides clear workflows for editorial teams.


How long does it take to get started?
Most journals complete onboarding and launch within 4–6 weeks.


Do we need to implement the policy across all submissions?
No. The policy is only applied to eligible manuscripts within the trial, and it does not require permanent adoption.



Project materials and documentation on OSF → https://osf.io/tshqu/ 

This research is funded by the NSF (grant #2152424)

Get in Touch!

If you are a journal editor potentially interested in participating in any part of this project,  e-mail Noah Haber (noah@cos.io) and Macie Daley (macie@cos.io). You can also sign up for our e-mail list for updates below.