Publication pre-commitment devices such as Preregistration and Registered Reports may substantially reduce publication biases, prepublication biases (e.g. p-hacking and HARKING), and other questionable research practices. This study explores a related device, Registered Revisions.
Registered Revisions are a pre-commitment device that are like a miniature registered report that occurs during journal peer review. When reviewers ask for additional data and/or analysis, authors can propose and detail a protocol of how those data additional revisions will be performed. Reviewers and editors can then agree to In-Principle Accept (IPA) the publication on the basis of this protocol, regardless of what the results are.
In theory, this style of review may reduce the impact of questionable research practices, publication biases, reduce uncertainty about peer review, and decrease review timelines through preventing back and forth multiple rounds of review.
The Center for Open Science (COS) is leading a semi-centrally organized set of within-journal randomized experiments on Registered Revisions under one umbrella. COS provides design and support for journals and journal consortia to perform in-journal randomized experiments testing Registered Revisions.
Between journals and journal consortia
|
Within journals and journal consortia
|
COS is currently organizing a pilot study. If you are a journal editor or publisher interested in being a part of this experiment, please email Noah Haber (noah@cos.io) and Macie Daley (macie@cos.io).
Additional details, including detailed protocols and the data and code repository will be made available at our OSF page here: https://osf.io/tshqu/.
This research is funded by the NSF (grant #2152424)
Rather than one study with many journals, COS is fostering a many-studies approach, under the umbrella of a prospective living meta-analysis. This design helps foster:
This research is funded by the NSF (grant #2152424)
The expected outcomes are a combination of process outcomes (e.g. time to reaching final decisions. acceptance/rejection rates, etc), research outcomes (e.g. statistical significance, effect sizes, etc.), and satisfaction outcomes (e.g. how researchers and editors feel about the process).
This research is funded by the NSF (grant #2152424)
COS provides strong study design and implementation support, while the journal (or journal group) editorial team implements their own logistical procedures and take on the design.
On the JournalJournals have full ownership of their RCT. Each RCT is specific to the journals’ individual needs and preferences for:
Each individual study is expected to be its own publication, with journal partners being the main authors. |
COS ProvidesBoilerplate study design, including:
|
This research is funded by the NSF (grant #2152424)
Standard manuscript submissions that receive a revise-&-resubmit decision that requests for new data are eligible. Editorial team identifies eligible submissions before randomization between initial editorial decision, and senior/final decision.
If consent is obtained, author team is randomized to standard procedure or RR
If RR arm:
This research is funded by the NSF (grant #2152424)
COS is looking for partner journals to test the idea, and help inform design decision-making, and gain experience useful for all other journals.
These initial experiments will be more intensely involved with COS, and could even include embedding COS researchers in the journal editorial team as part of the peer review process.
Journals in the pilot process will be first to publish their findings, will support the longer term efforts, and will have strong influence on future designs and research.
Timeline: COS is aiming to begin initial pilot studies by Q4 2023, begin main journal RCTs by Q2 2024, and complete living review by Q2 2027.
This project is part of a larger umbrella examining the impact of Registered Reports. This includes randomized trials at the idea phase, data collection phase, and additional journal-level experiments.
This effort is a first of its kind in several ways, including:
We hope that this paves the way for future experimental study.
This research is funded by the NSF (grant #2152424)
The pilot for this project is a pathfinding project to prepare for the main phase. The ultimate goal of the pilot phase is to successfully implement multiple test versions of the main phase. In this pilot we will be:
By the end of the pilot, we will have built the full kit needed for journals to successfully run their own experiments and gained a large amount of experience for support. Participating journal editors will be coathors on at least one published manuscript, plus any additional projects that spin off from this main project.
The main deliverables of the pilot phase are:
We are planning on having 5-10 pilot journals involved, and will be updating this page with the journals shortly.
If you would like to join the pilot group, we would love to have you! Please e-mail Noah Haber (noah@cos.io) and Macie Daley (macie@cos.io).
210 Ridge McIntire Road
Suite 500
Charlottesville, VA 22903-5083
Email: contact@cos.io
Unless otherwise noted, this site is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.
Responsible stewards of your support
COS has consistently earned a Guidestar rating of Platinum for its financial transparency, the highest rating available. You can see our profile on Guidestar. COS and the OSF were also awarded SOC 2 accreditation in 2022 after an independent assessment of our security and procedures by the American Institute of CPA’s (AICPA).
We invite all of our sponsors, partners, and members of the community to learn more about how our organization operates, our impact, our financial performance, and our nonprofit status.