TOP in Action: Modeling TOP Implementation for Funders

The Center for Open Science (COS) envisions research that is open across the lifecycle — linking the plans to research outputs to final outcomes. By making the basis of research claims transparently available, lifecycle open science enables research consumers to more readily assess the credibility of those claims. This openness enables the community to identify errors, discover new possibilities, and accelerate knowledge accumulation. The  Transparency and Openness Promotion Guidelines (TOP) framework is our primary policy tool to promote lifecycle open science, advancing a common set of guidelines to increase the verifiability of empirical research claims.

The TOP framework is adaptable for use by funders, institutions, journals, and other policymakers. In November, we announced that we will be implementing TOP’s revised framework into our own policies and initiatives. Today, we are showing how we implement TOP when acting as a funder in our Funding Consciousness Research with Registered Reports initiative.
 
Supported by the Templeton World Charity Foundation, this initiative promotes transparent and reproducible practices in consciousness research through the use of Registered Reports, one of the verification study types outlined in TOP. 

How the Project Works

Registered Reports are a publication format that involve a two-stage peer review process where studies are peer reviewed before results are known, emphasizing the importance of the research question and quality of methodology over exciting results. The Funding Consciousness Research with Registered Reports initiative aimed, in part, to raise awareness around Registered Reports among the consciousness research community. We used funding as an incentive to test, demonstrate, and adapt this publishing format, soliciting research proposals in collaboration with the Association for the Scientific Study of Consciousness. After receiving confirmation of eligibility for funding from COS, researchers submitted their project design to participating journals. Once a researcher received in-principle acceptance (IPA) to publish their final results, regardless of the main outcome of the study, they received at least half their requested budget. The remaining funds would then be distributed once the final report was published and linked, along with research outputs, to the plan.

Crafting a Policy that Matched our Funding Objectives

The funding policy we developed for the project: (a) required use of the Registered Reports format as a TOP verification study and (b) required TOP research practices that are embedded in the Registered Reports format. The Registered Reports format necessitates policies around the following TOP research practices: Study Registration, Study Protocol, and Analysis Plan. To receive the first half of their proposed budget, researchers had to submit a Stage 1 manuscript to a journal, which includes an introduction, methods, proposed analyses, and any results of pilot experiments. We implemented policies for these three practices at the most stringent level in the TOP Framework: certification. Researchers were required to submit these elements to a journal for peer review; the peer reviewers therefore certified that the registration, protocol, and analysis plan were complete and of sufficient quality to merit in-principle acceptance.

As a condition of grant funding, COS developed policies for data transparency, materials transparency, and analytic code transparency, requiring these materials be openly shared and cited to the extent possible in a repository. Certifying these practices were done according to disciplinary standards would have required further resourcing, either for COS to conduct these activities or for our partner journals to train their reviewers on certification practices. Training to essentially peer review data, materials, and code goes above and beyond common practice for most journals. We therefore chose to direct resources towards implementing more stringent policies related to the Registered Reports publishing format — which was the ultimate goal for this project — while still promoting TOP practices for sharing research outputs we value.

More detail on how we implemented different elements of the TOP Guidelines are shown in the table below.

Leveraging Infrastructure to Support Policy Implementation and Streamlined Compliance Monitoring

Good policies are only as good as their implementation. Compliance monitoring is essential for assuring that policies are translated into practice. Through engagements with research funders, we’ve learned that implementing and monitoring compliance with open policies can be daunting for the grant administrator. This burden can derail efforts to make science more transparent and verifiable. Here, well-designed research infrastructures can play an important role in enabling compliance through good research practices. The OSF Registry, which hosts all research plans that receive IPA, uses open science badges to indicate when outputs, such as data, code, and open papers, are available for readers to access. The registry links research plans to their associated outputs and outcomes, demonstrating lifecycle open science in action, and allows us, as a funder, to easily monitor compliance with the policies we’ve set.

In February, we’ll be hosting a webinar with research funders to share their experiences with open science policymaking, implementation, and compliance monitoring. Stay tuned for more details on how to register.