Preregistration is an unfamiliar practice in many areas of science. We will give 1,000 awards of $1,000 each to promote education and experience with preregistration. Award winners will have completed a preregistration using OSF and published the results in an eligible journal. All it takes is a single experiment and its analysis to be eligible.
Plan your study and analyses.
Design your study and how you will analyze the data. The Preregistration workflow on OSF will guide you through study design and analysis planning. You can save your progress and return later to continue. You may also use this Google doc template or this Word template and move the content to OSF later. See this guide on using OSF.
Once you and your collaborators complete the design and analysis plan, submit it for review to assess eligibility for the prize. We will review your plan to check for completeness and adherence to the eligibility criteria This is not peer review of your research. Once accepted, your research plan will be registered and eligible for the prize. Preregistrations must be pre-approved to be eligible.
This part is all up to you!
Report the results of all preregistered analyses. Any additional exploratory analysis is encouraged, but must be clearly differentiated from the confirmatory analyses in your report. Include a URL link to your preregistered project on OSF.
Submit your manuscript to an eligible journal (view the "Eligible Journals" tab on the left).
As soon as your article is published online, email email@example.com to confirm that the final article meets all eligibility requirements. If it does, the article is eligible for the next prize award date.
The 1,000 prizes will be awarded across 4 award dates. If more eligible articles are submitted than available awards during that award period, then the eligible articles will be ranked by Preregistration date with earlier registrations being awarded first. Non-awarded entries will remain in the eligible pool for the next award date:
There is a detailed description of the terms and conditions for the Preregistration Challenge to meet legal requirements and maximize the fair administration of the process. Here are highlights of key items:
As experience is gained with preregistration and the Challenge process, we expect that we will identify ways to improve the process and requirements. We will make updates as is needed and document those changes. Prior entrants will not lose eligibility based on future changes. The eligibility criteria and preregistration process is currently version 1.0.0. Check the change log here.
We welcome questions and feedback (firstname.lastname@example.org).
Entrants must publish their eligible preregistered study in a journal listed here to be eligible for a Preregistration Challenge award. Please note that the Preregistration Challenge is administered independently from any editorial, peer review, or publication process in these journals.
If a journal relevant to your research area does not appear on this list, you can assist in helping it become an eligible journal by encouraging the publisher or editor to take concrete steps to improve transparency and reproducibility such as signing the TOP Guidelines, making Registered Reports a submission option, and adopting badges to acknowledge open practices.
Each submission must pass a review process in which the statistical methods of the proposed study and its analyses are checked for completeness and adherence to Preregistration Challenge eligibility requirements. This review does not assess the substance of the research, or the validity of the research design or statistical methodology and has no impact on the independent editorial decisions of any journal.
Prereg Challenge administrators and reviewers will review the submitted study design and analysis descriptions, and determine whether all question fields were answered with enough detail to fully pre-specify the design and analysis plan, and follow the eligibility requirements. Reviews take no longer than 2 business days, and are often complete on the same business day.
Here are the guidelines that reviewers will use when evaluating your submitted plans.
Only studies that pass this review process are eligible for a $1000 prize. Studies that do not pass this review may still be registered on OSF independent of the Prereg Challenge.
Following independent peer review and acceptance for publication at a Prereg Challenge eligible journal, you will submit a link to the "in press" or published article online, and a copy of the journal-formatted published article for final review for eligibility for a Prereg Challenge award.
Prereg Challenge administrators and reviewers then review the article to ensure that it meets all of the eligibility criteria including: following the preregistered design, providing a link to the preregistration, making clear distinction between preregistered confirmatory analyses and other analyses, and following the preregistered analysis plan.
It is possible that deviations from the sampling, design, or analysis plans occurred in the course of conducting the research. Sometimes these deviations will render the preregistration ineligible for the Prereg Challenge award. In any case, deviations from preregistered plans must be documented explicitly in the article.
Here are some example scenarios about possible changes to the research plan and how to respond to them.
Questions? Please contact us as email@example.com or take advantage of our free statistical consulting services.
No. Confirmatory analyses are planned in advance, but they can be conditional. A pre-analysis plan might specify preconditions for certain analysis strategies and what alternative analysis will be performed if those conditions are not met. For example, if an analysis strategy requires data for a variable to be normally distributed, the analysis plan can specify evaluating normality and an alternate non-parametric test to be conducted if the normality assumption is violated.
For conditional analyses, we suggest that you define a 'decision-tree' containing logical IF-THEN rules that specify the analyses that will be used in specific situations. Here are some example decision trees. In the event that you need to conduct an unplanned analysis, preregistration does not prevent you from doing so. Preregistration simply makes clear which analyses were planned and which were not.
Yes. The central aims of preregistration are to distinguish confirmatory and exploratory analyses in order to retain the validity of their statistical inferences. Selective reporting of planned analyses is problematic for the latter.
Yes. Selective interpretation of pre-planned analyses can disrupt the diagnosticity of statistical inferences. For example, imagine that you planned 100 tests in your preregistration, and then reported all 100, 5 of which achieved p < .05. It is possible (even likely) that those five significant results are false positives. If the paper then discussed just those five and ignored the others, the interpretation could be highly misleading. Planning in advance is necessary but not sufficient for preserving diagnosticity.
To reduce interpretation biases, confirmatory research designs often have a small number of tests focused on the key questions in the research design, or adjustments for multiple-tests are included in the analysis plan. It may be that some preregistered analyses are dismissed as inappropriate or ill-conceived in retrospect, but doing that explicitly and transparently assists the reader in evaluating the rest of the confirmatory results.
No. Preregistration distinguishes confirmatory and exploratory analyses (Chambers et. al, 2014). Exploratory analysis is very important for discovery and hypothesis generation. Simultaneously, results from exploratory analyses are more tentative, p-values are less diagnostic, and additional data is required to subject an exploratory result to a confirmatory test. Making the distinction between exploratory and confirmatory analysis more transparent increases credibility of reports and helps the reader to fairly evaluate the evidence presented (Wagenmakers et al., 2012).
The purpose of preregistration is to make sure the distinction between these two processes are very clear. Once a researcher begins to slightly change the way to test the hypothesis, the work should be considered exploratory.
At least one confirmatory test must be specified in each preregistration.
Perhaps. A goal of pre-analysis plans is to avoid analysis decisions that are contingent on observed results (except when those contingencies are specified in advance, see above). This is more challenging for existing data, particularly when outcomes of the data have been observed or reported. Standards for effective preregistration using existing data do not yet exist. We are using the Preregistration Challenge to help develop such standards. As such, we have defined initial eligibility standards with pre-existing data and expect these to be refined over time.
When you submit your research plan, you will identify whether existing data is included in your planned analysis. For some circumstances, you will describe the steps that will ensure that the data or reported outcomes do not influence the analytical decisions. Below are the categories for which preregistration may still be eligible for the Preregistration Challenge:
There are several research circumstances that present challenges to conducting preregistered research.
If the present preregistration process does not fit your research approach effectively, and you believe that there are ways to conduct preregistered research in your field, we encourage you to contact us to help develop and specify a preregistration process for your work (firstname.lastname@example.org). The Preregistration Challenge is both an educational effort to encourage preregistration, and a research effort to develop effective preregistration processes that cover the wide variety of research approaches in science.
Registered Reports are a particular publication format in which the preregistered plan undergoes peer review in advance of observing the research outcomes. However, in the case of Registered Reports, that review is about the substance of the research and is overseen by journal editors. Research designs that pass peer review are offered ‘in principle acceptance’ (IPA) ensuring that the results are guaranteed to be published regardless of findings, as long as the methodology is carried out as described. Registered Reports are currently offered at 32 journals.
Participants in the Prereg Challenge are welcome and encouraged to submit their preregistered designs to the Registered Reports mechanism at their preferred journal. We recommend undergoing peer review at the journal first and preregistering for the Prereg Challenge after obtaining in-principle acceptance. That way, the registered Prereg Challenge will not need to be revised following substantive peer review at the journal.
Preregistration is relatively new to many people, so you may get questions from reviewers or editors during the review process. Below are some possible issues you may encounter and suggested strategies.
Possible editorial feedback: Reviewers or editors may request that you remove an experiment, study, analysis, variable, or design feature because the results are null results or marginal.
Prereg requirement: All preregistered analysis plans must be reported. Selective reporting undermines the diagnosticity of reported statistical inferences.
Possible response to the editor: The results of these tests are included because they stem from prespecified analyses in order to conduct a confirmatory test. Removing these results because of their non-significance would perpetuate publication bias already present in the literature (Chambers et al., 2014; Simmons et al., 2011; Wagenmakers et al., 2012).
Notes: If the reviewer/editor proposes a reason why they believe the null result could be explained by a design flaw, it can often be helpful/appropriate to leave the test in, but discuss the reviewers concerns about the validity of that particular test/design feature in a discussion section.
Possible editorial feedback: Why are you referring to a preregistered plan and reporting them separately from other analyses?
Prereg requirement: The published article must make clear which analyses were part of the confirmatory design (usually distinguished in the results section with confirmatory and exploratory results sections), and there must be a URL to the preregistration on OSF.
Possible response to the editor: The registration was certified prior to the start of data analysis. This defines analyses that were prespecified and confirmatory versus those which were not prespecified and therefore exploratory. Clarifying this allows readers to see that the hypotheses, analyses, and design that were prespecified have been accurately and fully reported (Jaeger & Halliday, 1998; Kerr, 1998, Thomas & Peterson, 2012).
Possible editorial feedback: Editor requests that you perform additional tests.
Prereg requirement: Additional tests are fine, they just need to be distinguished clearly from the confirmatory tests.
Possible response to the editor: Yes, these additional analyses are informative. We made sure to distinguish them from our preregistered analysis plan, which is the most resistant to the problem of alpha inflation. These analyses provide additional information for learning from our data.
When you have many planned studies being conducted from a single round of data collection, you have two possible ways to present and organize your preregistration. First, you could create a single preregistration including all of your planned studies and analyses. This could become difficult to organize. So the alternative is to create multiple preregistrations, one for each paper. If you decide to go this route, it is important to add clarity to each preregistration by mentioning that each is a single analysis plan in a wider project. Using a single OSF project when creating these plans, and citing the other preregistrations in each plan are two ways to ensure that they remain connected and discoverable. The goal is transparency, and a well organized set of studies will increase transparency.
Scientists at almost every career level are under exceptional pressure to publish. Also, evidence suggests that publishable results are often not easily reproduced (Begley & Ellis, 2012; Open Science Collaboration, 2015; Prinz et al., 2011). Therefore, we have created this incentive for researchers to try preregistration as a formalization of the idealized model of confirmatory hypothesis testing. An indicator of success will be measured by the number of participants who register analysis plans after participating in the Prereg Challenge because they have found it to improve their workflow and their confidence in their findings.
Preregistration has existed in a more limited form for clinical trials, but it is relatively new for basic and preclinical sciences. There are both strong theoretical reasons to preregister (Chambers et. al, 2014; Nosek & Lakens, 2014; Simmons, Nelson, & Simonsohn, 2011; Thomas & Peterson, 2012), and some empirical evidence that suggests that it does impact research outcomes (Kaplan and Irvin, 2015). We are conducting the Preregistration Challenge to increase experience and evidence about preregistration in the basic sciences. We will conduct and support research efforts to evaluate preregistration through the Prereg Challenge and registration on OSF more generally.
You may embargo your preregistration plan for up to 4 years following review to keep the details from public view. All registrations eventually become public because that is part of the purpose of a registry - to reduce the file-drawer effect (sometimes called the grey literature). It is possible to withdraw your preregistration, but a notification of the withdrawal will be public.
Yes! After you complete the research plan, you will have the option of registering it without submitting for review for the Prereg Challenge. Review is required to be eligible for a $1,000 prize, but the Preregistration workflow is available for general use.