Making conservation science more reliable with preregistration and registered reports

Citations Over TimeTop 10% of 2019 papers

Abstract

Considerable evidence suggests that in many disciplines, large portions (up to 50% or more) of published conclusions may be unreliable (e.g., Open Science Collaboration 2015; Camerer et al. 2018). This unreliability emerges for a variety of reasons, but the most common risk factors include conducting studies with small samples; selectively reporting or publishing statistically significant outcomes or outcomes from best models; conducting undisclosed exploratory data analyses; claiming to have tested a priori hypotheses that were instead generated in response to the result in question (HARKing, i.e., hypothesizing after results are known); and preferentially testing or reporting support for surprising hypotheses (those with low prior probability). These are not equally problematic in all disciplines, but we have good reason to believe that at least the first 4 are common in ecology and related fields (Parker et al. 2016; Fraser et al. 2018). Growing recognition of problems with reliability of research findings has spawned the "credibility revolution" (Vazire 2018). This revolution has many forms within and across disciplines. In ecology and conservation, a major effort has focused on encouraging journals to develop stronger standards for thorough reporting of methods and results (e.g., https://osf.io/g65cb/; https://onlinelibrary.wiley.com/page/journal/15231739/homepage/forauthors.html#TOR). Another effort aligns well with earlier calls for journals to require archiving of data (where possible) and, more recently, code. However, so far there has been little attention devoted to less familiar tools that have the potential to greatly increase the reliability of research findings: preregistration (Nosek et al. 2018) and registered reports (Chambers 2013; Hardwicke & Ioannidis 2018). Our goal here is to explain both preregistration and registered reports and to dispel some common misconceptions, though we encourage readers to explore these topics in more depth elsewhere (Chambers 2013; Hardwicke & Ioannidis 2018; Nosek et al. 2018). Despite their similar names, preregistration and registered reports are distinct in important ways. In preregistration, a researcher develops a study and analysis plan and files it in a public archive. The preregistration serves as a record of the researcher's plans, but does not bind the researcher through any formal process. In contrast, a registered report begins with a researcher submitting a study and analysis plan to a journal. The journal then sends that plan out for review, and if, after any suggested revisions, the plan seems well justified and the methods viable, the editor can offer in-principle acceptance. This in-principle acceptance means that, when the study is later completed, if the planned methods were followed, the journal guarantees publication of the final paper regardless of results. At its most thorough, preregistration involves archiving a detailed description of the study design, including research questions (and where relevant, hypotheses or predictions), data-collection methods, target sample size, and a data-analysis plan before collecting any data. The data-analysis plan is arguably the core element of a useful preregistration, and preregistration can occur after data gathering as long as the analysis plan is developed and archived before the data are examined for patterns. Preregistrations are archived in a public registry (e.g., the Open Science Framework [OSF.io] or AsPredicted.org). Depending on the archive, the preregistration may be embargoed (kept private) to maintain confidentiality of a research plan until it is completed. Once a preregistration is filed, it cannot be edited, though it can be updated with further preregistrations. When authors preregister an analysis plan and then publish work based on that plan, they cite the preregistration so that reviewers and readers can evaluate the extent to which the plan was followed. This helps demonstrate that authors are not selectively reporting and thus are not presenting a biased set of results. This alone is sufficient justification to preregister, but if preregistration becomes common, it may also promote greater transparency in published research that has not been preregistered. This is because if preregistration becomes the norm, it will be easier to distinguish between planned (preregistered) analyses, for which the likelihood of selective reporting is low, and unplanned exploratory analyses. This could also lead to more transparency in unplanned exploratory analyses because in the absence of a preregistration, it would become more difficult to claim that an unplanned exploratory analysis was a planned hypothesis test. Thus, there would be less incentive for researchers to claim inaccurately that exploratory analyses were planned hypothesis tests. So, in the short term, preregistration allows researchers to signal that their research is less likely to be biased, and, in the long term, preregistration should promote transparency in both preregistered and exploratory analyses, thus reducing bias throughout the published literature. Like a preregistration, a registered report has a detailed study and analysis plan at its core, but unlike a preregistration, those plans are peer reviewed at a journal prior to study initiation (stage 1 registered report) and if, after peer review, the journal editor is satisfied with the justification for the study and the detailed plans, the editor will offer an in-principle acceptance. The in-principle acceptance is a guarantee of publication, regardless of results, given that the researchers follow the methods that were approved during the original review process. The editor and reviewers evaluate conformity to the original methods at stage 2 when the final manuscript is submitted to the journal. Registered reports provide many of the same benefits as preregistrations, but registered reports are an even stronger bulwark against the often-subtle biases that have come to permeate the literature. A major benefit of a registered report is that the decision to accept or reject, and possibly more important, the decision to submit or not, is independent of the outcome of the study. In fact, registered reports include nonsignificant findings at a dramatically higher rate than typical papers (Allen & Mehler 2018), which is excellent evidence that they reduce bias. The spread of this results-blind publication is thus already increasing reliability of the literature. Also, registered reports should be better studies on average because peer review happens prior to study initiation. Thus, review can actually improve the implementation of the study itself, unlike traditional poststudy peer review. Finally, because methodological problems and shortcomings can be fixed by review, rejections due to flawed methods are eliminated. In fact, the rate of acceptance of registered reports is around 70% (B. Nosek, personal communication). This high rate of acceptance is good for researchers, who can reduce the chance of wasting time with a series of submissions to different journals prior to securing publication. Registered reports clearly present a higher hurdle than preregistration. Although a researcher can complete a preregistration any time before examining the data, a registered report must be completed prior to committing to a study design. Thus, registered reports will not suit all studies. However, when a registered report is suitable, the researcher can gain all the benefits of preregistration and various other important benefits from pursuing a registered report. Registered reports, and especially preregistrations, are already in wide use in some disciplines. They are rapidly increasing in social psychology and some other social sciences, and preregistration is standard practice in clinical trials research and in medical meta-analyses and systematic reviews. However, some researchers may still have concerns about the suitability of preregistration and registered reports for conservation science. In our experience, such concerns usually take one of several forms. Thus, we present some common concerns below and follow each with an explanation of either why we believe the concerns are not obstacles to preregistration or registered reports or why they might be. Concern: What if I have to change my methods partway through the project? Preregistration response: Such changes are common in ecology and conservation research, but they do not preclude preregistration. First, one can reduce the chance of deviating from a plan if methods are refined in a pilot study before preregistration. However, even after a pilot study, changes in methods may be needed. In this case, the options are to wait until writing the manuscript to explain why data-gathering methods changed or to file a new preregistration that acknowledges (and links to) the earlier protocol and introduces the new methods. The old protocol does not disappear, but the evolution of the project is now transparent. Registered report response: Substantial changes in an already-accepted registered report will typically nullify an in-principle acceptance, so it is important to conduct a pilot study to refine methods prior to submitting a registered report. However, some small changes may be approved if they are cleared with editors as they arise. Concern: I work in a system with a limited field season and so I do not have the time to conduct a pilot study, then submit a registered report, wait for reviews, and then implement my study. My field season will be over. Registered report response: For those working in systems with constrained field seasons who cannot afford, in terms of time or money, to wait 1 year between conducting a pilot study and initiating the primary study, a registered report is probably not the right choice. That said, we can imagine systems of expedited registered report review (possibly where the justification and basic methods are reviewed prior to the field season, and an updated methods section is submitted for fast review once pilot data confirm viability of the methods), but we know of no journals currently offering this option. Concern: I work with existing data (e.g., from long-term projects, from existing citizen science projects, from my own metaphorical file drawer, for meta-analysis, etc.), so I cannot preregister or submit a registered report prior to data gathering. Preregistration response: Preregistration can be useful at any point before one starts to examine data for biologically relevant patterns. If the researcher has not examined the data, the primary benefits of preregistration can still be realized. Registered report response: Some journals allow submission of registered reports to work with existing data sets (e.g., Royal Society Open Science, https://royalsocietypublishing.org/rsos/registered-reports) but there may be situations where a registered report for analyses of existing data is not possible. Concern: I need to see the data before I can develop my analysis plan. Preregistration response: Options here are guided by the principle that analytic decisions should not be dictated by the biological patterns that seem evident in the data. However, in many circumstances, analytical decisions must depend on certain features of the data. Physicists have solved this problem with a technique called blind analysis, which involves scrambling all or parts of a data set or adding noise to obscure patterns. Once the analyses are finalized using the dummy data set, they are then applied to the actual data set. Another option is to simulate data to test possible analysis methods. In many circumstances, it would be possible to create a decision tree, presented in the preregistration before examining the data, that describes rules for choosing among analysis methods. Finally, a preregistration does not prevent changing analysis methods, it just makes these changes transparent. One can still publish with the new methods and simply describe the changes in the manuscript. Among other things, this transparency makes it more likely that when a plan does change, it changes for a good reason. Registered report response: As with a preregistration, it is possible to develop a set of decision rules that will allow submission of a registered report while retaining the option of adjusting data-analysis methods in response to the conditions of the data. Concern: What if I see patterns in my data that I want to follow up on with analyses that I did not preregister or include in my registered report? Preregistration response: Preregistration does not prevent unplanned exploratory analyses; it just makes these analyses more transparent. Simply distinguish the post hoc exploratory analyses from the preregistered analyses in the published paper. Ideally, one would report all exploratory work and declare that one has done so. If there are too many exploratory analyses to cover in the paper, present them in supplementary material or even in a data repository. Registered report response: Many journals, including Conservation Biology (under some circumstances), will allow submission of additional analyses in a separate, clearly labeled section of the Results that distinguishes the registered analyses from the nonregistered analyses. These other results could also be submitted as a separate manuscript. Concern: I focus on discovery. I do not typically have a priori hypotheses when I start a project. Preregistration response: Preregistration is useful even in the absence of a priori hypotheses. The primary purpose of preregistration is to promote transparency, and transparency is especially valuable for exploratory work because unreported exploration is a major source of bias. Exploratory work is vital, but we want to know that when patterns from exploratory work are reported, these are not a biased subset of the patterns observed by the researcher. A preregistration for exploratory work would describe the plan for data gathering and analysis. Even if data were gathered opportunistically, it would often still be possible to preregister an analysis plan prior to exploring the data. Registered reports response: If you can compose a compelling justification and a clear study and analysis plan, then you could submit a registered report. If not, then the study would be better presented in another format as exploratory work. Concern: If I preregister, I might be scooped. Response: You can embargo your preregistration so that it is private until you choose to share it. Preregistrations on the site AsPredicted.org can remain private indefinitely. On the OSF.io embargos are limited to 4 years. Concern: Preregistration and registered reports create extra work for researchers. Preregistration response: In most cases, preregistration should not dramatically change workload and may even reduce work. If one has written a grant or ethics proposal, much of the work of preregistration will have been done. Even if all relevant elements of the research have not been described in a proposal, the methods would need to be described for the resultant manuscript anyway, so preregistration just shifts the timing of this writing. To the extent that one ends up writing more about analyses in a preregistration than in a paper that reports only a subset of the analyses, this is the price for conducting transparent and reliable science. Even if one does not create a preregistration, journals (including Conservation Biology) are increasingly requesting more details of methods and results, although some of those details end up as online appendices. Further, careful preregistration may actually save work because it forces researchers to devote time to planning. In the absence of preregistration, it is often tempting to begin collecting or analyzing data without a fully formulated plan. This can mean that researchers waste time correcting errors later on, for instance resolving a mismatch between the data collected and the intended analyses. Registered report response: Besides the workload benefits outlined for preregistration, registered reports can also save researcher time if reviewers catch potential problems with methods and by making it more likely that the first journal chosen for submission will publish the manuscript. Concern: Registered reports are extra work for the reviewers and editors. Response: In the same way that both preregistration and registered reports shifts the timing of work for researchers without substantially changing the overall burden, registered reports shift the timing of work for reviewers and editors. Most of the reviewing work happens during stage 1, and those reviewers have the satisfaction of knowing that if they recognize a flaw in the methods, their concerns can actually solve the problem and not just condemn the study as flawed. Although reviewers will see the manuscript a second time (at stage 2, once results are in), there will be less work to do. Also, to the extent that registered reports reduce the frequency of manuscript rejection and thus resubmission to multiple journals, the overall reviewing burden will decline across the discipline. Concern: Why should I go through the trouble of submitting a registered report when I can just archive a preregistrations? Response: The individual researcher gets substantial benefits from choosing a registered report over a preregistration. The registered report provides peer review of the research methods prior to conducting the study, which is so much more useful than peer review of methods after the fact. Further, an in-principle acceptance means the researcher can conduct a study without fear that it might not be accepted, either because the methods will be criticized after the study is complete or because the wrong result will make publication difficult. Conservation scientists and ecologists can and should preregister their research and even conduct and publish registered reports. Both preregistration and registered reports send a strong signal to others that one is committed to transparent science and that the work is less likely to suffer from biased reporting. Registered reports have the added advantages of peer-reviewed methods prior to implementation and, most important, reducing publication bias. We predict the registered reports of early adopters are likely to garner more attention because these works will become exemplars for their respective area of study. Preregistration and registered reports are good for researchers, good for research, and good for the reliability of science as a whole. Some of the content and ideas in this essay first appeared as a blog post at Transparency in Ecology and Evolution on 26 September 2017 (http://www.ecoevotransparency.org/2017/09/26/ecologists-and-evolutionary-biologists-can-and-should-pre-register-their-research/). We thank F. Fidler for discussion on this topic and comments on the manuscript.

Related Papers

• → On the Credibility of Employment in Graduates(2010)
• Absence of Credibility for College Students(2007)
• The Analysis of Economics on Lack of Credibility(2006)
• A Probe into the Influence of Family Members on the Formation of Children's Credibility(2013)
• Credibility Education of College Students and Construction of Harmonious Soceity(2007)