Verifying authors’ claims to have conducted a Systematic Review? A checklist for journal editors and peer reviewers

Since 2018, the Collaboration for Environmental Evidence (CEE, environmentalevidence.org) has been assessing the conduct of globally published evidence reviews relevant to environmental management [1] and collating them into the publicly available database– CEEDER (https://environmentalevidence.org/ceeder/). CEEDER is a free service to individuals or organisations who want their decision making to be informed by the best available scientific evidence. Hundreds of environmental reviews are published annually in a broad range of journals and organisations. As part of its evidence service, CEE critically appraises each review for its reliability (risk of bias), rigour, comprehensiveness, and transparency of reporting.

From data collected for CEEDER it is apparent that the large majority (over 95%) of published environmental reviews that claim to be Systematic Reviews actually fall short of the standards of conduct and reporting expected of this methodology as described by global evidence synthesis organisations such as Evidence Synthesis International, Campbell Collaboration and Cochrane, as well as CEE [2]. This misnaming of a respected methodology risks undermining evidence synthesis and the value of properly conducted Systematic Reviews to inform decision making [3]. To address this problem, CEE now provides a Checklist for Editors and Peer Reviewers (see Table 1) that covers elements of conduct and reporting that are expected in a Systematic Review.

The checklist is based on the current CEE guidance for standards of conduct (https://environmentalevidence.org/information-for-authors/) and ROSES reporting standards [4]. The checklist is designed to enable a rapid assessment of the validity of authors’ claims to have conducted a Systematic Review which implies high procedural transparency and replicability, and comprehensive, reliable and rigorous findings with minimal bias.

The checklist is structured according to the stages of Systematic Review conduct and can be used for any review or synthesis in the environmental sector that claims to provide a Systematic Review of available evidence on a specific topic. Whilst the list of questions is not exhaustive and omits many important questions that might be the subject of full peer review, it provides key questions that might quickly identify manuscripts that do not qualify as Systematic Reviews. The checklist is offered as an additional tool for editors and peer reviewers and is not intended to replace any established editorial procedures or checks a journal may have in place. ‘Yes’ to all checklist questions is expected for a Systematic Review (see guidance notes below). Where a ‘No’ is selected, editors or peer reviewers may wish to ask authors for revision of their methodology or clarification to their reporting or ultimately ask authors to withdraw their claim. The checklist will be available from the CEE website (https://environmentalevidence.org/ceeder/support-for-authors-and-editors/) and our intention is to work with the editorial community to improve the checklist using feedback from users.

We hope this checklist brings clarity for editors and peer-reviewers, assists in standardizing expectations for Systematic Reviews and ensures that authors adhere to best practices and essential methodological steps. We anticipate this checklist will be a valuable tool for improving transparency, rigor, and trust in findings of published Systematic Reviews. Finally, by increasing awareness of Systematic Review methodology and necessary requirements for review conduct and reporting, this checklist aims to enhance the reliability of environmental evidence synthesis, ultimately providing environmental policy and management with the best available evidence.

Table 1 A checklist for editors and peer reviewers for assessing validity of Systematic Reviews in the environmental sector