Table 1 A checklist for editors and peer reviewers for assessing validity of Systematic Reviews in the environmental sector

1. General methods

Yes/No

Have the authors pre-registered and/or published a protocol for the review? The protocol should be cited in the review report, be freely available online and contain methodological detail of all stages of the review process

The registration/publication of an a-priori protocol for the review process to avoid post-hoc decisions that might increase risk of bias is a fundamental requirement of a Systematic Review.

Does the review include a defined methods section providing a description of all the review stages conducted (e.g. Review question, Searching, Screening, Critical Appraisal, Data Extraction, Synthesis) in sufficient detail to enable the method to be replicated?

As with all scientific studies, the methods should be sufficiently reported so as to be replicable.

2. Searching for articles

Are all search terms and search strings, with Boolean operators (‘AND’, ‘OR’ etc.) and wildcards, clearly stated for each major source (e.g. databases, search engines) so that the exact search is replicable by a third party? Search terms for grey literature and other sources (e.g. specialist websites, search engines), if used, may be simplified.

Search strategies should be outlined in the protocol or review methods. An optimal search for literature should avoid assembling a biased or unrepresentative body of evidence and possess three key properties: comprehensive (maximises the number of potentially relevant studies found, aiming to identify all relevant studies), systematic (avoiding ad hoc search strategies reduces the susceptibility to bias resulting from e.g. no defined endpoint of search or ‘cherry picking’) and transparent (readers should be able to replicate and evaluate the search). Where possible, advice should be sought from an expert such as an information specialist/scientist.

3. Screening articles and including studies

Are study eligibility criteria precisely defined so as to be replicable (e.g. reliance on broad and potentially ambiguous terms should be avoided) and expressly related to each key element of the question? Other eligibility criteria (such as study design, publication language and date) may also be considered.

Clearly stated criteria for eligibility decisions minimise the potential for subjective decisions to influence which studies are included in the review, increase the transparency of the synthesis, and allow readers to assess the validity of the criteria to the review question. In addition to following the review question, eligibility criteria may define limits on the type of primary research to be considered in terms of (for example): geographic scope, type of data reported, type of intervention or impact, study design, date.

Comprehensive searches may generate a large number of records (e.g. articles) that vary widely in their relevance to the review scope. Authors must then determine whether or not each article is sufficiently relevant (eligible) for inclusion in the data synthesis stage. However, the choice of eligibility criteria can influence the conclusions of the synthesis, and the application of inadequately defined criteria can be subjective and lead to biases. Decisions over which studies are relevant for inclusion should therefore be based on clearly defined criteria, and should be replicable and transparent.

Is the number of records (e.g. articles) found during the searches as well as the number of records excluded at each stage of the screening process provided? For full transparency, numbers of included and included records maybe be provided via a flow diagram.

Listing all articles that were screened for eligibility and indicating whether each was included or excluded in data synthesis (usually as supplementary material), makes it clear whether potentially relevant studies have been omitted according to the eligibility criteria or were not captured by the search. Documenting the reasons for article exclusion at full text is essential for transparency and replicability.

4. Critical appraisal of primary study validity

Is an effort made to identify all sources of bias (threats to internal validity) relevant to each of the individual included studies? Ideally, authors should use a recognised critical appraisal tool.

Primary research can vary widely in methodological validity (internal validity) and study context (external validity). Internal validity can influence the findings of the research, and, if not properly accounted for, the subsequent findings of syntheses. External validity can influence the relevance/applicability of the study to users of the findings in individual contexts. Critical appraisal of internal validity (risk of bias) involves transparently evaluating the design and conduct of each included study based on methodologies, which can then help to objectively account for variation in study quality by placing greater emphasis on the most reliable studies. Online tools are available for such assessments and should be used and cited.

5. Data Extraction

Are all extracted data reported in a table or spreadsheet so that the synthesis can be replicated? This includes the data used in the synthesis from each primary study (e.g. ‘raw’ outcome metrics: means and variance measures) as well as meta-data (e.g. methodological details, population, intervention, outcomes and comparator descriptors, study context, etc.).

The volume and type of data collected by primary research articles varies substantially, even when similar questions are addressed. Authors of evidence syntheses must make decisions on which data to extract and on how to extract this information. These decisions may influence the findings of the synthesis, and so to minimise bias, the approach to data extraction should be clearly stated and, the extracted information should be comparable and consistent between studies.

Transparently identifying a consistent set of data to extract from each study, for example into a structured data extraction sheet, allows the process to be replicated and evaluated by a third party, and reduces the potential for bias over which data are extracted from individual studies. Typically, extracted information from each study included in the review comprises: study aims; intervention details, study design; population characteristics; comparator details and results (point estimates and measures of variance).

Providing a summary in which the population, intervention/exposure and outcome for each study are stated makes data extraction transparent, and makes it easier for readers to locate the most relevant primary literature and conduct supplementary analyses if required. Data may be provided in additional files or in an open access repository.

6. Data synthesis

Is the choice of synthesis method (i.e. quantitative meta-analysis, qualitative, or mixed-methods and narrative synthesis) described in sufficient detail to be replicable? Is the choice of the synthesis method justified on the basis of characteristics of included studies (i.e. for quantitative synthesis taking into consideration variability between studies in sample size, study design, context, outcomes, etc.)?

The approach to synthesising included studies varies substantially, and some approaches are more effective at ensuring objectivity and minimising potential bias than others. If appropriate, data should be pooled in a quantitative synthesis (e.g. meta-analysis, meta-regression). If substantial differences between populations, interventions, comparators or outcomes exist, meta-analysis (i.e. combining effect sizes across different studies) may not be appropriate. Since meta-analysis effectively treats all individual studies as part of one large study, meta-analysis is only appropriate when calculating an average effect is meaningful. If it is not appropriate to pool data across studies in meta-analysis, a reason for this should be given, and structured approach to some other quantitative or narrative synthesis taken, with efforts made to make sense of the whole of the data set, beyond describing results from individual studies in turn, noting differences in the weight of evidence behind statements made, and appropriate use of table and graphical presentations of results. Vote-counting (summing the studies which gave positive or negative findings) is not an appropriate synthesis method as an indication of impact or effectiveness.

7. Review limitations

Is an explicit section or identifiable passage of text devoted to the authors’ consideration of risk of bias in the synthesized evidence base due to limitations of the conduct of the review process as well of the primary research/data?

All reviews will have limitations, and it is important that authors are explicit about the known limitations of the primary data as well as the limitations in the methodology of the review process. Here we acknowledge the subjective nature of this criterion and the appraiser must use some subjective judgement to decide on the adequacy of any statement on limitations.