Aim

To describe how a proper systematic review should be undertaken.

Description

A systematic review consists of a single research question, a search strategy for the various databases, selection criteria, a methodological assessment of the quality of the included studies and a qualitative or quantitative analysis (meta-analysis). A well-conducted review will follow the steps described below in more detail:

1. Preparation: Make sure you have some background knowledge of systematic reviews;
2. Write a protocol;
3. Consult a search specialist;
4. Use at least 2 independent reviewers for the selection and methodological evaluation of the abstracts and articles;
5. Document the selection and evaluation of the abstracts and articles in a logbook;
6. Select a report format (Cochrane review or a journal article) and follow the guidelines described for a proper report;

See details.

A. Preparation

There are a number of sources you could consult prior to undertaking a review.
The book “Inleiding in evidence-based medicine. Klinisch handelen gebaseerd in bewijsmateriaal” (Introduction to evidence-based medicine: Clinical treatment based on evidence), by M. Offringa, WJJ Assendelft, and RJPM Scholten, contains a description of what you need to look for when carrying out a systematic review (e.g. how to find the literature, which search filters to use, evaluation lists, effect measures, etc.). Check the Prospero database for ongoing reviews.
The Handbook of the Cochrane Collaboration provides guidelines for writing a systematic review of (R)CT’s http://www.cochrane.org/resources/handbook/index.htm. It is anticipated that a handbook will also be available soon for writing a systematic review of diagnostic research: http://www.cochrane.org/docs/diagnostictestreviews.htm. Guidelines for writing a systematic review of measurement instruments can be found on the COSMIN website.  

There are also guidelines for reporting systematic reviews (Prisma statement). It is useful to read through these guidelines when preparing the review (don’t wait until you start the search or once the selection and evaluation have already been undertaken, etc.). These guidelines are available from www.consort-statement.org under “further initiatives” and contain the guidelines for reporting RCTs (CONSORT), systematic reviews (PRISMA), observational research (MOOSE) and diagnostic research (STARD).
Consider taking a course in writing up systematic reviews: www.epidm.nl/cursussen/.

You can contact people in the clinimetric group (Caroline Terwee or Riekie de Vet) for advice about undertaking a systematic review of measurement instruments (which compares the measurement properties of all measurement instruments measuring a certain concept).

B. Protocol

Write a protocol for your systematic review before you start your search. Examples of protocols for a Cochrane review of intervention studies or for diagnostic reviews can be found in the Cochrane Library. Examples of other types of reviews, such as diagnostic, prognostic, aetiology, measurement instruments etc. are not (as yet) included in the Cochrane Library. In these cases you could use a good, previously published review as an example, e.g.: 
Kuijpers T, van der Windt DA, van der Heijden GJ, Bouter LM. Systematic review of prognostic cohort studies on shoulder disorders. Pain. 2004 Jun;109(3):420-31.
Albers G, Echteld MA, de Vet HC, Onwuteaka-Philipsen BD, van der Linden MH, Deliens L. Evaluation of quality-of-life measures for use in palliative care: A systematic review. Palliat Med. 2009 Oct 20. [Epub ahead of print]

A systematic review will age rapidly; therefore you need to plan properly when you are going to start the literature search and when you will have time to evaluate the articles. It is therefore best if you have your search strategy, selection criteria, data extraction formula, and evaluation list ready prior to actually searching the databases.

The difference between a Cochrane review and a systematic review, published in a journal, is that Cochrane reviews need to be updated every 2 to 3 years. Other important differences are: 1) Cochrane reviews only explore the efficacy of therapy and diagnostic studies; 2) a protocol needs to be submitted and evaluated firstly for a Cochrane review, before you are allowed to undertake the review. Only the end product is evaluated for a journal article, and you can get to work immediately. In addition, the Cochrane Collaboration does not set any limits on the number of words, meaning that Cochrane reviews are often more detailed.

It is advised to register your review in the Prospero database of systematic reviews.

C. Searching and Search specialist

It is advisable to consult a search specialist when undertaking a systematic review. The university library employs search specialists, who will be able to give you good advice about searching the various databases. Contact René Otten. The library also organises a search course:Literature Searching For Authors Of Systematic Reviews. They will teach you all you need to know about literature search and those for systematic reviews in particular. From selection of databases and smart search techniques to documentation and accounting of your search strategy. All the ins and outs in just three hours. You can use your own review as test case during the course.

D. Section and evaluation of articles

The selection and methodological evaluation of the articles should be undertaken by at least 2 independent reviewers. It is possible to blind the reviewers regarding the source of the articles to be evaluated (authors, journal, year of publication, acknowledgement, any results). This blinding should be undertaken by an independent researcher not involved in the definitive selection and methodological evaluation of the articles. This requires a lot of work and isn’t always necessary!
The first selection of articles should be based on the title, abstract and keywords. The definitive election should be made on the basis of the full text of the article and is often combined with the methodological evaluation of the article.
There are various standardized checklists available for assessing the methodological quality of (randomised) controlled trials, including the List of Jadad, the Delphi list from AP Verhagen and the Amsterdam-Maastricht list. Various checklists are also available for evaluating diagnostic research (.e.g QUADAS) and other forms of observational studies. A number of these checklists are listed in the systematic reviews course (http://www.epidm.nl/cursussen/). A checklist is also available for evaluating the quality of studies on measurement properties (COSMIN, www.cosmin.nl) Disagreements need to be discussed between the reviewers, and attempts need to be made to arrive at a consensus about the methodological assessment of the article. If no consensus can be arrived at, then a third, independent reviewer should be consulted.

E. Documentation

Proper documentation of all decisions and choices made is essential for a systematic review. It is advisable to maintain a logbook (see the “Logbooks” guideline) similar to other studies, but particular attention needs to be paid to this for systematic reviews. A number of tips:

• Document the date you performed the search;
• Document the exact search strategy, particularly, for instance, when you are using a programme such as Ovid, in which you have to make a number of successive steps;
• Store all references retrieved in a database (e.g. Reference Manager and support). Make a selection of the references included at a later stage and store this in a second database. Provide a description of the contents of the databases. This will help you to easily retrieve previous references that have been evaluated when you are updating your search;
• Make a note of the reason for exclusion when excluding any references. You could, for instance, use a table for this in which you note down the exclusion criterion for each abstract or article per abstract (one reason per abstract is sufficient, as long as you continue to score the exclusion criteria in the same order). Note down all the comments from consensus meetings. This prevents you from forgetting at a later stage why a given article was or wasn't included;
• Also note down for each excluded reference whether this was excluded on the basis of the title, abstract or full-text article;
• Note down how many references in total have been retrieved, how many references you have excluded on the basis of the title, how many on the basis of the abstract and how many on the basis of full-text articles. Create a flow diagram for this (see example).
• Note down (e.g. in a table) the reasons for allocating a particular score when evaluating the methodological quality, particularly when there has been a discussion about this, or if you have had doubts about the score. This may appear to be a lot of work, but it saves a lot of time, since you won’t have to read or evaluate the article again due to forgetting why you made a given choice.

F. Reporting

A number of tools are available for summarising results. The researcher can make a decision on the basis of the available data whether and how the data should be used. The first option is a quantitative analysis (statistical pooling of results). However, statistical pooling is often not possible, or only available for part of the review, due to heterogeneity (too much variation) between studies, or because insufficient data are available. A qualitative analysis (sometimes referred to as best-evidence synthesis) can be conducted at any stage. In a qualitative analysis the results are summarised in a systematic way, taking into consideration various aspects such as methodological quality, consistency of results, statistical significance and clinical relevance of the effects uncovered. Finally, the conclusions can be formulated by using so-called levels of evidence (strength of the available evidence). Various methods are available for formulating levels of evidence (e.g. GRADE).

When performing a systematic review of measurement instruments, a separate step is to judge the quality of the included measurement instruments, i.e. to judge whether the measurement properties are ‘good enough’ to use the instrument for a specific purpose. A checklist has been developed for evaluating the measurement properties of measurement instruments (Terwee et al. J CLin Epidemiol 2007). This checklist can be found in the guideline Evaluation of measurement properties .

Links to Questionnaires guidelines, the Logbook guideline, an example of a flowchart, and articles by Ton Kuijpers and Sandra Bot (see text details)

Literature Searching For Authors Of Systematic Reviews

Systematic review: A comprehensive as possible overview of the available evidence based on the literature regarding a certain topic.

V1.3: 1 Dec 2011: Reference to the Prospero database
V1.2:  22 Feb 2010: Addition search course.
V1.1:  1 Jan 2010: Translation into English and updated.
V1.0:  29 Sep 2006.

1. Preparation: Have existing guidelines and other sources been consulted prior to the search?
2. Has a protocol been written?
3. Has a search specialist been consulted?
4. Has the selection of articles been independently carried out by at least two reviewers?
5. Has the methodological quality of the included studies been evaluated?
6. Has the methodological evaluation of articles been independently carried out by at least two reviewers?
7. In the event of blinding, has the blinding been carried out by an independent researcher?
8. Has an existing checklist been used for evaluating methodological quality?
9. Does the conclusion section of the review include a discussion of the methodological quality of the articles?
10. Documentation:

• Have the search criteria and selection criteria been clearly described?
  
• Has the full search (all search terms) been documented?
  
• Has the date of the search been noted?
  
• Has the selection of the articles been clearly and comprehensively described?
  
• Has a flow diagram been made regarding the inclusion of the articles?
  
• Has the evaluation of the methodological quality been clearly and comprehensively documented?