A Step-By-Step Guide to Conducting a Systematic Review and Meta-Analysis

Freelance biostatistician and public health consultant Kingsley Nanna Ukwaja provides a comprehensive step-by-step guide on how to conduct a meta analysis and systematic review.

What is a systematic review and meta-analysis?

A systematic review is defined by the Cochrane collaboration as “a scientific process where all empirical evidence that fits pre-specified eligibility criteria are collated in order to answer a specific research question”. It involves the systematic identification, selection, synthesis and appraisal of primary research studies in order to develop a high-quality overview of a topic while answering a pre-specified research question.

A meta-analysis is an advancement over systematic review and involves the use of mathematical and statistical approaches to summarise the findings of studies used for a systematic review.

Why do we need a systematic review and meta-analysis?

There are several reasons for performing a systematic review and meta-analysis:

• It can help to end confusion in conflicting findings reported by individual studies that might have specific bias or errors
• It can help highlight areas in a field where there is insufficient evidence and areas where further studies should focus on
• It enables the combination of findings from different studies which can highlight new findings that may be important for practice or policy
• It can help mitigate the need for further trials
• Writing a systematic review and meta-analysis may help define a researcher’s area of interest, as such reviews get published in high impact journal and gets a substantial number of citations

Steps to planning a systematic review and meta-analysis

The three components to a successful systematic review and meta-analysis writing are:

1. The research question
2. Good protocol
3. Research synthesis

1. The research question

It is important to define the review question clearly. The question should be precise and it should help identify studies to be included in the review. Well-focussed review questions are more likely to get completed and result in a comprehensive review because they lead to better searches and clearer criteria for selection. Also, a focused review question will more likely generate a clear message for the clinician/researcher and will more likely highlight the relevance of the work. In addition, a focused review question is more likely to identify questions for future research

However, before starting work on a research question for a systematic review, it is very important to perform a mini-literature search to ensure that the question has not already been the subject of a systematic review. This can be done through a simple google or google scholar search of the review topic/question. Also, search databases like the Cochrane library and PubMed as these databases are regularly updated. In addition, check PROSPERO, the primary database for registering systematic review protocols and search for published protocols to ensure that no individual or group is currently working on the planned systematic review. If the review has been previously done with an inconclusive finding and there have been additional primary studies since the review was published, consider performing an updated review. If a recent and high-quality systematic review and meta-analysis has been performed on the proposed review question, consider a different review question.

In general, developing a focussed review question is key. A review question takes the format: “To assess the effects of [an intervention or comparison] for [a health problem] in [types of people and disease or problem and setting, if possible]”’.

For example: “to assess the effect of [exercise interventions]on [weight loss]in [children]”.

It is often recommended that review questions follow the PICOS statement i.e.

• Participant, (Who is the Patient or Problem being addressed)
• Intervention, (What is the Intervention or Exposure)
• Comparator condition (What is the Comparison group)
• Outcomes (What is the Outcomes or Endpoint)
• Studies (What Study Designs are considered e.g., randomised controlled trials (RCTs) vs. non-RCTs)

For example, a systematic review with the following modified title: “In HIV‐positive children with no known TB contact on active TB, is 6 months of isoniazid effective in reducing the risk of active TB, as compared to placebo” (please, see for details)

The PICOS statement could be analysed thus:

• Participant/Problem, (children with HIV infection)
• Intervention, (isoniazid)
• Comparators (placebo)
• Outcomes (active TB)
• Studies: Consider RCTs

Given that we are interested in a therapeutic intervention here (isoniazid vs. placebo), it is better to consider RCTs for this review.

Generally, the studies selected for a systematic review are tailored to the domain of the review questions. These domains are usually addressed by different study designs. Domains of a review question could be: aetiology [cohort, case-control studies], therapeutic interventions [RCTs, cohort studies], diagnosis [cross-sectional, case-control studies], etc. An example of a systematic review question addressing risk factor (aetiology) could read: “Are people who smoke tobacco regularly at a greater risk of developing lung cancer as compared to those who do not smoke”

The PICOS statement for this title could be analysed thus:

• Participant/Problem, (people who smoke tobacco)
• Intervention/Exposure, (smoke regularly)
• Comparators (do not smoke)
• Outcomes (lung cancer)
• Studies: Consider cohort & case-control studies

In some cases, the review question may just attempt to summarize the prevalence or outcomes of a disease or event. Please, see: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0180996

The overriding key to a successful review is to have a clear and well-focused research question.

2.0 Protocol

Once you have a focused review question and you decide to proceed with the review, the next step is to develop a short review protocol. The protocol help gets you started and ensures you have understood the question and its contexts. It also helps you in developing the search strategy and in clearly highlighting what the inclusion and exclusion criteria should be. In addition, having a protocol and performing a dry run helps in the creation of data extraction form and to detect whether there may be a need to collect additional information on sub-groups. Table 1 below summarises the outline of a review protocol.

2.1 Background and objectives

This section briefly summarises the gaps in the field and why you want to perform the systematic review. The objectives of the review therefore could simply be derived from the review question.

Table 1: Outline of systematic review protocol

Background
Objectives
Review questions	Types of patients, interventions, outcomes and studies
Search strategy	Databases, study period, grey literature
Review Methods
–        Databases and article sources
–        Screening
–        Data extraction
–        Assessment of data quality
Data analysis
References

 

2.2 Search Strategy

In conducting the literature search, there is a need to define the sources/databases to be searched, the search process and how the studies found will be selected – these processes need to be documented. It is usually best to seek the help of a librarian during this stage. There is a need for detailed brainstorming to identify key search terms. The following approaches could be used to identify the search terms:

• Review key studies in the field and identify the keywords in the titles and abstracts
• Review the search strategies used in previous or similar systematic reviews
• Use database tools such as the Medical Subject Heading (MeSH) terms in Pubmed to identify controlled vocabulary, synonyms or keywords.

For example, in performing the SR on tobacco smoking and lung cancer in adults the possible keywords to consider in performing the search are:

• Smoking: “tobacco smoking”[MeSH Terms], “tobacco”[text word]”, “smoking”[text word]), and “smoking”[All Fields]
• Lung cancer: “lung neoplasms”[MeSH Terms], “lung”[All Fields]”, “neoplasms”[All Fields], “lung neoplasms”[All Fields], “lungtumor”[All Fields], “lung cancer”[All Fields]
• Adults: “adult”[MeSH Terms] OR “adult”[All Fields] OR “adults”[All Fields]

2.2.1 Databases and article sources

Once you have identified all the potential search terms, the next step will be to perform the search in suitable databases. These databases commonly store the title and abstracts of high-quality studies in the field. There are two broad types of databases:  general, and subject-specific databases. The general databases include: PubMed/Medline, Scopus, Web of Science , LILACS, and the search of these general databases could be supplemented by searching Google Scholar. The subject-specific databases include: CINAHL, PsycINFO, ERIC, CANCERLIT, TOXNET and AIDSLINE. The subject-specific database to be searched while performing the review depends on the review question.It should be noted that most of these databases were subscription-based. It is generally advised that a systematic review should search at least six databases. You can learn how to build a search in PubMed using the PubMed Tutorial).

Other potential sources of relevant articles beyond the databases include references of relevant articles in the field and citations/references from existing systematic reviews & meta-analyses. In addition, given that not all journals are indexed in databases and some of them may be incorrectly indexed, it is advised that you select one or more “high-yield” journals in your topic field and “handsearch” it for relevant articles.

2.2.2 Performing the search and screening

Once the databases to be searched and article sources has been identified, you can perform the search by combining the earlier identified search hits using Boolean operators [AND, OR, NOT]. Also, you can use study design filters & limits to focus the search (if necessary). For example, in the study on smoking and lung cancer, using these different Boolean operators will identify the coverage/breadth of articles summarised in Figure 1 below. You can practice searches with the title of the proposed review above. After performing the search, gather all the retrieved records of studies from each database into a reference manager, such as Endnote, Zotero or Mendeley, and remove all duplicates prior to screening (please, keep a record of the number of studies retrieved before and after de-duplication) as these are important in writing up the review.

The screening process begins with the title/abstract screening to identify and exclude studies that are clearly not useful for the review.Following the identification of all relevant articles, then move to full text screening. It is best to use your inclusion/exclusion criteria to screen the full-text of studies.Provide reasons for exclusion based on the participants not meeting the inclusion criteria using the PICOS statement as a guide. It is highly recommended that you use 2 independent reviewers at this step and any disagreements are discussed or referred to a third reviewer. Once you have identified all relevant studies for your review, progress to data extraction.

Figure 1: Boolean operators during search

2.2.3 Data extraction

It is also recommended that two of the investigators need to perform the data extraction. This is because data is not always found in consistent places in articles and a single data extractor may miss or misinterpret information.

Thee data to be extracted will relate to the protocol that you are following. For example, Cochrane reviews require as a minimum the following data should be retrieved from each article:

• Source data: e.g. citation and contact details
• Eligibility: Confirmed eligibility or reason for exclusion
• Methods: e.g. Study design, study duration, study methodology relating to bias
• Participants: e.g. number, setting, age, sex, diagnostic criteria, country
• Intervention: e.g. number of groups, specific intervention for each (giving enough detail for replication)
• Outcomes: e.g. measurement time points, outcome definition, units of measurements
• Results: e.g. number of participants in each intervention group, sample size, missing participants, summary data
• Miscellaneous: Funding source/s, authors conclusions, references to other relevant studies

It is best to have a standalone paper or electronic data extraction tool that you have developed. Some data extraction tool or templates could be obtained from Cochrane or Joanna Briggs Institute.

 2.2.4 Assessment of data quality

In the context of systematic reviews, the quality of evidence reflects the extent of confidence that an estimate of effect is correct. All studies included in a systematic review should be assessed for quality/risk of bias. Bias or quality issues in the study may be due to the study design and several other factors. The assessment of study quality involves the use of standardised critical appraisal tools. It is crucial to read the instructions carefully before applying the tool to ensure it is used correctly. Also, two reviewers usually complete this step and extracts the study quality data using the same process as data extraction. Record the agreement level between reviewers as this is likely to be required for most journals publishing systematic reviews. If there are several high-quality studies, you may decide to remove studies from the review with low quality/high risk of bias.

There are several tools that are available for the assessment of the quality of studies to be included in a review. The tool that could be used depends on the type/design of the studies studies identified for the review:

• For Cochrane reviews, the risk of bias tool
• Joanna Briggs Institute
• Newcastle-Ottawa Scale: cohort studies/case-control studies

3. Data Synthesis

The first part of the data synthesis is to describe the studies included in the review: this may include: the number of studies found, how the screening was performed and how the final studies were selected. These are commonly summarised using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart. You can obtain an editable template of the flow chart at (http://prisma-statement.org/PRISMAStatement/FlowDiagram.aspx). Other details of the studies included in the SR that could be reported here are: study country, number of participants, male: female ratio, duration of follow-up, study design and quality assessment of the included studies

Further analysis and synthesis of the data obtained may be narrative or quantitative. A narrative synthesis involves a structured description of the studies’ characteristics and findings as well its overall implications for the field. This is commonly done when the study design or approaches are too heterogeneous to be combined together.Overall, the tools for narrative synthesis of review include: Tables, Groupings and clusters, Vote counting as a descriptive tool and Examination of moderator variables (elements of e.g. setting, population).

A quantitative synthesis of the data involves statistical analysis (such as. meta-analysis) of the findings of multiple studies included in the review. The findings of the studies have to be “similar enough” to be combined into a single numerical result through meta-analysis (Please, you will need the help of a statistician/meta-analyst at this stage). The quantitative synthesis aims to answer the following questions:

• What is the direction of effect?
• What is the size of effect?
• Is the effect consistent across all the studies included?
• What is the strength of evidence for the effect?

In general, meta-analysis could be based on either the fixed-effect model or random effects model (Please, you will need the help of a statistician/meta-analyst at this stage). For reporting these quantitative synthesis tools such as forest plots are used. Overall, meta-analysis of the quantitative data could be performed using several software such as:

• RevMan,
• Comprehensive Meta-analysis Software,
• JBI tool, etc.

Undertaking a meta-analysis of the included studies may not always be feasible due to heterogeneity among these studies. The presence of heterogeneity can be evaluated either through a visual inspection of the overlapsin confidence intervals of effect sizes in the forest plot of the included studies or by undertaking a statistical test for heterogeneity using tests like the Cochrane Q (a type of Chis-Square – like) test.

In systematic reviews where meta-analysis is performed, there is a need to assess for the risk for publication bias. Publication bias is defined as the tendency to publish “only results that are statistically or clinically significant”. Given that this bias exists in the published literature, there is a need to assess how it may affect the findings of the systematic review. Publication bias in systematic reviews could be assessed either using graphical or mathematical methods (Please, you will need the help of a statistician/meta-analyst at this stage). The approaches commonly used include: funnel plot analysis, Egger’s regression test, Begg rank correlation test, Duval and Tweedle’s trim and fill technique, and “fail-safe N method” or “Rosenthal analysis”.

Write-up of the systematic review

The write up of the systematic review and meta-analysis requires the use of the PRISMA or theMeta-analysis of Observational Studies in Epidemiology [MOOSE] checklist. This is a checklist for reporting systematic literature reviews and meta-analyses and it outlines what details should be reported in each section of a high-quality systematic review. The checklist can be obtained from the PRISMA website (Please see: http://prisma-statement.org/prismastatement/Checklist.aspx) or the (MOOSE) guidelines (Please see: https://jamanetwork.com/journals/jama/article-abstract/192614).

The discussion of the findings of the systematic review should focus on the strength of evidence and limitations of the original studies used for the review. It is important to also discuss the limitations of the review, the applicability (generalizability) of results and the implications of the findings for patient care, public health and future research.

Once the write-up of the review is complete, there are several journal matching technologies based on your manuscript title and abstract that could be used to find potential journals for submission of the review. This include: Elsevier Journal Finder, Jane Biosemantics, JournalGuide , Springer Journal Suggester and Wiley Journal Recommendation.

Conclusion

Preparing a systematic review and meta-analysis is a worthwhile venture. Planning the review to publication is key to a successful outcome. A team approach is best and you will need the help of a librarian and statistician (meta-analyst).

Need help to conduct a systematic review and meta-analysis? Hire freelance systematic review experts and meta-analysis experts on Kolabtree. It’s free to post your project and get quotes.

References (Additional sources for Suggested reading)

• Higgins JPT, Green S (editors).Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
• Scott SD, et al. (2011) A protocol for a systematic review of knowledge translation strategies in the allied health professions. Implement Sci 658.
• Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-6
• Joanna Briggs Institute: http://joannabriggs.org/assets/docs/sumari/reviewersmanual-2014.pdf
• TeachEpi courses:  https://www.teachepi.org/courses/systematic-reviews-and-meta-analyses-in-tb/
• Rodgers M, Sowden A, Petticrew M, Arai L, Roberts H, Britten N et al. Testing methodological guidance on the conduct of narrative synthesis in systematic reviews: effectiveness of interventions to promote smoke alarm ownership and function. Evaluation 2009; 15(1):49-72.
• University of Maryland. Resources for conducting a systematic review research.https://lib.guides.umd.edu/SR/definition
• Harris JD, Quatman CE, Manring MM, Siston RA, Flanigan DC.How to write a systematic review.Am J Sports Med. 2014;42(11):2761-8