Literature Review and Evidence Synthesis

Due to the complex methodology of a systematic review and the extended timeline required ( >1 year), assigning this type of review for a semester long assignment may be frustrating for both the students and the faculty. (See A Step by Step Guide to a Systematic Review).

If the goal is for the students to develop life-long research skills and learn how to explore the scholarly conversation within the profession, many types of reviews, with less demanding methodologies, will help the students develop these skills successfully! 

Below are various types of assignments which incorporate essential elements of a systematic review in semester size chunks. Learning objectives are included for each of the examples. 

Please reach out to your subject librarian with any questions. 

Descriptions of the various types of reviews and the associated guidelines can be found in these articles: 

• Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health information and libraries journal, 26(2), 91–108. https://pubmed.ncbi.nlm.nih.gov/19490148/
• Sutton, A., Clowes, M., Preston, L. & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health information and libraries journal. 36(3).  https://pubmed.ncbi.nlm.nih.gov/31541534/

A narrative literature review is an integrated analysis of the existing literature used to summarize a body of literature, draw conclusions about a topic, and identify research gaps.  By understanding the current state of the literature, you can show how new research fits into the larger research landscape.  

The purposes of a narrative literature review:

• Explain the background of research on a topic
• Demonstrate the importance of a topic
• Suggest new areas of research
• Identify major themes, concepts, and researchers in a topic
• Identify critical gaps, points of disagreement, or flawed approaches for a research topic

More information about Narrative Reviews can be found here

Learning outcomes for a Narrative Literature Review would show that students have learned: 

• How to develop a research question
• How to search for literature to answer the research question
• Which databases are applicable to their topic of inquiry
• How to format a literature review
• How to synthesize sources of information
• How to cite literature sources

The complexity of this assignment can be modified dependent upon the course objectives. 

Peer review is a process all scholars will experience in their career whether it is in the form of receiving or providing this feedback. Many of the  exercises mentioned on this page would be appropriate for a peer review exercise.

A narrative literature review can be a good starting point for novice peer reviewers.

The protocol development exercise and the search strategy development exercises can be reviewed using the PRISMA-P and the PRISMA-S checklists.

The critical appraisal exercise can be reviewed using the CASP or JBI Critical Appraisal Checklists.

The data extraction exercise can be reviewed by comparing the results to a previously developed protocol.

For this exercise students would benefit from faculty developed examples of constructive review feedback along with an explanation of how the peer review process works, and why it is important in health science scholarship.

Learning objectives for the peer review exercise: 

• Students will learn the importance of the peer review process

• Students will learn how peer review supports the scholarly process

• Students will learn more about the conduct and reporting standards through the critique process

• Students will learn how to provide and learn from constructive feedback

One of the initial steps for completing an evidence synthesis project, such as a Scoping or Systematic Review, is to develop and register a protocol. The purpose of a protocol is to provide a detailed plan for the review project, reduce the risk of bias within the review and to establish providence of the research question to be evaluated. 

The PLoS Medicine Editors (2011) Best Practice in Systematic Reviews: The Importance of Protocols and Registration. PLoS Med 8(2): e1001009. https://doi.org/10.1371/journal.pmed.1001009

Protocols are developed according to the PRISMA-P protocol reporting guidelines. 

Assigning the development of a protocol will help the students to: 

• learn the basics of the PRISMA guidelines
• learn how to develop a research question
• learn how to develop and identify the inclusion and exclusion criteria
• learn how to identify which information sources are most appropriate to answer the research question
• learn how to develop a search strategy
• learn how to document and record the data management process
• learn how to define the data that will be collected and will answer the research question
• learn about the resources available to assess for the risk of bias

Detailed instructions for protocol development and additional protocol resources. 

Developing an equally sensitive and specific search strategy to ensure that the results are comprehensive for a systematic or scoping review is a complex task which often involves days of work for information professionals/academic librarians. This process involves learning which databases are applicable to the research topic, learning the controlled vocabulary of those databases and learning how to translate the search strategy from one database to the next to ensure a consistent search. Developing a systematic searching method is the basis of a systematic review. 

A systematic review is considered strong evidence when the methodology is reproducible. In this type of assignment, the students will also learn how to document their search strategy to ensure that it is transparent and reproducible. PRISMA-S provides reporting standards for systematic review search strategy documentation. The PRISMA-S may be used as a rubric for the assignment. 

Learning outcomes for the Search Method Exercise would show that students have learned: 

• which databases are applicable to their profession
• how to navigate an academic database and use the controlled vocabulary
• valuable expert searching skills
• the process of systematic searching
• how to document a transparent and reproducible search
• why transparency and reproducibility are an important element of strong evidence

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in particular context. It is an essential skill for evidence-based practice because it allows healthcare professionals to find and use research evidence reliably and efficiently" 

Burls, A. (2009). What is critical appraisal? In What Is This Series: Evidence-based medicine. Available online at What is Critical Appraisal?

Critical appraisal of the evidence to be synthesized is a vital step in the methodology of a systematic review. The process involves becoming familiar with the JBI or CASP checklists, learning the whys and hows of critical appraisal, and using critical thinking to determine the validity, reliability and relevance of a study in relation to the research question. 

Learning outcomes for a Critical Appraisal Exercise would show that students have learned:

• The purpose of the critical appraisal process
• Where to find critical appraisal checklists for specific types of studies
• The concepts of validity, reliability, transparency and relevance in relation to scientific study methodology
• What is required to ensure a study is valid, reliable and relevant
• How to use critical thinking to determine if a study would be included in the evidence synthesis of their review

This exercise could be modified to include few to many studies, multiple types of studies and could be an assignment on its own or part of a larger project. 

< id="docs-internal-guid-596bd17a-7fff-8c21-8dcb-280fbc9ba492">Data extraction is the first step in the synthesis process of a systematic review. This step requires that the review team adhere strictly to the PICO elements of the research question and extract data according to those elements. For this exercise, the students may work independently and then come together as a team to discuss their findings. Additional information regarding the data extraction process can be found in chapter 5 of the Cochrane Handbook for Systematic Reviews 29.

Learning objectives for the data extraction exercise: 

• Students will learn the data extraction process of a systematic review

• Students will learn the process of adhering to predefined criteria

• Students will gain experience with data collection, management, and documentation

• Students will gain experience working with a team

Qualitative synthesis of the data extracted is a required aspect of a systematic review. This process is completed regardless of a quantitative analysis. The qualitative synthesis is a narrative thematic analysis of the themes found within the data. A summarization of general characteristics of the data are analyzed along with relationships, patterns, heterogeneity and degrees of consistency. This narrative will also describe how the data answers the research question, determine the strengths and weaknesses of the data, identify evidence gaps and compare the findings with known scholarship. 

Learning objectives for the qualitative data synthesis exercise: 

• Students will learn how to analyze and synthesize qualitative data

• Students will apply critical thinking skills

• Students will develop higher order writing skills

• Students will learn the qualitative data synthesis process of a systematic review

Introduced by Grant & Booth (2009), the systematized review is an introductory review process for students. While a systematized review does not meet the strict criteria for conducting and reporting a systematic review, it can work as an introduction to the process. This type of review can also be completed by a single researcher or student whereas a systematic review requires a team of experts. 

"Attempt to include elements of systematic review process while stopping short of systematic review. Typically conducted as a postgraduate student assignment" Grant (2009).

Learning objectives for a systematized review will allow students to learn:

• An overview of the methodology for a systematic review according to the guidelines
• How to develop a research question
• How to determine inclusion and exclusion criteria
• How to determine which databases and resources are appropriate for their research question
• How to develop an systematic search strategy
• How to document a transparent and reproducible search strategy
• How to critically appraise scholarly evidence
• How to synthesize evidence
• How to report a systematic review according to the PRISMA guidelines

Prior to initiating a systematic review it is common practice to ensure that a review does not already exist related to your research question. In this search you may run across a systematic review that is outdated, this presents the opportunity to update the data. It is essential that prior to starting the update of a review, that the initial review is critically appraised to ensure that the methodology is correct and that all of the essential elements needed to update the review are available. 

"The update of a systematic review should be conducted according to the standards for any review, including the additional requirements of an update. These standards will ensure that any changes are managed appropriately and reported clearly to readers" 

Please refer to these resources for guidance to update systematic reviews:

• Cochrane Handbook Ch IV: Updating a Review
• Cochrane MECIR Manual: Reporting Standards Specific to Updates
• Cochrane Handbook Ch 22.2 Maintaining the currency of systematic reviews

The PRISMA flow diagram for SR updates is required in the reporting of the update. 

Student learning objectives for updating a systematic review: 

• The standard guidelines for conducting a systematic review
• How to modify an existing search strategy
• How to implement a comprehensive and systematic search strategy
• How to document/report a systematic review
• Why the conduct and reporting standards of a systematic review are essential to produce rigorous evidence
• How to complete a group project
• How to use citation managers

A rapid review provides a rapid synthesis of knowledge about a policy or clinical practice issue and attempts to inform an evidence-based decision quickly. It follows many of the stages of a systematic knowledge synthesis but may modify stages to shorten the timeline.

Read more:

• Garritty, C., Gartlehner, G., Nussbaumer-Streit, B., King, V. J., Hamel, C., Kamel, C., … & Stevens, A. (2020). Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2020.10.007
• Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 1, 10. http://doi.org/10.1186/2046-4053-1-10

Learning outcomes for Rapid or Scoping Review Exercises: 

Students will learn: 

• to develop an answerable research question
• to develop advanced search skills for various databases
• to develop an understanding of the methodology for rapid or scoping reviews
• how to determine the scholars of a specific topic within the profession
• how to synthesize the evidence acquired from the studies 

Source: Price, C. (n.d.) Is there better teaching opportunity than assigning systematic reviews in a semester? Covidence. https://www.covidence.org/blog/elementor-2112/