which type of literature review is used to consolidate quantitative effect sizes

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

Types of Literature Review — A Guide for Researchers

Table of Contents

Researchers often face challenges when choosing the appropriate type of literature review for their study. Regardless of the type of research design and the topic of a research problem , they encounter numerous queries, including:

What is the right type of literature review my study demands?

• How do we gather the data?
• How to conduct one?
• How reliable are the review findings?
• How do we employ them in our research? And the list goes on.

If you’re also dealing with such a hefty questionnaire, this article is of help. Read through this piece of guide to get an exhaustive understanding of the different types of literature reviews and their step-by-step methodologies along with a dash of pros and cons discussed.

Heading from scratch!

What is a Literature Review?

A literature review provides a comprehensive overview of existing knowledge on a particular topic, which is quintessential to any research project. Researchers employ various literature reviews based on their research goals and methodologies. The review process involves assembling, critically evaluating, and synthesizing existing scientific publications relevant to the research question at hand. It serves multiple purposes, including identifying gaps in existing literature, providing theoretical background, and supporting the rationale for a research study.

What is the importance of a Literature review in research?

Literature review in research serves several key purposes, including:

• Background of the study: Provides proper context for the research. It helps researchers understand the historical development, theoretical perspectives, and key debates related to their research topic.
• Identification of research gaps: By reviewing existing literature, researchers can identify gaps or inconsistencies in knowledge, paving the way for new research questions and hypotheses relevant to their study.
• Theoretical framework development: Facilitates the development of theoretical frameworks by cultivating diverse perspectives and empirical findings. It helps researchers refine their conceptualizations and theoretical models.
• Methodological guidance: Offers methodological guidance by highlighting the documented research methods and techniques used in previous studies. It assists researchers in selecting appropriate research designs, data collection methods, and analytical tools.
• Quality assurance and upholding academic integrity: Conducting a thorough literature review demonstrates the rigor and scholarly integrity of the research. It ensures that researchers are aware of relevant studies and can accurately attribute ideas and findings to their original sources.

Types of Literature Review

Literature review plays a crucial role in guiding the research process , from providing the background of the study to research dissemination and contributing to the synthesis of the latest theoretical literature review findings in academia.

However, not all types of literature reviews are the same; they vary in terms of methodology, approach, and purpose. Let's have a look at the various types of literature reviews to gain a deeper understanding of their applications.

1. Narrative Literature Review

A narrative literature review, also known as a traditional literature review, involves analyzing and summarizing existing literature without adhering to a structured methodology. It typically provides a descriptive overview of key concepts, theories, and relevant findings of the research topic.

Unlike other types of literature reviews, narrative reviews reinforce a more traditional approach, emphasizing the interpretation and discussion of the research findings rather than strict adherence to methodological review criteria. It helps researchers explore diverse perspectives and insights based on the research topic and acts as preliminary work for further investigation.

Steps to Conduct a Narrative Literature Review

Source:- https://www.researchgate.net/figure/Steps-of-writing-a-narrative-review_fig1_354466408

Define the research question or topic:

The first step in conducting a narrative literature review is to clearly define the research question or topic of interest. Defining the scope and purpose of the review includes — What specific aspect of the topic do you want to explore? What are the main objectives of the research? Refine your research question based on the specific area you want to explore.

Conduct a thorough literature search

Once the research question is defined, you can conduct a comprehensive literature search. Explore and use relevant databases and search engines like SciSpace Discover to identify credible and pertinent, scholarly articles and publications.

Select relevant studies

Before choosing the right set of studies, it’s vital to determine inclusion (studies that should possess the required factors) and exclusion criteria for the literature and then carefully select papers. For example — Which studies or sources will be included based on relevance, quality, and publication date?

*Important (applies to all the reviews): Inclusion criteria are the factors a study must include (For example: Include only peer-reviewed articles published between 2022-2023, etc.). Exclusion criteria are the factors that wouldn’t be required for your search strategy (Example: exclude irrelevant papers, preprints, written in non-English, etc.)

Critically analyze the literature

Once the relevant studies are shortlisted, evaluate the methodology, findings, and limitations of each source and jot down key themes, patterns, and contradictions. You can use efficient AI tools to conduct a thorough literature review and analyze all the required information.

Synthesize and integrate the findings

Now, you can weave together the reviewed studies, underscoring significant findings such that new frameworks, contrasting viewpoints, and identifying knowledge gaps.

Discussion and conclusion

This is an important step before crafting a narrative review — summarize the main findings of the review and discuss their implications in the relevant field. For example — What are the practical implications for practitioners? What are the directions for future research for them?

Write a cohesive narrative review

Organize the review into coherent sections and structure your review logically, guiding the reader through the research landscape and offering valuable insights. Use clear and concise language to convey key points effectively.

Structure of Narrative Literature Review

A well-structured, narrative analysis or literature review typically includes the following components:

• Introduction: Provides an overview of the topic, objectives of the study, and rationale for the review.
• Background: Highlights relevant background information and establish the context for the review.
• Main Body: Indexes the literature into thematic sections or categories, discussing key findings, methodologies, and theoretical frameworks.
• Discussion: Analyze and synthesize the findings of the reviewed studies, stressing similarities, differences, and any gaps in the literature.
• Conclusion: Summarizes the main findings of the review, identifies implications for future research, and offers concluding remarks.

Pros and Cons of Narrative Literature Review

• Flexibility in methodology and doesn’t necessarily rely on structured methodologies
• Follows traditional approach and provides valuable and contextualized insights
• Suitable for exploring complex or interdisciplinary topics. For example — Climate change and human health, Cybersecurity and privacy in the digital age, and more
• Subjectivity in data selection and interpretation
• Potential for bias in the review process
• Lack of rigor compared to systematic reviews

Example of Well-Executed Narrative Literature Reviews

Paper title:  Examining Moral Injury in Clinical Practice: A Narrative Literature Review

Source: SciSpace

While narrative reviews offer flexibility, academic integrity remains paramount. So, ensure proper citation of all sources and maintain a transparent and factual approach throughout your critical narrative review, itself.

2. Systematic Review

A systematic literature review is one of the comprehensive types of literature review that follows a structured approach to assembling, analyzing, and synthesizing existing research relevant to a particular topic or question. It involves clearly defined criteria for exploring and choosing studies, as well as rigorous methods for evaluating the quality of relevant studies.

It plays a prominent role in evidence-based practice and decision-making across various domains, including healthcare, social sciences, education, health sciences, and more. By systematically investigating available literature, researchers can identify gaps in knowledge, evaluate the strength of evidence, and report future research directions.

Steps to Conduct Systematic Reviews

Source:- https://www.researchgate.net/figure/Steps-of-Systematic-Literature-Review_fig1_321422320

Here are the key steps involved in conducting a systematic literature review

Formulate a clear and focused research question

Clearly define the research question or objective of the review. It helps to centralize the literature search strategy and determine inclusion criteria for relevant studies.

Develop a thorough literature search strategy

Design a comprehensive search strategy to identify relevant studies. It involves scrutinizing scientific databases and all relevant articles in journals. Plus, seek suggestions from domain experts and review reference lists of relevant review articles.

Screening and selecting studies

Employ predefined inclusion and exclusion criteria to systematically screen the identified studies. This screening process also typically involves multiple reviewers independently assessing the eligibility of each study.

Data extraction

Extract key information from selected studies using standardized forms or protocols. It includes study characteristics, methods, results, and conclusions.

Critical appraisal

Evaluate the methodological quality and potential biases of included studies. Various tools (BMC medical research methodology) and criteria can be implemented for critical evaluation depending on the study design and research quetions .

Data synthesis

Analyze and synthesize review findings from individual studies to draw encompassing conclusions or identify overarching patterns and explore heterogeneity among studies.

Interpretation and conclusion

Interpret the findings about the research question, considering the strengths and limitations of the research evidence. Draw conclusions and implications for further research.

The final step — Report writing

Craft a detailed report of the systematic literature review adhering to the established guidelines of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). This ensures transparency and reproducibility of the review process.

By following these steps, a systematic literature review aims to provide a comprehensive and unbiased summary of existing evidence, help make informed decisions, and advance knowledge in the respective domain or field.

Structure of a systematic literature review

A well-structured systematic literature review typically consists of the following sections:

• Introduction: Provides background information on the research topic, outlines the review objectives, and enunciates the scope of the study.
• Methodology: Describes the literature search strategy, selection criteria, data extraction process, and other methods used for data synthesis, extraction, or other data analysis..
• Results: Presents the review findings, including a summary of the incorporated studies and their key findings.
• Discussion: Interprets the findings in light of the review objectives, discusses their implications, and identifies limitations or promising areas for future research.
• Conclusion: Summarizes the main review findings and provides suggestions based on the evidence presented in depth meta analysis.

*Important (applies to all the reviews): Remember, the specific structure of your literature review may vary depending on your topic, research question, and intended audience. However, adhering to a clear and logical hierarchy ensures your review effectively analyses and synthesizes knowledge and contributes valuable insights for readers.

Pros and Cons of Systematic Literature Review

• Adopts rigorous and transparent methodology
• Minimizes bias and enhances the reliability of the study
• Provides evidence-based insights
• Time and resource-intensive
• High dependency on the quality of available literature (literature research strategy should be accurate)
• Potential for publication bias

Example of Well-Executed Systematic Literature Review

Paper title: Systematic Reviews: Understanding the Best Evidence For Clinical Decision-making in Health Care: Pros and Cons.

Read this detailed article on how to use AI tools to conduct a systematic review for your research!

3. Scoping Literature Review

A scoping literature review is a methodological review type of literature review that adopts an iterative approach to systematically map the existing literature on a particular topic or research area. It involves identifying, selecting, and synthesizing relevant papers to provide an overview of the size and scope of available evidence. Scoping reviews are broader in scope and include a diverse range of study designs and methodologies especially focused on health services research.

The main purpose of a scoping literature review is to examine the extent, range, and nature of existing studies on a topic, thereby identifying gaps in research, inconsistencies, and areas for further investigation. Additionally, scoping reviews can help researchers identify suitable methodologies and formulate clinical recommendations. They also act as the frameworks for future systematic reviews or primary research studies.

Scoping reviews are primarily focused on —

• Emerging or evolving topics — where the research landscape is still growing or budding. Example — Whole Systems Approaches to Diet and Healthy Weight: A Scoping Review of Reviews .
• Broad and complex topics : With a vast amount of existing literature.
• Scenarios where a systematic review is not feasible: Due to limited resources or time constraints.

Steps to Conduct a Scoping Literature Review

While Scoping reviews are not as rigorous as systematic reviews, however, they still follow a structured approach. Here are the steps:

Identify the research question: Define the broad topic you want to explore.

Identify Relevant Studies: Conduct a comprehensive search of relevant literature using appropriate databases, keywords, and search strategies.

Select studies to be included in the review: Based on the inclusion and exclusion criteria, determine the appropriate studies to be included in the review.

Data extraction and charting : Extract relevant information from selected studies, such as year, author, main results, study characteristics, key findings, and methodological approaches.  However, it varies depending on the research question.

Collate, summarize, and report the results: Analyze and summarize the extracted data to identify key themes and trends. Then, present the findings of the scoping review in a clear and structured manner, following established guidelines and frameworks .

Structure of a Scoping Literature Review

A scoping literature review typically follows a structured format similar to a systematic review. It includes the following sections:

• Introduction: Introduce the research topic and objectives of the review, providing the historical context, and rationale for the study.
• Methods : Describe the methods used to conduct the review, including search strategies, study selection criteria, and data extraction procedures.
• Results: Present the findings of the review, including key themes, concepts, and patterns identified in the literature review.
• Discussion: Examine the implications of the findings, including strengths, limitations, and areas for further examination.
• Conclusion: Recapitulate the main findings of the review and their implications for future research, policy, or practice.

Pros and Cons of Scoping Literature Review

• Provides a comprehensive overview of existing literature
• Helps to identify gaps and areas for further research
• Suitable for exploring broad or complex research questions
• Doesn’t provide the depth of analysis offered by systematic reviews
• Subject to researcher bias in study selection and data extraction
• Requires careful consideration of literature search strategies and inclusion criteria to ensure comprehensiveness and validity.

In short, a scoping review helps map the literature on developing or emerging topics and identifying gaps. It might be considered as a step before conducting another type of review, such as a systematic review. Basically, acts as a precursor for other literature reviews.

Example of a Well-Executed Scoping Literature Review

Paper title: Health Chatbots in Africa Literature: A Scoping Review

Check out the key differences between Systematic and Scoping reviews — Evaluating literature review: systematic vs. scoping reviews

4. Integrative Literature Review

Integrative Literature Review (ILR) is a type of literature review that proposes a distinctive way to analyze and synthesize existing literature on a specific topic, providing a thorough understanding of research and identifying potential gaps for future research.

Unlike a systematic review, which emphasizes quantitative studies and follows strict inclusion criteria, an ILR embraces a more pliable approach. It works beyond simply summarizing findings — it critically analyzes, integrates, and interprets research from various methodologies (qualitative, quantitative, mixed methods) to provide a deeper understanding of the research landscape. ILRs provide a holistic and systematic overview of existing research, integrating findings from various methodologies. ILRs are ideal for exploring intricate research issues, examining manifold perspectives, and developing new research questions.

Steps to Conduct an Integrative Literature Review

• Identify the research question: Clearly define the research question or topic of interest as formulating a clear and focused research question is critical to leading the entire review process.
• Literature search strategy: Employ systematic search techniques to locate relevant literature across various databases and sources.
• Evaluate the quality of the included studies : Critically assess the methodology, rigor, and validity of each study by applying inclusion and exclusion criteria to filter and select studies aligned with the research objectives.
• Data Extraction: Extract relevant data from selected studies using a structured approach.
• Synthesize the findings : Thoroughly analyze the selected literature, identify key themes, and synthesize findings to derive noteworthy insights.
• Critical appraisal: Critically evaluate the quality and validity of qualitative research and included studies by using BMC medical research methodology.
• Interpret and present your findings: Discuss the purpose and implications of your analysis, spotlighting key insights and limitations. Organize and present the findings coherently and systematically.

Structure of an Integrative Literature Review

• Introduction : Provide an overview of the research topic and the purpose of the integrative review.
• Methods: Describe the opted literature search strategy, selection criteria, and data extraction process.
• Results: Present the synthesized findings, including key themes, patterns, and contradictions.
• Discussion: Interpret the findings about the research question, emphasizing implications for theory, practice, and prospective research.
• Conclusion: Summarize the main findings, limitations, and contributions of the integrative review.

Pros and Cons of Integrative Literature Review

• Informs evidence-based practice and policy to the relevant stakeholders of the research.
• Contributes to theory development and methodological advancement, especially in the healthcare arena.
• Integrates diverse perspectives and findings
• Time-consuming process due to the extensive literature search and synthesis
• Requires advanced analytical and critical thinking skills
• Potential for bias in study selection and interpretation
• The quality of included studies may vary, affecting the validity of the review

Example of Integrative Literature Reviews

Paper Title: An Integrative Literature Review: The Dual Impact of Technological Tools on Health and Technostress Among Older Workers

5. Rapid Literature Review

A Rapid Literature Review (RLR) is the fastest type of literature review which makes use of a streamlined approach for synthesizing literature summaries, offering a quicker and more focused alternative to traditional systematic reviews. Despite employing identical research methods, it often simplifies or omits specific steps to expedite the process. It allows researchers to gain valuable insights into current research trends and identify key findings within a shorter timeframe, often ranging from a few days to a few weeks — unlike traditional literature reviews, which may take months or even years to complete.

When to Consider a Rapid Literature Review?

• When time impediments demand a swift summary of existing research
• For emerging topics where the latest literature requires quick evaluation
• To report pilot studies or preliminary research before embarking on a comprehensive systematic review

Steps to Conduct a Rapid Literature Review

• Define the research question or topic of interest. A well-defined question guides the search process and helps researchers focus on relevant studies.
• Determine key databases and sources of relevant literature to ensure comprehensive coverage.
• Develop literature search strategies using appropriate keywords and filters to fetch a pool of potential scientific articles.
• Screen search results based on predefined inclusion and exclusion criteria.
• Extract and summarize relevant information from the above-preferred studies.
• Synthesize findings to identify key themes, patterns, or gaps in the literature.
• Prepare a concise report or a summary of the RLR findings.

Structure of a Rapid Literature Review

An effective structure of an RLR typically includes the following sections:

• Introduction: Briefly introduce the research topic and objectives of the RLR.
• Methodology: Describe the search strategy, inclusion and exclusion criteria, and data extraction process.
• Results: Present a summary of the findings, including key themes or patterns identified.
• Discussion: Interpret the findings, discuss implications, and highlight any limitations or areas for further research
• Conclusion: Summarize the key findings and their implications for practice or future research

Pros and Cons of Rapid Literature Review

• RLRs can be completed quickly, authorizing timely decision-making
• RLRs are a cost-effective approach since they require fewer resources compared to traditional literature reviews
• Offers great accessibility as RLRs provide prompt access to synthesized evidence for stakeholders
• RLRs are flexible as they can be easily adapted for various research contexts and objectives
• RLR reports are limited and restricted, not as in-depth as systematic reviews, and do not provide comprehensive coverage of the literature compared to traditional reviews.
• Susceptible to bias because of the expedited nature of RLRs. It would increase the chance of overlooking relevant studies or biases in the selection process.
• Due to time constraints, RLR findings might not be robust enough as compared to systematic reviews.

Example of a Well-Executed Rapid Literature Review

Paper Title: What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature

A Summary of Literature Review Types

Tools and resources for conducting different types of literature reviews, online scientific databases.

Platforms such as SciSpace , PubMed , Scopus , Elsevier , and Web of Science provide access to a vast array of scholarly literature, facilitating the search and data retrieval process.

Reference management software

Tools like SciSpace Citation Generator , EndNote, Zotero , and Mendeley assist researchers in organizing, annotating, and citing relevant literature, streamlining the review process altogether.

Automate Literature Review with AI tools

Automate the literature review process by using tools like SciSpace literature review which helps you compare and contrast multiple papers all on one screen in an easy-to-read matrix format. You can effortlessly analyze and interpret the review findings tailored to your study. It also supports the review in 75+ languages, making it more manageable even for non-English speakers.

Goes without saying — literature review plays a pivotal role in academic research to identify the current trends and provide insights to pave the way for future research endeavors. Different types of literature review has their own strengths and limitations, making them suitable for different research designs and contexts. Whether conducting a narrative review, systematic review, scoping review, integrative review, or rapid literature review, researchers must cautiously consider the objectives, resources, and the nature of the research topic.

If you’re currently working on a literature review and still adopting a manual and traditional approach, switch to the automated AI literature review workspace and transform your traditional literature review into a rapid one by extracting all the latest and relevant data for your research!

There you go!

Frequently Asked Questions

Narrative reviews give a general overview of a topic based on the author's knowledge. They may lack clear criteria and can be biased. On the other hand, systematic reviews aim to answer specific research questions by following strict methods. They're thorough but time-consuming.

A systematic review collects and analyzes existing research to provide an overview of a topic, while a meta-analysis statistically combines data from multiple studies to draw conclusions about the overall effect of an intervention or relationship between variables.

A systematic review thoroughly analyzes existing research on a specific topic using strict methods. In contrast, a scoping review offers a broader overview of the literature without evaluating individual studies in depth.

A systematic review thoroughly examines existing research using a rigorous process, while a rapid review provides a quicker summary of evidence, often by simplifying some of the systematic review steps to meet shorter timelines.

A systematic review carefully examines many studies on a single topic using specific guidelines. Conversely, an integrative review blends various types of research to provide a more comprehensive understanding of the topic.

You might also like

Introducing SciSpace’s Citation Booster To Increase Research Visibility

AI for Meta-Analysis — A Comprehensive Guide

How To Write An Argumentative Essay

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Supplements
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 4, Issue Suppl 1
• Synthesising quantitative evidence in systematic reviews of complex health interventions
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Julian P T Higgins 1 ,
• José A López-López 1 ,
• Betsy J Becker 2 ,
• Sarah R Davies 1 ,
• Sarah Dawson 1 ,
• Jeremy M Grimshaw 3 , 4 ,
• Luke A McGuinness 1 ,
• Theresa H M Moore 1 , 5 ,
• Eva A Rehfuess 6 ,
• James Thomas 7 ,
• Deborah M Caldwell 1
• 1 Population Health Sciences , Bristol Medical School, University of Bristol , Bristol , UK
• 2 Department of Educational Psychology and Learning Systems, College of Education , Florida State University , Tallahassee , Florida , USA
• 3 Clinical Epidemiology Program , Ottawa Hospital Research Institute, The Ottawa Hospital , Ottawa , Ontario , Canada
• 4 Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
• 5 NIHR Collaboration for Leadership in Applied Health Care (CLAHRC) West , University Hospitals Bristol NHS Foundation Trust , Bristol , UK
• 6 Institute for Medical Information Processing , Biometry and Epidemiology, Pettenkofer School of Public Health, LMU Munich , Munich , Germany
• 7 EPPI-Centre, Department of Social Science , University College London , London , UK
• Correspondence to Professor Julian P T Higgins; julian.higgins{at}bristol.ac.uk

Public health and health service interventions are typically complex: they are multifaceted, with impacts at multiple levels and on multiple stakeholders. Systematic reviews evaluating the effects of complex health interventions can be challenging to conduct. This paper is part of a special series of papers considering these challenges particularly in the context of WHO guideline development. We outline established and innovative methods for synthesising quantitative evidence within a systematic review of a complex intervention, including considerations of the complexity of the system into which the intervention is introduced. We describe methods in three broad areas: non-quantitative approaches, including tabulation, narrative and graphical approaches; standard meta-analysis methods, including meta-regression to investigate study-level moderators of effect; and advanced synthesis methods, in which models allow exploration of intervention components, investigation of both moderators and mediators, examination of mechanisms, and exploration of complexities of the system. We offer guidance on the choice of approach that might be taken by people collating evidence in support of guideline development, and emphasise that the appropriate methods will depend on the purpose of the synthesis, the similarity of the studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team and the resources available.

• meta-analysis
• complex interventions
• systematic reviews
• guideline development

Data availability statement

No additional data are available.

This is an open access article distributed under the terms of the Creative Commons Attribution IGO License ( CC BY NC 3.0 IGO ), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.Disclaimer: The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.

https://doi.org/10.1136/bmjgh-2018-000858

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Summary box

Quantitative syntheses of studies on the effects of complex health interventions face high diversity across studies and limitations in the data available.

Statistical and non-statistical approaches are available for tackling intervention complexity in a synthesis of quantitative data in the context of a systematic review.

Appropriate methods will depend on the purpose of the synthesis, the number and similarity of studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team and the resources available.

We offer considerations for selecting methods for synthesis of quantitative data to address important types of questions about the effects of complex interventions.

Public health and health service interventions are typically complex. They are usually multifaceted, with impacts at multiple levels and on multiple stakeholders. Also, the systems within which they are implemented may change and adapt to enhance or dampen their impact. 1 Quantitative syntheses ('meta-analyses’) of studies of complex interventions seek to integrate quantitative findings across multiple studies to achieve a coherent message greater than the sum of their parts. Interest is growing on how the standard systematic review and meta-analysis toolkit can be enhanced to address complexity of interventions and their impact. 2 A recent report from the Agency for Healthcare Research and Quality and a series of papers in the Journal of Clinical Epidemiology provide useful background on some of the challenges. 3–6

This paper is part of a series to explore the implications of complexity for systematic reviews and guideline development, commissioned by WHO. 7 Clearly, and as covered by other papers in this series, guideline development encompasses the consideration of many different aspects, 8 such as intervention effectiveness, economic considerations, acceptability 9 or certainty of evidence, 10 and requires the integration of different types of quantitative as well as qualitative evidence. 11 12 This paper is specifically concerned with methods available for the synthesis of quantitative results in the context of a systematic review on the effects of a complex intervention. We aim to point those collating evidence in support of guideline development to methodological approaches that will help them integrate the quantitative evidence they identify. A summary of how these methods link to many of the types of complexity encountered is provided in table 1 , based on the examples provided in a table from an earlier paper in the series. 1 An annotated list of the methods we cover is provided in table 2 .

• View inline

Quantitative synthesis possibilities to address aspects of complexity

Quantitative graphical and synthesis approaches mentioned in the paper, with their main strengths and weaknesses in the context of complex interventions

We begin by reiterating the importance of starting with meaningful research questions and an awareness of the purpose of the synthesis and any relevant background knowledge. An important issue in systematic reviews of complex interventions is that data available for synthesis are often extremely limited, due to small numbers of relevant studies and limitations in how these studies are conducted and their results are reported. Furthermore, it is uncommon for two studies to evaluate exactly the same intervention, in part because of the interventions’ inherent complexity. Thus, each study may be designed to provide information on a unique context or a novel intervention approach. Outcomes may be measured in different ways and at different time points. We therefore discuss possible approaches when data are highly limited or highly heterogeneous, including the use of graphical approaches to present very basic summary results. We then discuss statistical approaches for combining results and for understanding the implications of various kinds of complexity.

In several places we draw on an example of a review undertaken to inform a recent WHO guideline on protecting, promoting and supporting breast feeding. 13 The review seeks to determine the effects of interventions to promote breast feeding delivered in five types of settings (health services, home, community, workplace, policy context or a combination of settings). 8 The included interventions were predominantly multicomponent, and were implemented in complex systems across multiple contexts. The review included 195 studies, including many from low-income and middle-income countries, and concluded that interventions should be delivered in a combination of settings to achieve high breastfeeding rates.

The importance of the research question

The starting point in any synthesis of quantitative evidence is a clear purpose. The input of stakeholders is critical to ensure that questions are framed appropriately, addressing issues important to those commissioning, delivering and affected by the intervention. Detailed discussion of the development of research questions is provided in an earlier paper in the series, 1 and a subsequent paper explains the importance of taking context into account. 9 The first of these papers describes two possible perspectives. A complex interventions perspective emphasises the complexities involved in conceptualising, specifying and implementing the intervention per se, including the array of possibly interacting components and the behaviours required to implement it. A complex systems perspective emphasises the complexity of the systems into which the intervention is introduced, including possible interactions between the intervention and the system, interactions between individuals within the system and how the whole system responds to the intervention.

The simplest purpose of a systematic review is to determine whether a particular type of complex intervention (or class of interventions) is effective compared with a ‘usual practice’ alternative. The familiar PICO framework is helpful for framing the review: 14 in the PICO framework, a broad research question about effectiveness is uniquely specified by describing the participants (‘P’, including the setting and prevailing conditions) to which the intervention is to be applied; the intervention (‘I’) and comparator (‘C’) of interest, and the outcomes (‘O’, including their time course) that might be impacted by the intervention. In the breastfeeding review, the primary synthesis approach was to combine all available studies, irrespective of setting, and perform separate meta-analyses for different outcomes. 15

More useful than a review that asks ‘does a complex intervention work?’ is one that determines the situations in which a complex intervention has a larger or smaller effect. Indeed, research questions targeted by syntheses in the presence of complexity often dissect one or more of the PICO elements to explore how intervention effects vary both within and across studies (ie, treating the PICO elements as ‘moderators’). For instance, analyses may explore variation across participants, settings and prevailing conditions (including context); or across interventions (including different intervention components that may be present or absent in different studies); or across outcomes (including different outcome measures, at different levels of the system and at different time points) on which effects of the intervention occur. In addition, there may be interest in how aspects of the underlying system or the intervention itself mediate the effects, or in the role of intermediate outcomes on the pathway from intervention to impact. 16 In the breastfeeding review, interest moved from the overall effects across interventions to investigations of how effects varied by such factors as intervention delivery setting, high-income versus low-income country, and urban versus rural setting. 15

The role of logic models to inform a synthesis

An earlier paper describes the benefits of using system-based logic models to characterise a priori theories about how the system operates. 1 These provide a useful starting point for most syntheses since they encourage consideration of all aspects of complexity in relation to the intervention or the system (or both). They can help identify important mediators and moderators, and inform decisions about what aspects of the intervention and system need to be addressed in the synthesis. As an example, a protocol for a review of the health effects of environmental interventions to reduce the consumption of sugar-sweetened beverages included a system-based logic model, detailing how the characteristics of the beverages, and the physiological characteristics and psychological characteristics of individuals, are thought to impact on outcomes such as weight gain and cardiovascular disease. 17 The logic model informs the selection of outcomes and the general plans for synthesis of the findings of included studies. However, system-based models do not usually include details of how implementation of an intervention into the system is likely to affect subsequent outcomes. They therefore have a limited role in informing syntheses that seek to explain mechanisms of action.

A quantitative synthesis may draw on a specific proposed framework for how an intervention might work; these are sometimes referred to as process-orientated logic models, and may be strongly driven by qualitative research evidence. 12 They represent causal processes, describing what components or aspects of an intervention are thought to impact on what behaviours and actions, and what the further consequences of these impacts are likely to be. 18 They may encompass mediators of effect and moderators of effect. A synthesis may simply adopt the proposed causal model at face value and attempt to quantify the relationships described therein. Where more than one possible causal model is available, a synthesis may explore which of the models is better supported by the data, for example, by examining the evidence for specific links within the model or by identifying a statistical model that corresponds to the overall causal model. 18 19

A systematic review on community-level interventions for improving access to food in low-income and middle-income countries was based on a logic model that depicts how interventions might lead to improved health status. 20 The model includes direct effects, such as increased financial resources of individuals and decreased food prices; intermediate effects, such as increased quantity of food available and increase in intake; and main outcomes of interest, such as nutritional status and health indicators. The planned statistical synthesis, however, was to tackle these one at a time.

Considering the types of studies available

Studies of the effects of complex interventions may be randomised or non-randomised, and often involve clustering of participants within social or organisational units. Randomised trials, if sufficiently large, provide the most convincing evidence about the effects of interventions because randomisation should result in intervention and comparator groups with similar distributions of both observed and unobserved baseline characteristics. However, randomised trials of complex interventions may be difficult or impossible to undertake, or may be performed only in specific contexts, yielding results that are not generalisable. Non-randomised study designs include so-called ‘quasi-experiments’ and may be longitudinal studies, including interrupted time series and before-after studies, with or without a control group. Non-randomised studies are at greater risk of bias, sometimes substantially so, although may be undertaken in contexts that are more relevant to decision making. Analyses of non-randomised studies often use statistical controls for confounders to account for differences between intervention groups, and challenges are introduced when different sets of confounders are used in different studies. 21 22

Randomised trials and non-randomised studies might both be included in a review, and analysts may have to decide whether to combine these in one synthesis, and whether to combine results from different types of non-randomised studies in a single analysis. Studies may differ in two ways: by answering different questions, or by answering similar questions with different risks of bias. The research questions must be sufficiently similar and the studies sufficiently free of bias for a synthesis to be meaningful. In the breastfeeding review, randomised, quasi-experimental and observational studies were combined; no evidence suggested that the effects differed across designs. 15 In practice, many methodologists generally recommend against combining randomised with non-randomised studies. 23

Preparing for a quantitative synthesis

Before undertaking a quantitative synthesis of complex interventions, it can be helpful to begin the synthesis non-quantitatively, looking at patterns and characteristics of the data identified. Systematic tabulation of information is recommended, and this might be informed by a prespecified logic model. The most established framework for non-quantitative synthesis is that proposed by Popay et al . 24 The Cochrane Consumers and Communication group succinctly summarise the process as an 'investigation of the similarities and the differences between the findings of different studies, as well as exploration of patterns in the data’. 25 Another useful framework was described by Petticrew and Roberts. 26 They identify three stages in the initial narrative synthesis: (1) Organisation of studies into logical categories, the structure of which will depend on the purpose of the synthesis, possibly relating to study design, outcome or intervention types. (2) Within-study analysis, involving the description of findings within each study. (3) Cross-study synthesis, in which variations in study characteristics and potential biases are integrated and the range of effects described. Aspects of this process are likely to be implemented in any systematic review, even when a detailed quantitative synthesis is undertaken.

In some circumstances the available data are too diverse, too non-quantitative or too sparse for a quantitative synthesis to be meaningful even if it is possible. The best that can be achieved in many reviews of complex interventions is a non-quantitative synthesis following the guidance given in the above frameworks.

Options when effect size estimates cannot be obtained or studies are too diverse to combine

Graphical approaches.

Graphical displays can be very valuable to illustrate patterns in results of studies. 27 We illustrate some options in figure 1 . Forest plots are the standard illustration of the results of multiple studies (see figure 1 , panel A), but require a similar effect size estimate from each study. For studies of complex interventions, the diversity of approaches to the intervention, the context, 1 evaluation approaches and reporting differences can lead to considerable variation across studies in what results are available. Some novel graphical approaches have been proposed for such situations. A recent development is the albatross plot, which plots p values against sample sizes, with approximate effect-size contours superimposed (see figure 1 , panel B). 28 The contours are computed from the p values and sample sizes, based on an assumption about the type of analysis that would have given rise to the p values. Although these plots are designed for situations when effect size estimates are not available, the contours can be used to infer approximate effect sizes from studies that are analysed and reported in highly diverse ways. Such an advantage may prove to be a disadvantage, however, if the contours are overinterpreted.

• Download figure
• Open in new tab
• Download powerpoint

Example graphical displays of data from a review of interventions to promote breast feeding, for the outcome of continued breast feeding up to 23 months. 15 Panel A: Forest plot for relative risk (RR) estimates from each study. Panel B: Albatross plot of p value against sample size (effect contours drawn for risk ratios assuming a baseline risk of 0.15; sample sizes and baseline risks extracted from the original papers by the current authors); Panel C: Harvest plot (heights reflect design: randomised trials (tall), quasi-experimental studies (medium), observational studies (short); bar shading reflects follow-up: longest follow-up (black) to shortest follow-up (light grey) or no information (white)). Panel D: Bubble plot (bubble sizes and colours reflect design: randomised trials (large, green), quasi-experimental studies (medium, red), observational studies (small, blue); precision defined as inverse of the SE of each effect estimate (derived from the CIs); categories are: “Potential Harm”: RR <0.8; “No Effect”: RRs between 0.8 and 1.25; “Potential Benefit”: RR >1.25 and CI includes RR=1; “Benefit”: RR >1.25 and CI excludes RR=1).

Harvest plots have been proposed by Ogilvie et al as a graphical extension of a vote counting approach to synthesis (see figure 1 , panel C). 29 However, approaches based on vote counting of statistically significant results have been criticised on the basis of their poor statistical properties, and because statistical significance is an outdated and unhelpful notion. 30 The harvest plot is a matrix of small illustrations, with different outcome domains defining rows and different qualitative conclusions (negative effect, no effect, positive effect) defining columns. Each study is represented by a bar that is positioned according to its measured outcome and qualitative conclusion. Bar heights and shadings can depict features of the study, such as objectivity of the outcome measure, suitability of the study design and study quality. 29 31 A similar idea to the harvest plot is the effect direction plot proposed by Thomson and Thomas. 32

A device to plot the findings from a large and complex collection of evidence is a bubble plot (see figure 1 , panel D). A bubble plot illustrates the direction of each finding (or whether the finding was unclear) on a horizontal scale, using a vertical scale to indicate the volume of evidence, and with bubble sizes to indicate some measure of credibility of each finding. Such an approach can also depict findings of collections of studies rather than individual studies, and was used successfully, for example, to summarise findings from a review of systematic reviews of the effects of acupuncture on various indications for pain. 33

Statistical methods not based on effect size estimates

We have mentioned that a frequent problem is that standard meta-analysis methods cannot be used because data are not available in a similar format from every study. In general, the core principles of meta-analysis can be applied even in this situation, as is highlighted in the Cochrane Handbook , by addressing the questions: ‘What is the direction of effect?’; 'What is the size of effect?’; ‘Is the effect consistent across studies?’; and 'What is the strength of evidence for the effect?’. 34

Alternatives to the estimation of effect sizes could be used more often than they are in practice, allowing some basic statistical inferences despite diversely reported results. The most fundamental analysis is to test the overall null hypothesis of no effect in any of the studies. Such a test can be undertaken using only minimally reported information from each study. At its simplest, a binomial test can be performed using only the direction of effect observed in each study, irrespective of its CI or statistical significance. 35 Where exact p values are available as well as the direction of effect, a more powerful test can be performed by combining these using, for example, Fisher’s combination of p values. 36 It is important that these p values are computed appropriately, however, accounting for clustering or matching of participants within the studies. Rejecting the null model based on such tests provides no information about the magnitude of the effect, providing information only on whether at least one study shows an effect is present, and if so, its direction. 37

Standard synthesis methods

Meta-analysis for overall effect.

Probably the most familiar approach to meta-analysis is that of estimating a single summary effect across similar studies. This simple approach lends itself to the use of forest plots to display the results of individual studies as well as syntheses, as illustrated for the breastfeeding studies in figure 1 (panel A). This analysis addresses the broad question of whether evidence from a collection of studies supports an impact of the complex intervention of interest, and requires that every study makes a comparison of a relevant intervention against a similar alternative. In the context of complex interventions, this is described by Caldwell and Welton as the ‘lumping’ approach, 38 and by Guise et al as the ‘holistic’ approach. 5 6 One key limitation of the simple approach is that it requires similar types of data from each study. A second limitation is that the meta-analysis result may have limited relevance when the studies are diverse in their characteristics. Fixed-effect models, for instance, are unlikely to be appropriate for complex interventions because they ignore between-studies variability in underlying effect sizes. Results based on random-effects models will need to be interpreted by acknowledging the spread of effects across studies, for example, using prediction intervals. 39

A common problem when undertaking a simple meta-analysis is that individual studies may report many effect sizes that are correlated with each other, for example, if multiple outcomes are measured, or the same outcome variable is measured at several time points. Numerous approaches are available for dealing with such multiplicity, including multivariate meta-analysis, multilevel modelling, and strategies for selecting effect sizes. 40 A very simple strategy that has been used in systematic reviews of complex interventions is to take the median effect size within each study, and to summarise these using the median of these effect sizes across studies. 41

Exploring heterogeneity

Diversity in the types of participants (and contexts), interventions and outcomes are key to understanding sources of complexity. 9 Many of these important sources of heterogeneity are most usefully examined—to the extent that they can reliably be understood—using standard approaches for understanding variability across studies, such as subgroup analyses and meta-regression.

A simple strategy to explore heterogeneity is to estimate the overall effect separately for different levels of a factor using subgroup analyses (referring to subgrouping studies rather than participants). 42 As an example, McFadden et al conducted a systematic review and meta-analysis of 73 studies of support for healthy breastfeeding mothers with healthy term babies. 43 They calculated separate average effects for interventions delivered by a health professional, a lay supporter or with mixed support, and found that the effect on cessation of exclusive breast feeding at up to 6 months was greater for lay support compared with professionals or mixed support (p=0.02). Guise et al provide several ways of grouping studies according to their interventions, for example, grouping studies by key components, by function or by theory. 5 6

Meta-regression provides a flexible generalisation to subgroup analyses, whereby study-level covariates are included in a regression model using effect size estimates as the dependent variable. 44 45 Both continuous and categorical covariates can be included in such models; with a single categorical covariate, the approach is essentially equivalent to subgroup analyses. Meta-regression with continuous covariates in theory allows the extrapolation of relationships to contexts that were not examined in any of the studies, but this should generally be avoided. For example, if the effect of an interventional approach appears to increase as the size of the group to which it is applied decreases, this does not mean that it will work even better when applied to a single individual. More generally, the mathematical form of the relationship modelled in a meta-regression requires careful selection. Most often a linear relationship is assumed, but a linear relationship does not permit step changes such as might occur if an interventional approach requires a particular level of some feature of the underlying system before it has an effect.

Several texts provide guidance for using subgroup analysis and meta-regression in a general context 45 46 and for complex interventions. 3 4 47 In principle, many aspects of complexity in interventions can be addressed using these strategies, to create an understanding of the ‘response surface’. 48–50 However, in practice, the number of studies is often too small for reliable conclusions to be drawn. In general, subgroup analysis and meta-regression are fraught with dangers associated with having few studies, many sources of variation across study features and confounding of these features with each other as well as with other, often unobserved, variables. It is therefore important to prespecify a small number of plausible sources of diversity so as to reduce the danger of reaching spurious conclusions based on study characteristics that correlate with the effects of the interventions but are not the cause of the variation. The ability of statistical analyses to identify true sources of heterogeneity will depend on the number of studies, the sizes of the studies and the true differences between effects in studies with different characteristics.

Synthesis methods for understanding components of the intervention

When interventions comprise distinct components, it is attractive to separate out the individual effects of these components. 51 Meta-regression can be used for this, using covariates to code the presence of particular features in each intervention implementation. As an example, Blakemore et al analysed 39 intervention comparisons from 33 independent studies aiming to reduce urgent healthcare use in adults with asthma. 52 Effect size estimates were coded according to components used in the interventions, and the authors found that multicomponent interventions including skills training, education and relapse prevention appeared particularly effective. In another example, of interventions to support family caregivers of people with Alzheimer’s disease, 53 the authors used methods for decomposing complex interventions proposed by Czaja et al , 54 and created covariates that reduced the complexity of the interventions to a small number of features about the intensity of the interventions. More sophisticated models for examining components have been described by Welton et al , 55 Ivers et al 56 and Madan et al . 57

A component-level approach may be useful when there is a need to disentangle the ‘active ingredients’ of an intervention, for example, when adapting an existing intervention for a new setting. However, components-based approaches require assumptions, such as whether individual components are additive or interact with each other. Furthermore, the effects of components can be difficult to estimate if they are used only in particular contexts or populations, or are strongly correlated with use of other components. An alternative approach is to treat each combination of components as a separate intervention. These separate interventions might then be compared in a single analysis using network meta-analysis. A network meta-analysis combines results from studies comparing two or more of a larger set of interventions, using indirect comparisons via common comparators to rank-order all interventions. 47 58 59 As an example, Achana et al examined the effectiveness of safety interventions on the uptake of three poisoning prevention practices in households with children. Each singular combination of intervention components was defined as a separate intervention in the network. 60 Network meta-analysis may also be useful when there is a need to compare multiple interventions to answer an ‘in principle’ question of which intervention is most effective. Consideration of the main goals of the synthesis will help those aiming to prepare guidelines to decide which of these approaches is most appropriate to their needs.

A case study exploring components is provided in box 1 , and an illustration is provided in figure 2 . The component-based analysis approach can be likened to a factorial trial, in that it attempts to separate out the effects of individual components of the complex interventions, and the network meta-analysis approach can be likened to a multiarm trial approach, where each complex intervention in the set of studies is a different arm in the trial. 47 Deciding between the two approaches can leave the analyst caught between the need to ‘split’ components to reflect complexity (and minimise heterogeneity) and ‘lump’ to make an analysis feasible. Both approaches can be used to examine other features of interventions, including interventions designed for delivery at different levels. For example, a review of the effects of interventions for children exposed to domestic violence and abuse included studies of interventions targeted at children alone, parents alone, children and parents together, and parents and children separately. 61 A network meta-analysis approach was taken to the synthesis, with the people targeted by the intervention used as a distinguishing feature of the interventions included in the network.

Example of understanding components of psychosocial interventions for coronary heart disease

Welton et al reanalysed data from a Cochrane review 89 of randomised controlled trials assessing the effects of psychological interventions on mortality and morbidity reduction for people with coronary heart disease. 55 The Cochrane review focused on the effectiveness of any psychological intervention compared with usual care, and found evidence that psychological interventions reduced non-fatal reinfarctions and depression and anxiety symptoms. The Cochrane review authors highlighted the large heterogeneity among interventions as an important limitation of their review.

Welton et al were interested in the effects of the different intervention components. They classified interventions according to which of five key components were included: educational, behavioural, cognitive, relaxation and psychosocial support ( figure 2 ). Their reanalysis examined the effect of each component in three different ways: (1) An additive model assuming no interactions between components. (2) A two-factor interaction model, allowing for interactions between pairs of components. (3) A network meta-analysis, defining each combination of components as a separate intervention, therefore allowing for full interaction between components. Results suggested that interventions with behavioural components were effective in reducing the odds of all-cause mortality and non-fatal myocardial infarction, and that interventions with behavioural and/or cognitive components were effective for reducing depressive symptoms.

Intervention components in the studies integrated by Welton et al (a sample of 18 from 56 active treatment arms). EDU, educational component; BEH, behavioural component; COG, cognitive component; REL, relaxation component; SUP, psychosocial support component.

A common limitation when implementing these quantitative methods in the context of complex interventions is that replication of the same intervention in two or more studies is rare. Qualitative comparative analysis (QCA) might overcome this problem, being designed to address the ’small N; many variables’ problem. 62 QCA involves: (1) Identifying theoretically driven thresholds for determining intervention success or failure. (2) Creating a 'truth table’, which takes the form of a matrix, cross-tabulating all possible combinations of conditions (eg, participant and intervention characteristics) against each study and its associated outcomes. (3) Using Boolean algebra to eliminate redundant conditions and to identify configurations of conditions that are necessary and/or sufficient to trigger intervention success or failure. QCA can usefully complement quantitative integration, sometimes in the context of synthesising diverse types of evidence.

Synthesis methods for understanding mechanisms of action

An alternative purpose of a synthesis is to gain insight into the mechanisms of action behind an intervention, to inform its generalisability or applicability to a particular context. Such syntheses of quantitative data may complement syntheses of qualitative data, 11 and the two forms might be integrated. 12 Logic models, or theories of action, are important to motivate investigations of mechanism. The synthesis is likely to focus on intermediate outcomes reflecting intervention processes, and on mediators of effect (factors that influence how the intervention affects an outcome measure). Two possibilities for analysis are to use these intermediate measurements as predictors of main outcomes using meta-regression methods, 63 or to use multivariate meta-analysis to model the intermediate and main outcomes simultaneously, exploiting and estimating the correlations between them. 64 65 If the synthesis suggests that hypothesised chains of outcomes hold, this lends weight to the theoretical model underlying the hypothesis.

An approach to synthesis closely identified with this category of interventions is model-driven meta-analysis, in which different sources of evidence are integrated within a causal path model akin to a directed acyclic graph. A model-driven meta-analysis is an explanatory analysis. 66 It attempts to go further than a standard meta-analysis or meta-regression to explore how and why an intervention works, for whom it works, and which aspects of the intervention (factors) are driving overall effect. Such syntheses have been described in frequentist 19 67–70 and Bayesian 71 72 frameworks and are variously known as model-driven meta-analysis, linked meta-analysis, meta-mediation analysis and meta-analysis of structural equation models. In their simplest form, standard meta-analyses estimate a summary correlation independently for each pair of variables in the model. The approach is inherently multivariate, requiring the estimation of multiple correlations (which, if obtained from a single study, are also not independent). 73–75 Each study is likely to contribute fragments of the correlation matrix. A summary correlation matrix, combined either by fixed-effects or random-effects methods, then serves as the input for subsequent analysis via a standardised regression or structural equation model.

An example is provided in box 2 . The model in figure 3 postulates that the effect of ‘Dietary adherence’ on ‘Diabetes complications’ is not direct but is mediated by ‘Metabolic control’. 76 The potential for model-driven meta-analysis to incorporate such indirect effects also allows for mediating effects to be explicitly tested and in so doing allows the meta-analyst to identify and explore the mechanisms underpinning a complex intervention. 77

Theoretical diabetes care model (adapted from Brown et al 68 ).

Example of a model-driven meta-analysis for type 2 diabetes

Brown et al present a model-driven meta-analysis of correlational research on psychological and motivational predictors of diabetes outcomes, with medication and dietary adherence factors as mediators. 76 In a linked methodological paper, they present the a priori theoretical model on which their analysis is based. 68 The model is simplified in figure 3 , and summarised for the dietary adherence pathway only. The aim of their full analysis was to determine the predictive relationships among psychological factors and motivational factors on metabolic control and body mass index (BMI), and the role of behavioural factors as possible mediators of the associations among the psychological and motivational factors and metabolic control and BMI outcomes.

The analysis is based on a comprehensive systematic review. Due to the number of variables in their full model, 775 individual correlational or predictive studies reported across 739 research papers met eligibility criteria. Correlations between each pair of variables in the model were summarised using an overall average correlation, and homogeneity assessed. Multivariate analyses were used to estimate a combined correlation matrix. These results were used, in turn, to estimate path coefficients for the predictive model and their standard errors. For the simplified model illustrated here, the results suggested that coping and self-efficacy were strongly related to dietary adherence, which was strongly related to improved glycaemic control and, in turn, a reduction in diabetic complications.

Synthesis approaches for understanding complexities of the system

Syntheses may seek to address complexities of the system to understand either the impact of the system on the effects of the intervention or the effects of the intervention on the system. This may start by modelling the salient features of the system’s dynamics, rather than focusing on interventions. Subgroup analysis and meta-regression are useful approaches for investigating the extent to which an intervention’s effects depend on baseline features of the system, including aspects of the context. Sophisticated meta-regression models might investigate multiple baseline features, using similar approaches to the component-based meta-analyses described earlier. Specifically, aspects of context or population characteristics can be regarded as ‘components’ of the system into which the intervention is introduced, and similar statistical modelling strategies used to isolate effects of individual factors, or interactions between them.

When interventions act at multiple levels, it may be important to understand the effects at these different levels. Outcomes may be measured at different levels (eg, at patient, clinician and clinical practice levels) and analysed separately. Qualitative research plays a particularly important role in identifying the outcomes that should be assessed through quantitative synthesis. 12 Care is needed to ensure that the unit of analysis issues are addressed. For example, if clinics are the unit of randomisation, then outcomes measured at the clinic level can be analysed using standard methods, whereas outcomes measured at the level of the patient within the clinic would need to account for clustering. In fact, multiple dependencies may arise in such data, when patients receive care in small groups. Detailed investigations of effect at different levels, including interactions between the levels, would lend themselves to multilevel (hierarchical) models for synthesis. Unfortunately, individual participant data at all levels of the hierarchy are needed for such analyses.

Model-based approaches also offer possibilities for addressing complex systems; these include economic models, mathematical models and systems science methods generally. 78–80 Broadly speaking, these provide mathematical representations of logic models, and analyses may involve incorporation of empirical data (eg, from systematic reviews), computer simulation, direct computation or a mixture of these. Multiparameter evidence synthesis methods might be used. 81 82 Approaches include models to represent systems (eg, systems dynamics models) and approaches that simulate individuals within the system (eg, agent-based models). 79 Models can be particularly useful when empirical evidence does not address all important considerations, such as ‘real-world’ contexts, long-term effects, non-linear effects and complexities such as feedback loops and threshold effects. An example of a model-based approach to synthesis is provided in box 3 . The challenge when adopting these approaches is often in the identification of system components, and accurately estimating causes and effects (and uncertainties). There are few examples of the use of these analytical tools in systematic reviews, but they may be useful when the focus of analysis is on understanding the causes of complexity in a given system rather than on the impact of an intervention.

Example of a mathematical modelling approach for soft drinks industry levy

Briggs et al examined the potential impact of a soft drinks levy in the UK, considering possible different types of response to the levy by industry. 90 Various scenarios were posited, with effects on health outcomes informed by empirical data from randomised trials and cohort studies of association between sugar intake and body weight, diabetes and dental caries. Figure 4 provides a simple characterisation of how the empirical data were fed into the model. Inputs into the model included levels of consumption of various types of drinks (by age and sex), volume of drinks sales, and baseline levels of obesity, diabetes and dental caries (by age and sex). The authors concluded that health gains would be greatest if industry reacted by reformulating their products to include less sugar.

Simplified version of the conceptual model used by Briggs et al ( a dapted from Briggs et al 90 ).

Considerations of bias and relevance

It is always important to consider the extent to which (1) The findings from each study have internal validity, particularly for non-randomised studies which are typically at higher risk of bias. (2) Studies may have been conducted but not reported because of unexciting findings. (3) Each study is applicable to the purposes of the review, that is, has external validity (or ‘directness’), in the language of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group. 83 At minimum, internal and external validity should be examined and reported, and the risk of publication bias assessed, and these can be achieved through the GRADE framework. 10 With sufficient studies, information collected might be used in meta-regression analyses to evaluate empirically whether studies with and without specific sources of bias or indirectness differ in their results.

It may be desirable to learn about a specific setting, intervention type or outcome measure more directly than others. For example, to inform a decision for a low-income setting, emphasis should be placed on results of studies performed in low-income countries. One option is to restrict the synthesis to these studies. An alternative is to model the dependence of an intervention’s effect on some feature(s) related to the income setting, and extract predictions from the model that are most relevant to the setting of interest. This latter approach makes fuller use of available data, but relies on stronger assumptions.

Often, however, the accumulated studies are too few or too disparate to draw conclusions about the impact of bias or relevance. On rare occasions, syntheses might implement formal adjustments of individual study results for likely biases. Such adjustments may be made by imposing prior distributions to depict the magnitude and direction of any biases believed to exist. 84 85 The choice of a prior distribution may be informed by formal assessments of risk of bias, by expert judgement, or possibly by empirical data from meta-epidemiological studies of biases in randomised and/or non-randomised studies. 86 For example, Wolf et al implemented a prior distribution based on findings of a meta-epidemiological study 87 to adjust for lack of blinding in studies of interventions to improve quality of point-of-use water sources in low-income and middle-income settings. 88 Unfortunately, empirical evidence of bias is mostly limited to clinical trials, is weak for trials of public health and social care interventions, and is largely non-existent for non-randomised studies.

Our review of quantitative synthesis methods for evaluating the effects of complex interventions has outlined many possible approaches that might be considered by those collating evidence in support of guideline development. We have described three broad categories: (1) Non-quantitative methods, including tabulation, narrative and graphical approaches. (2) Standard meta-analysis methods, including meta-regression to investigate study-level moderators of effect. (3) More advanced synthesis methods, in which models allow exploration of intervention components, investigation of both moderators and mediators, examination of mechanisms, and exploration of complexities of the system.

The choice among these approaches will depend on the purpose of the synthesis, the similarity of the studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team, and the resources available. Clearly the advanced methods require more expertise and resources than the simpler methods. Furthermore, they require a greater level of detail and typically a sizeable evidence base. We therefore expect them to be used seldomly; our aim here is largely to articulate what they can achieve so that they can be adopted when they are appropriate. Notably, the choice among these approaches will also depend on the extent to which guideline developers and users at global, national or local levels understand and are willing to base their decisions on different methods. Where possible, it will thus be important to involve concerned stakeholders during the early stages of the systematic review process to ensure the relevance of its findings.

Complexity is common in the evaluation of public health interventions at individual, organisational or community levels. To help systematic review and guideline development teams decide how to address this complexity in syntheses of quantitative evidence, we summarise considerations and methods in tables 1 and 2 . We close with the important remark that quantitative synthesis is not always a desirable feature of a systematic review. Whereas some sophisticated methods are available to deal with a variety of complex problems, on many occasions—perhaps even the majority in practice—the studies may be too different from each other, too weak in design or have data too sparse, for statistical methods to provide insight beyond a commentary on what evidence has been identified.

Acknowledgments

The authors thank the following for helpful comments on earlier drafts of the paper: Philippa Easterbrook, Matthias Egger, Anayda Portela, Susan L Norris, Mark Petticrew.

• Petticrew M ,
• Thomas J , et al
• Anderson L ,
• Elder R , et al
• Rehfuess E ,
• Noyes J , et al
• Viswanathan M , et al
• World Health Organization
• Rehfuess EA ,
• Stratil JM ,
• Scheel IB , et al
• Flemming K , et al
• Montgomery P ,
• Movsisyan A ,
• Grant SP , et al
• Flemming K ,
• Garside R , et al
• Moore G , et al
• Richardson WS ,
• Wilson MC ,
• Nishikawa J , et al
• Chowdhury R ,
• Sankar MJ , et al
• Collins D ,
• Johnson K ,
• von Philipsborn P ,
• Burns J , et al
• Schoonees A ,
• Becker BJ ,
• Duvendack M , et al
• Reeves BC ,
• Higgins JPT , et al
• Roberts H ,
• Ryan R , Cochrane Consumers and Communication Review Group
• Anzures-Cabrera J ,
• Harrison S ,
• Martin RM , et al
• Ogilvie D ,
• Petticrew M , et al
• Sterne JA ,
• Davey Smith G
• Crowther M ,
• Avenell A ,
• MacLennan G , et al
• Thomson HJ ,
• Taylor SL ,
• Higgins JPT ,
• Bushman BJ ,
• Caldwell DM ,
• Thompson SG ,
• Spiegelhalter DJ
• López-López JA ,
• Lipsey MW , et al
• Grimshaw JM ,
• Thomas RE ,
• Borenstein M ,
• McFadden A ,
• Renfrew MJ , et al
• van Houwelingen HC ,
• Arends LR ,
• Melendez-Torres GJ ,
• Ioannidis JP ,
• Steckelberg A ,
• Richter B , et al
• Presseau J ,
• Newham JJ , et al
• Blakemore A ,
• Dickens C ,
• Anderson R , et al
• Schulz R , et al
• Lee CC , et al
• Welton NJ ,
• Adamopoulos E , et al
• Tricco AC ,
• Trikalinos TA , et al
• Aveyard P , et al
• Calderbank-Batista T
• Achana FA ,
• Cooper NJ ,
• Bujkiewicz S , et al
• Howarth E ,
• Moore THM ,
• Welton NJ , et al
• O’Mara-Eves A ,
• Thompson SG
• Jackson D ,
• Raudenbush SW ,
• García AA , et al
• Cheung MW ,
• Watson SI ,
• Stewart GB ,
• Mengersen K ,
• García AA ,
• Brown A , et al
• Liu H , et al
• Johnson L ,
• Althaus C , et al
• Stamatakis KA
• Greenwood-Lee J ,
• Nettel-Aguirre A , et al
• Caldwell D , et al
• Colbourn T ,
• Asseburg C ,
• Bojke L , et al
• Guyatt GH ,
• Oxman AD , GRADE Working Group
• Turner RM ,
• Spiegelhalter DJ ,
• Smith GCS , et al
• Carlin JB , et al
• Schulz KF , et al
• Savović J ,
• Altman DG , et al
• Prüss-Ustün A ,
• Cumming O , et al
• Bennett P ,
• West R , et al
• Briggs ADM ,
• Mytton OT ,
• Kehlbacher A , et al

Handling editor Soumyadeep Bhaumik

Contributors JPTH co-led the project, conceived the paper, led discussions and wrote the first draft. JAL-L undertook analyses, contributed to discussions and contributed to writing the manuscript. BJB drafted material on mechanisms, contributed to discussions and contributed extensively to writing the manuscript. SRD screened and categorised the results of the literature searches, collated examples and contributed to discussions. SD undertook searches to identify relevant literature and contributed to discussions. JMG contributed to discussions and commented critically on drafts. LAM undertook analyses, contributed to discussions and commented critically on drafts. THMM contributed examples, contributed to discussions and commented critically on drafts. EAR and JT contributed to discussions and commented critically on drafts. DMC co-led the project, contributed to discussions and drafted extensive parts of the paper. All authors approved the final version of the manuscript.

Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation. JPTH was funded in part by Medical Research Council (MRC) grant MR/M025209/1, by the MRC Integrative Epidemiology Unit at the University of Bristol (MC_UU_12013/9) and by the MRC ConDuCT-II Hub (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures – MR/K025643/1). BJB was funded in part by grant DRL-1252338 from the US National Science Foundation (NSF). JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake. LAM is funded by a National Institute for Health Research (NIHR) Systematic Review Fellowship (RM-SR-2016-07 26). THMM was funded by the NIHR Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West). JT is supported by the NIHR Collaboration for Leadership in Applied Health Research and Care North Thames at Bart’s Health NHS Trust. DMC was funded in part by NIHR grant PHR 15/49/08 and by the Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer –MR/KO232331/1).

Disclaimer The views expressed are those of the authors and not necessarily those of the CRC program, the MRC, the NSF, the NHS, the NIHR or the UK Department of Health.

Competing interests JMG reports personal fees from the Campbell Collaboration. EAR reports being a Methods Editor with Cochrane Public Health.

Patient consent Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

473 Accesses

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Ohio State nav bar

The Ohio State University

• BuckeyeLink
• Find People
• Search Ohio State

Literature Review

What exactly is a literature review.

• Critical Exploration and Synthesis: It involves a thorough and critical examination of existing research, going beyond simple summaries to synthesize information.
• Reorganizing Key Information: Involves structuring and categorizing the main ideas and findings from various sources.
• Offering Fresh Interpretations: Provides new perspectives or insights into the research topic.
• Merging New and Established Insights: Integrates both recent findings and well-established knowledge in the field.
• Analyzing Intellectual Trajectories: Examines the evolution and debates within a specific field over time.
• Contextualizing Current Research: Places recent research within the broader academic landscape, showing its relevance and relation to existing knowledge.
• Detailed Overview of Sources: Gives a comprehensive summary of relevant books, articles, and other scholarly materials.
• Highlighting Significance: Emphasizes the importance of various research works to the specific topic of study.

How do Literature Reviews Differ from Academic Research Papers?

• Focus on Existing Arguments: Literature reviews summarize and synthesize existing research, unlike research papers that present new arguments.
• Secondary vs. Primary Research: Literature reviews are based on secondary sources, while research papers often include primary research.
• Foundational Element vs. Main Content: In research papers, literature reviews are usually a part of the background, not the main focus.
• Lack of Original Contributions: Literature reviews do not introduce new theories or findings, which is a key component of research papers.

Purpose of Literature Reviews

• Drawing from Diverse Fields: Literature reviews incorporate findings from various fields like health, education, psychology, business, and more.
• Prioritizing High-Quality Studies: They emphasize original, high-quality research for accuracy and objectivity.
• Serving as Comprehensive Guides: Offer quick, in-depth insights for understanding a subject thoroughly.
• Foundational Steps in Research: Act as a crucial first step in conducting new research by summarizing existing knowledge.
• Providing Current Knowledge for Professionals: Keep professionals updated with the latest findings in their fields.
• Demonstrating Academic Expertise: In academia, they showcase the writer’s deep understanding and contribute to the background of research papers.
• Essential for Scholarly Research: A deep understanding of literature is vital for conducting and contextualizing scholarly research.

A Literature Review is Not About:

• Merely Summarizing Sources: It’s not just a compilation of summaries of various research works.
• Ignoring Contradictions: It does not overlook conflicting evidence or viewpoints in the literature.
• Being Unstructured: It’s not a random collection of information without a clear organizing principle.
• Avoiding Critical Analysis: It doesn’t merely present information without critically evaluating its relevance and credibility.
• Focusing Solely on Older Research: It’s not limited to outdated or historical literature, ignoring recent developments.
• Isolating Research: It doesn’t treat each source in isolation but integrates them into a cohesive narrative.

Steps Involved in Conducting a Research Literature Review (Fink, 2019)

1. choose a clear research question., 2. use online databases and other resources to find articles and books relevant to your question..

• Google Scholar
• OSU Library
• ERIC. Index to journal articles on educational research and practice.
• PsycINFO . Citations and abstracts for articles in 1,300 professional journals, conference proceedings, books, reports, and dissertations in psychology and related disciplines.
• PubMed . This search system provides access to the PubMed database of bibliographic information, which is drawn primarily from MEDLINE, which indexes articles from about 3,900 journals in the life sciences (e.g., health, medicine, biology).
• Social Sciences Citation Index . A multidisciplinary database covering the journal literature of the social sciences, indexing more than 1,725 journals across 50 social sciences disciplines.

3. Decide on Search Terms.

• Pick words and phrases based on your research question to find suitable materials
• You can start by finding models for your literature review, and search for existing reviews in your field, using “review” and your keywords. This helps identify themes and organizational methods.
• Narrowing your topic is crucial due to the vast amount of literature available. Focusing on a specific aspect makes it easier to manage the number of sources you need to review, as it’s unlikely you’ll need to cover everything in the field.
• Use AND to retrieve a set of citations in which each citation contains all search terms.
• Use OR to retrieve citations that contain one of the specified terms.
• Use NOT to exclude terms from your search.
• Be careful when using NOT because you may inadvertently eliminate important articles. In Example 3, articles about preschoolers and academic achievement are eliminated, but so are studies that include preschoolers as part of a discussion of academic achievement and all age groups.

4. Filter out articles that don’t meet criteria like language, type, publication date, and funding source.

• Publication language Example. Include only studies in English.
• Journal Example. Include all education journals. Exclude all medical journals.
• Author Example. Include all articles by Andrew Hayes.
• Setting Example. Include all studies that take place in family settings. Exclude all studies that take place in the school setting.
• Participants or subjects Example. Include children that are younger than 6 years old.
• Program/intervention Example. Include all programs that are teacher-led. Exclude all programs that are learner-initiated.
• Research design Example. Include only longitudinal studies. Exclude cross-sectional studies.
• Sampling Example. Include only studies that rely on randomly selected participants.
• Date of publication Example. Include only studies published from January 1, 2010, to December 31, 2023.
• Date of data collection Example. Include only studies that collected data from 2010 through 2023. Exclude studies that do not give dates of data collection.
• Duration of data collection Example. Include only studies that collect data for 12 months or longer.

5. Evaluate the methodological quality of the articles, including research design, sampling, data collection, interventions, data analysis, results, and conclusions.

• Maturation: Changes in individuals due to natural development may impact study results, such as intellectual or emotional growth in long-term studies.
• Selection: The method of choosing and assigning participants to groups can introduce bias; random selection minimizes this.
• History: External historical events occurring simultaneously with the study can bias results, making it hard to isolate the study’s effects.
• Instrumentation: Reliable data collection tools are essential to ensure accurate findings, especially in pretest-posttest designs.
• Statistical Regression: Selection based on extreme initial measures can lead to misleading results due to regression towards the mean.
• Attrition: Loss of participants during a study can bias results if those remaining differ significantly from those who dropped out.
• Reactive Effects of Testing: Pre-intervention measures can sensitize participants to the study’s aims, affecting outcomes.
• Interactive Effects of Selection: Unique combinations of intervention programs and participants can limit the generalizability of findings.
• Reactive Effects of Innovation: Artificial experimental environments can lead to uncharacteristic behavior among participants.
• Multiple-Program Interference: Difficulty in isolating an intervention’s effects due to participants’ involvement in other activities or programs.
• Simple Random Sampling : Every individual has an equal chance of being selected, making this method relatively unbiased.
• Systematic Sampling : Selection is made at regular intervals from a list, such as every sixth name from a list of 3,000 to obtain a sample of 500.
• Stratified Sampling : The population is divided into subgroups, and random samples are then taken from each subgroup.
• Cluster Sampling : Natural groups (like schools or cities) are used as batches for random selection, both at the group and individual levels.
• Convenience Samples : Selection probability is unknown; these samples are easy to obtain but may not be representative unless statistically validated.
• Study Power: The ability of a study to detect an effect, if present, is known as its power. Power analysis helps identify a sample size large enough to detect this effect.
• Test-Retest Reliability: High correlation between scores obtained at different times, indicating consistency over time.
• Equivalence/Alternate-Form Reliability: The degree to which two different assessments measure the same concept at the same difficulty level.
• Homogeneity: The extent to which all items or questions in a measure assess the same skill, characteristic, or quality.
• Interrater Reliability: Degree of agreement among different individuals assessing the same item or concept.
• Content Validity: Measures how thoroughly and appropriately a tool assesses the skills or characteristics it’s supposed to measure. Face Validity: Assesses whether a measure appears effective at first glance in terms of language use and comprehensiveness. Criterion Validity: Includes predictive validity (forecasting future performance) and concurrent validity (agreement with already valid measures). Construct Validity: Experimentally established to show that a measure effectively differentiates between people with and without certain characteristics.
• Relies on factors like the scale (categorical, ordinal, numerical) of independent and dependent variables, the count of these variables, and whether the data’s quality and characteristics align with the chosen statistical method’s assumptions.

6. Use a standard form for data extraction, train reviewers if needed, and ensure quality.

7. interpret the results, using your experience and the literature’s quality and content. for a more detailed analysis, a meta-analysis can be conducted using statistical methods to combine study results., 8. produce a descriptive review or perform a meta-analysis..

• Example: Bryman, A. (2007). Effective leadership in higher education: A literature review. Studies in higher education, 32(6), 693-710.
• Clarify the objectives of the analysis.
• Set explicit criteria for including and excluding studies.
• Describe in detail the methods used to search the literature.
• Search the literature using a standardized protocol for including and excluding studies.
• Use a standardized protocol to collect (“abstract”) data from each study regarding study purposes, methods, and effects (outcomes).
• Describe in detail the statistical method for pooling results.
• Report results, conclusions, and limitations.

• Example: Yu, Z. (2023). A meta-analysis of the effect of virtual reality technology use in education. Interactive Learning Environments, 31 (8), 4956-4976.
• Essential and Multifunctional Bibliographic Software: Tools like EndNote, ProCite, BibTex, Bookeeper, Zotero, and Mendeley offer more than just digital storage for references; they enable saving and sharing search strategies, directly inserting references into reports and scholarly articles, and analyzing references by thematic content.
• Comprehensive Literature Reviews: Involve supplementing electronic searches with a review of references in identified literature, manual searches of references and journals, and consulting experts for both unpublished and published studies and reports.
• One of the most famous reporting checklists is the Consolidated Standards of Reporting Trials ( CONSORT ). CONSORT consists of a checklist and flow diagram. The checklist includes items that need to be addressed in the report.

References:

Bryman, A. (2007). Effective leadership in higher education: A literature review.  Studies in higher education ,  32 (6), 693-710.

Fink, A. (2019).  Conducting research literature reviews: From the internet to paper . Sage publications.

Yu, Z. (2023). A meta-analysis of the effect of virtual reality technology use in education. Interactive Learning Environments, 31 (8), 4956-4976.

Change Password

Your password must have 8 characters or more and contain 3 of the following:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

• Sign in / Register

Request Username

Can't sign in? Forgot your username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

The Other Half of the Story: Effect Size Analysis in Quantitative Research

• Jessica Middlemis Maher
• Jonathan C. Markey
• Diane Ebert-May

Search for more papers by this author

Address correspondence to: Diane Ebert-May ( E-mail Address: [email protected] ).

Statistical significance testing is the cornerstone of quantitative research, but studies that fail to report measures of effect size are potentially missing a robust part of the analysis. We provide a rationale for why effect size measures should be included in quantitative discipline-based education research. Examples from both biological and educational research demonstrate the utility of effect size for evaluating practical significance. We also provide details about some effect size indices that are paired with common statistical significance tests used in educational research and offer general suggestions for interpreting effect size measures. Finally, we discuss some inherent limitations of effect size measures and provide further recommendations about reporting confidence intervals.

INTRODUCTION

Quantitative research in biology education is primarily focused on describing relationships between variables. Authors often rely heavily on analyses that determine whether the observed effect is real or attributable to chance, that is, the statistical significance, without fully considering the strength of the relationship between those variables ( Osbourne, 2008 ). While most researchers would agree that determining the practical significance of their results is important, statistical significance testing alone may not provide all information about the magnitude of the effect or whether the relationship between variables is meaningful ( Vaske, 2002 ; Nakagawa and Cuthill, 2007 ; Ferguson, 2009 ).

In education research, statistical significance testing has received valid criticisms, primarily because the numerical outcome of the test is often promoted while the equally important issue of practical significance is ignored ( Fan, 2001 ; Kotrlik and Williams, 2003 ). As a consequence, complete reliance on statistical significance testing limits understanding and applicability of research findings in education practice. Therefore, authors and referees are increasingly calling for the use of statistical tools that supplement traditionally performed tests for statistical significance (e.g., Thompson, 1996 ; Wilkinson and American Psychological Association [APA] Task Force on Statistical Inference, 1999 ). One such tool is the confidence interval , which provides an estimate of the magnitude of the effect and quantifies the uncertainly around this estimate. A similarly useful statistical tool is the effect size , which measures the strength of a treatment response or relationship between variables. By quantifying the magnitude of the difference between groups or the relationship among variables, effect size provides a scale-free measure that reflects the practical meaningfulness of the difference or the relationship among variables ( Coe, 2002 ; Hojat and Xu, 2004 ).

In this essay, we explain the utility of including effect size in quantitative analyses in educational research and provide details about effect size metrics that pair well with the most common statistical significance tests. It is important to note that effect size and statistical significance testing (which we will shorten to “significance testing,” also known as hypothesis testing) are complementary analyses, and both should be considered when evaluating quantitative research findings ( Fan, 2001 ). To illustrate this point, we begin with two hypothetical examples: one in biology and one in education.

Effect Size and Statistical Significance Testing: Why Both Are Necessary

Imagine that a researcher set up two treatment conditions: for example, unfertilized and fertilized plants in a greenhouse or, similarly, reformed and traditional teaching approaches in different sections of an introductory biology course. The researcher is interested in knowing whether the first treatment is more or less effective than the second, using some measurable outcome (e.g., dried plant biomass or student performance on an exam); this constitutes the research hypothesis. The null hypothesis states that there is no difference between the treatments. Owing to sampling variation in a finite sample size, even if the two treatments are equally effective (i.e., the null hypothesis is true), one sample mean will nearly always be greater than the other. Therefore, the researcher must employ a statistical significance test to determine the probability of a difference between the sample means occurring by chance when the null hypothesis is true. Using the appropriate test, the researcher may determine that sampling variability is not a likely explanation for the observed difference and may reject the null hypothesis in favor of the alternative research hypothesis. The ability to make this determination is afforded by the statistical power, which is the probability of detecting a treatment effect when one exists, of the significance test. Statistical power is primarily determined by the size of the effect and the size of the sample: as either or both increase, the significance test is said to have greater statistical power to reject the null hypothesis.

The basis for rejection of the null hypothesis is provided by the p value, which is the output of statistical significance testing that is upheld as nearly sacred by many quantitative researchers. The p value represents the probability of the observed data (or more extreme data) given that the null hypothesis is true: Pr(observed data|H 0 ), assuming that the sampling was random and done without error ( Kirk, 1996 ; Johnson, 1999 ). A low value of p , typically below 0.05, usually leads researchers to reject the null hypothesis. However, as critics of significance testing have pointed out, the abuse of this rather arbitrary cutoff point tends to reduce the decision to a reject/do not reject dichotomy ( Kirk, 1996 ). In addition, many researchers believe that the smaller the value of p , the larger the treatment effect ( Nickerson, 2000 ), equating the outcome of significance testing to the importance of the findings ( Thompson, 1993 ). This misunderstanding is likely due to the fact that when sample size is held constant, the value of p correlates with effect size for some statistical significance tests. However, that relationship completely breaks down when sample size changes. As described earlier, the ability of any significance test to detect a fixed effect depends entirely on the statistical power afforded by the size of the sample. Thus, for a set difference between two populations, simply increasing sample size may allow for easier rejection of the null hypothesis. Therefore, given enough observations to afford sufficient statistical power, any small difference between groups can be shown to be “significant” using a statistical significance test.

The sensitivity of significance testing to sample size is an important reason why many researchers advocate reporting effect sizes and confidence intervals alongside test statistics and p values ( Kirk, 1996 ; Thompson, 1996 ; Fan, 2001 ). Kotrlik and Williams (2003) highlight a particularly clear example in which statistical and practical significance differ. In their study, Williams (2003) was interested in comparing the percent time that faculty members spend teaching with the percent time that they would prefer to spend teaching. Despite the fact that the mean differences between actual and preferred teaching time were statistically significant ( t 154 = 2.20, p = 0.03), the effect size (Cohen's d = 0.09) was extremely small (see Tables 1 and 2 for effect size metrics and interpretations). As a result, the author did not suggest that there were practically important differences between actual and preferred teaching time commitments ( Williams, 2003 ). Reporting the confidence interval would have also illustrated the small effect in this study: while the confidence interval would not have contained zero, one of its end points would have been very close to zero, suggesting that the population mean difference could be quite small.

a Cohen, 1992 , 1988 ; Rosenthal, 1996 .

Although Williams (2003) presents a case in which a small “significant” p value could have led to an erroneous conclusion of practically meaningful difference, the converse also occurs. For example, Thomas and Juanes (1996) present an example from a study of juvenile rainbow trout willingness to forage under the risk of predation ( Johnsson, 1993 ). An important part of the study tested the null hypothesis that large and small juveniles do not differ in their susceptibility to the predator, an adult trout. Using eight replicate survivorship trials, Johnsson (1993) found no significant difference in the distribution of risk between the two size classes (Wilcoxon signed-rank test: T + = 29, p = 0.15). However, the data suggest that there may in fact be a biologically significant effect: on average, 19 ± 4.9% (mean ± SE) of the large fish and 45 ± 7% of the small fish were killed by the predator ( Johnsson, 1993 ). This difference likely represents a medium effect size (see Table 2 ; Thomas and Juanes, 1996 ). Not reporting effect size resulted in the researchers failing to reject the null hypothesis, possibly due to low statistical power (small sample size), and the potential to erroneously conclude that there were no differences in relative predation risk between size classes of juvenile trout.

Thus, metrics of effect size and statistical significance provide complementary information: the effect size indicates the magnitude of the observed effect or relationship between variables, whereas the significance test indicates the likelihood that the effect or relationship is due to chance. Therefore, interpretations derived from statistical significance testing alone have the potential to be flawed, and inclusion of effect size reporting is essential to inform researchers about whether their findings are practically meaningful or important. Despite the fact that effect size metrics have been available since the 1960s ( Huberty, 2002 ) and have been recognized as being a potentially useful aspect of analyses since the 1990s (e.g., Cohen, 1994 ; Thompson, 1996 ; Wilkinson and APA Task Force on Statistical Inference, 1999 ), the adoption of effect size as a complement to significance testing has been a slow process, even in high-impact research ( Tressoldi et al. , 2013 ). Nevertheless, many journals are beginning to develop editorial policies requiring some measure of effect size to be reported in quantitative studies (e.g., Royer, 2000 ). In response to this need for implementation, we next discuss the various methods used to calculate effect sizes and provide guidance regarding the interpretation of effect size indices.

Measures of Effect Size: Two Categories

We concentrate on parametric tests and group effect sizes into two main categories: those for 1) comparing two or more groups and 2) determining strength of associations between variables. The most frequently used statistical tests in these two categories are associated with specific effect size indices (see Table 1 ; Cohen, 1992 ), and we will discuss some of the more common methods used for each below. Refer to Figure 1 for a general guide to selecting the appropriate effect size measure for your data.

Figure 1. A dichotomous key to selecting an appropriate measure of effect size. Because many quantitative researchers are already accustomed to employing statistical significance tests but may want to begin reporting effect sizes as well, we suggest effect size metrics that are appropriate for data analyzed using common significance tests. Although not intended to be a comprehensive guide to effect size indices, this key indicates many of the measures relevant for common quantitative analyses in educational research. Researchers are encouraged to gather more information about these metrics, including their assumptions and limitations.

Comparing Two or More Groups.

A common approach to both biological and educational research questions is to compare two or more groups, such as in our earlier examples comparing the effects of a treatment on plant growth or student performance. For these kinds of analyses, the appropriate measure of effect size will depend on the type of data collected and the type of statistical test used. We present here a sample of effect size metrics relevant to χ 2 , t , or F tests.

When comparing the distribution of a dichotomous variable between two groups, for instance, when using a χ 2 test of homogeneity, the odds ratio is a useful effect size measure that describes the likelihood of an outcome occurring in the treatment group compared with the likelihood of the outcome occurring in the control group (see Table 1 ; Cohen, 1994 ; Thompson, 1996 ). An odds ratio equal to 1 means that the odds of the outcome occurring is the same in the control and treatment groups. An odds ratio of 2 indicates that the outcome is two times more likely to occur in the treatment group when compared with the control group. Likewise, an odds ratio of 0.5 indicates that the outcome is two times less likely to occur in the treatment group when compared with the control group. Granger et al . (2012) provide an example of reporting odds ratios in educational research. In their study, the effectiveness of a new student-centered curriculum and aligned teacher professional development was compared with a control group. One of the instruments used to measure student outcomes produced dichotomous data, and the odds ratio provided a means for reporting the treatment's effect size on this student outcome. However, the odds ratio alone does not quantify treatment effect, as the magnitude of the effect depends not only on the odds ratio but also on the underlying value of one of the odds in the ratio. For example, if a new treatment for an advanced cancer increases the odds of survival by 50% compared with the existing treatment, then the odds ratio of survival is 1.5. However, if odds control = 0.002 and odds treatment = 0.003, the increase is most likely not practically meaningful. On the other hand, if an odds control = 0.5 and the odds treatment = 0.75, this could be interpreted as a substantial increase that one might find practically meaningful.

When comparing means of continuous variables between two groups using a t test, Cohen's d is a useful effect size measure that describes the difference between the means normalized to the pooled standard deviation (SD) of the two groups (see Table 1 ; Cohen, 1988 ). This measure can be used only when the SDs of two populations represented by the two groups are the same, and the population distributions are close to normal. If the sample sizes between the two groups differ significantly, Hedges’ g is a variation of Cohen's d that can be used to weight the pooled SD based on sample sizes (see Table 1 for calculation; Hedges, 1981 ). If the SDs of the populations differ, then pooling the sample SDs is not appropriate, and other ways to normalize the mean difference should be used. Glass's Δ normalizes the difference between two means to the SD of the control sample (see Table 1 ). This method assumes that the control group's SD is most similar to the population SD, because no treatment is applied ( Glass et al. , 1981 ). There are many relevant examples in the educational research literature that employ variations on Cohen's d to report effect sizes. Abraham et al . (2012) used Cohen's d to show how an instructional treatment affected students’ post scores on a test of the acceptance of evolutionary theory. Similarly, Matthews et al . (2010) used Cohen's d to show the magnitude of change in student's beliefs about the role of mathematics in biology due to changes in course materials, delivery, and assessment between different years of the same course. Gottesman and Hoskins (2013) applied Cohen's d to compare pre/post means of data collected using an instrument measuring students’ critical thinking, experimental design ability, attitudes, and beliefs.

When comparing means of three or more groups, for instance, when using an analysis of variance (ANOVA) test, Cohen's f is an appropriate effect size measure to report ( Cohen, 1988 ). In this method, the sum of the deviations of the sample means from the combined sample mean is normalized to the combined sample SD (see Table 1 ). Note that this test does not distinguish which means differ, but rather just determines whether all means are the same. Other effect size measures commonly reported with ANOVA, multivariate analysis of covariance (MANCOVA), and analysis of covariance (ANCOVA) results are eta-squared and partial eta-squared. Eta-squared is calculated as the ratio of the between-groups sum of squares to the total sum of squares (see Table 1 ; Kerlinger, 1964 ). Alternatively, partial eta-squared is calculated as the ratio of the between-groups sum of squares to the sum of the between-groups sum of squares and the error sum of squares ( Cohen, 1973 ). For example, Quitadamo and Kurtz (2007) reported partial eta-squared, along with ANCOVA/MANCOVA results, to show effect sizes of a writing treatment on student critical thinking. However, eta-squared is deemed by some as a better measure to report, because it describes the variance accounted for by the dependent measure ( Levine and Hullett, 2002 ), which bears similarities to typical measures reported in correlational studies.

Determining Strength of Association between Variables.

Another common approach in both biological and educational research is to measure the strength of association between two or more variables, such as determining the factors that predict student performance on an exam. Many researchers using this type of analysis already report appropriate measures of effect size, perhaps without even realizing they are doing so. In most cases, the regression coefficient or analogous index provides information regarding the magnitude of the effect.

The Pearson product-moment correlation coefficient (Pearson's r ) measures the association between two continuous variables, such as in a linear regression (see Table 1 ). Squaring the r value when performing a simple linear regression results in the coefficient of determination ( r 2 ), a measure that provides information about the amount of variance shared between the two variables. For multiple-regression analysis, the coefficient of multiple determination ( R 2 ) is an appropriate effect size metric to report. If one of the study variables is dichotomous, for example, male versus female or pass versus fail, then the point-biserial correlation coefficient ( r pb ) is the appropriate metric of effect size. The point-biserial correlation coefficient is similar in nature to Pearson's r (see Table 1 ). An easy-to-use Web-based calculator to calculate r pb is located at www.vassarstats.net/pbcorr.html . Spearman's rank correlation coefficient (ρ) is a nonparametric association measure that can be used when both variables are measured on an ordinal or ranked scale or when variables on a continuous scale are not normally distributed. This measure can be used only after one applies a transformation to the data that ranks the values. Because this is a nonparametric measure, Spearman's ρ is not as sensitive to outliers as Pearson's r . Note that there are also variations of Spearman's ρ that handle different formats of data. Most statistical software packages can calculate all of these measures of variable association, as well as most of the measures comparing differences between groups. However, one must be careful to be sure that values provided by the software are indeed what they are claimed to be ( Levine and Hullett, 2002 ).

How to Interpret Effect Sizes

Once you have calculated the effect size measure, how do you interpret the results? With Cohen's d and its variants, mean differences are normalized to SD units. This indicates that a d value of 0.5 can be interpreted as the group means differing by 0.5 SDs. Measures of association report the strength of the relationship between the independent and dependent variables. Additional manipulation of these association values, for example, r 2 , can tell us the amount of shared variance between the variables. For the case of regression analysis, we can assume that an r 2 value of 0.3 means that 30% of the variance in the dependent variable can be explained by the independent variable. Additionally, McGraw and Wong (1992) developed a measure to report what they call “the common language effect size indicator,” which describes the probability that a random value sampled from one group will be greater than a random value sampled from a comparison group ( McGraw and Wong, 1992 ).

Statisticians have determined qualitative descriptors for specific values of each type of effect size measure ( Cohen, 1988 , 1992 ; Rosenthal, 1996 ). For more interpretation of these types of measures, see Table 2 . These values can help guide a researcher to make some sort of statement about the qualitative nature of the effect size, which is useful for communicating the meaning of results. Additionally, effect size interpretations impact the use of data in meta-analyses. Please refer to Box 1 to see an example of how interpretations of the different types of effect size measures can be converted from one type to another for the purpose of meta-analysis.

Effect size measures are an important tool used when performing meta-analyses because they provide a standardized method for comparing results across different studies with similar designs. Two of the more common measures are Pearson's r and Cohen's d . Cohen's d describes the difference between the means of two groups normalized to the pooled standard deviation of the two groups. Pearson's r measures the association between two continuous variables. A problem arises when comparing a study that reports an r value with one that reports a d value. To address this problem, statisticians have developed methods to convert r values into d values, and vice-versa. The equations are listed below:

Limitations of Effect Size

We have built a justification for the reporting of effect sizes as a complement to standard statistical significance testing. However, we do not wish to mislead the reader to construe effect size as a panacea in quantitative analyses. Effect size indices should be used and interpreted just as judiciously as p values. Effect sizes are abstract statistics that experience biases from sampling effort and quality and do not differentiate among relationships of similar magnitude that may actually have more or less practical significance ( Coe, 2002 ; Nakagawa and Cuthill, 2007 ; Ferguson, 2009 ). Rather, determination of what constitutes an effect of practical significance depends on the context of the research and the judgment of the researcher, and the values listed in Table 2 represent somewhat arbitrary cutoffs that are subject to interpretation. Just as researchers may have logical reasons to choose an alpha level other than p = 0.05 with which to interpret statistical significance, the interpretation of practical relationships based on effect size may be more or less conservative, depending on the context. For example, an r of 0.1 for a treatment improving survival of a fatal disease may be of large practical significance. Furthermore, as we mentioned earlier, one should always accompany the proper effect size measure with an appropriate confidence interval whenever possible ( Cohen, 1994 ; Nakagawa and Cuthill, 2007 ; Ellis, 2010 ; Tressoldi et al ., 2013 ). For example, Lauer et al . (2013) reported Cohen's d along with 95% confidence intervals to describe the effects of an administration of a values-affirmation exercise on achievement gaps between men and women in introductory science courses.

By highlighting the problems with relying on statistical significance testing alone to interpret quantitative research results, we hope to have convinced the reader that significance testing is, as Fan (2001) puts it, only one-half of the coin. Our intent is to emphasize that no single statistic is sufficient for describing the strength of relationships among variables or evaluating the practical significance of quantitative findings. Therefore, measures of effect size, including confidence interval reporting, should be used thoughtfully and in concert with significance testing to interpret findings. Already common in such fields as medical and psychological research due to the real-world ramifications of the findings, the inclusion of effect size reporting in results sections is similarly important in educational literature. The measures of effect size described here do not by any means represent the numerous possible indices, but rather are intended to provide an overview of some of the most common and applicable analyses for educational research and a starting point for their inclusion in the reporting of results. In addition to the references cited throughout this article, we recommend several informative and accessible authorities on the subject of effect sizes, summarized in Table 3 .

ACKNOWLEDGMENTS

We thank Alla Sikorskii for helpful comments and edits on an earlier draft of this essay.

• Abraham JK, Perez KE, Downey N, Herron JC, Meir E ( 2012 ). Short lesson plan associated with increased acceptance of evolutionary theory and potential change in three alternate conceptions of macroevolution in undergraduate students . CBE Life Sci Educ 11 , 152-164. Link ,  Google Scholar
• Coe R ( 2002 ). It's the effect size, stupid: what effect size is and why it is important In: Paper presented at the Annual Conference of the British Educational Research Association, held 12–14 September 2002, at the University of Exeter, UK. www.leeds.ac.uk/educol/documents/00002182.htm (accessed 11 March 2013). Google Scholar
• Cohen J ( 1973 ). Eta-squared and partial eta-squared in fixed factor ANOVA designs . Educ Psychol Meas 33 , 107-112. Google Scholar
• Cohen J ( 1988 ). Statistical Power Analysis for the Behavioral Sciences, 2nd ed. In: Hillsdale, NJ: Lawrence Erlbaum. Google Scholar
• Cohen J ( 1992 ). A power primer . Psychol Bull 112 , 155-159. Medline ,  Google Scholar
• Cohen J ( 1994 ). The earth is round ( p < .05) . Am Psychol 49 , 997-1003. Google Scholar
• Ellis PD ( 2010 ). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results , Cambridge, UK: Cambridge University Press. Google Scholar
• Fan X ( 2001 ). Statistical significance and effect size in education research: two sides of a coin . J Educ Res 94 , 275-282. Google Scholar
• Ferguson CJ ( 2009 ). An effect size primer: a guide for clinicians and researchers . Prof Psychol Res Pract 40 , 532-538. Google Scholar
• Glass GV, McGaw B, Smith M ( 1981 ). Meta-Analysis in Social Research , Beverly Hills, CA: Sage. Google Scholar
• Gottesman AJ, Hoskins SG ( 2013 ). CREATE Cornerstone: Introduction to Scientific Thinking, a new course for STEM-interested freshmen, demystifies scientific thinking through analysis of scientific literature . CBE Life Sci Educ 12 , 59-72. Link ,  Google Scholar
• Granger EM, Bevis TH, Saka Y, Southerland SA, Sampson V, Tate RL ( 2012 ). The efficacy of student-centered instruction in supporting science learning . Science 338 , 105-108. Medline ,  Google Scholar
• Hedges LV ( 1981 ). Distribution theory for Glass's estimator of effect size and related estimators . J Educ Stat 6 , 106-128. Google Scholar
• Hojat M, Xu G ( 2004 ). A visitor's guide to effect sizes—statistical significance versus practical (clinical) importance of research findings . Adv Health Sci Educ 9 , 241-249. Medline ,  Google Scholar
• Huberty CJ ( 2002 ). A history of effect size indices . Educ Psychol Meas 62 , 227-240. Google Scholar
• Johnson DH ( 1999 ). The insignificance of statistical significance testing . J Wildlife Manag 63 , 763-772. Google Scholar
• Johnsson JI ( 1993 ). Big and brave: size selection affects foraging under risk of predation in juvenile rainbow trout, Oncorhynchus mykiss . Anim Behav 45 , 1219-1225. Google Scholar
• Kerlinger FH ( 1964 ). Foundations of Behavioral Research , New York: Holt, Rinehart and Winston. Google Scholar
• Kirk RE ( 1996 ). Practical significance: a concept whose time has come . Educ Psychol Meas 56 , 746-759. Google Scholar
• Kotrlik JW, Williams HA ( 2003 ). The incorporation of effect size in information technology, learning, and performance research . Inform Technol Learn Perform J 21 , 1-7. Google Scholar
• Lauer S, Momsen J, Offerdahl E, Kryjevskaia M, Christensen W, Montplaisir L ( 2013 ). Stereotyped: investigating gender in introductory science courses . CBE Life Sci Educ 12 , 30-38. Link ,  Google Scholar
• Levine TR, Hullett CR ( 2002 ). Eta squared, partial eta squared, and misreporting of effect size in communication research . Hum Commun Res 28 , 612-625. Google Scholar
• Matthews KE, Adams P, Goos M ( 2010 ). Using the principles of BIO2010 to develop an introductory, interdisciplinary course for biology students . CBE Life Sci Educ 9 , 290-297. Link ,  Google Scholar
• McGraw KO, Wong SP ( 1992 ). A common language effect size statistic . Psychol Bull 111 , 361-365. Google Scholar
• Nakagawa S, Cuthill IC ( 2007 ). Effect size, confidence interval and statistical significance: a practical guide for biologists . Biol Rev Camb Philos Soc 82 , 591-605. Medline ,  Google Scholar
• Nickerson RS ( 2000 ). Null hypothesis significance testing: a review of an old and continuing controversy . Psychol Methods 5 , 241. Medline ,  Google Scholar
• Osbourne J ( 2008 ). Sweating the small stuff in educational psychology: how effect size and power reporting failed to change from 1969 to 1999, and what that means for the future of changing practices . Educ Psychol 28 , 151-160. Google Scholar
• Quitadamo IJ, Kurtz MJ ( 2007 ). Learning to improve: using writing to increase critical thinking performance in general education biology . CBE Life Sci Educ 6 , 140-154. Link ,  Google Scholar
• Rosenthal JA ( 1996 ). Qualitative descriptors of strength of association and effect size . J Social Serv Res 21 , 37-59. Google Scholar
• Rosenthal R, Rubin DB ( 2003 ). r-equivalent: a simple effect size indicator . Psychol Methods 8 , 492-496. Medline ,  Google Scholar
• Royer J ( 2000 ). A policy on reporting of effect sizes . Contemp Educ Psychol 25 , 239. Medline ,  Google Scholar
• Thomas L, Juanes F ( 1996 ). The importance of statistical power analysis: an example from animal behaviour . Anim Behav 52 , 856-859. Google Scholar
• Thompson B ( 1993 ). The use of statistical significance tests in research: bootstrap and other alternatives . J Exp Educ 61 , 361-377. Google Scholar
• Thompson B ( 1996 ). AERA editorial policies regarding statistical significance testing: three suggested reforms . Educ Res 25 , 26-30. Google Scholar
• Tressoldi PE, Giofré D, Sella F, Cumming G ( 2013 ). High impact = high statistical standards? Not necessarily so . PLoS One 8 , e56180. Medline ,  Google Scholar
• Vaske JJ ( 2002 ). Communicating judgments about practical significance: effect size, confidence intervals and odds ratios . Human Dimens Wildl 7 , 287-300. Google Scholar
• Wilkinson L, American Psychological Association Task Force on Statistical Inference ( 1999 ). Statistical methods in psychology journals: guidelines and explanations . Am Psychol 54 , 594-604. Google Scholar
• Williams HA ( 2003 ). A mediated hierarchical regression analysis of factors related to research productivity of human resource development postsecondary faculty. Doctoral Dissertation , Baton Rouge: Louisiana State University, http://etd.lsu.edu/docs/available/etd-0326103-212409 (accessed 22 March 2013). Google Scholar
• Mediating effect of psychological capital on the relationship between mental health literacy and coping styles among newly recruited nurses 14 March 2024 | BMC Nursing, Vol. 23, No. 1
• Household food insecurity among child and adolescent athletics practitioners: A cross-sectional, descriptive, and exploratory study Nutrition, Vol. 123
• Sagittal Full-Spine vs. Sectional Cervical Lateral Radiographs: Are the Measurements of Cervical Alignment Interchangeable? 24 April 2024 | Journal of Clinical Medicine, Vol. 13, No. 9
• Defense styles and comorbid psychiatric symptoms among convicts, Pakistan 20 January 2024 | Current Psychology, Vol. 43, No. 20
• Acute isometric and dynamic exercise do not alter cerebral sympathetic nerve activity in healthy humans 13 April 2024 | Journal of Cerebral Blood Flow & Metabolism, Vol. 96
• Peer Mentoring Develops Undergraduate Career Goals in a Community of Research Teams 10 April 2024 | The Journal of Experimental Education, Vol. 42
• Illusions of online readiness: the counter-intuitive impact of rapid immersion in digital learning due to COVID-19 17 April 2022 | Irish Educational Studies, Vol. 43, No. 2
• Sports participation plays a relevant role in the relationship between birth weight and bone mineral content in adolescents 1 April 2024 | American Journal of Human Biology, Vol. 20
• Engineered algal systems for the treatment of anaerobic digestate: A meta-analysis Journal of Environmental Management, Vol. 356
• Predictive tools for university performance: an explorative study 31 January 2022 | International Journal of Mathematical Education in Science and Technology, Vol. 55, No. 3
• The Effects of Core Stabilization Exercises on Respiratory Muscle Strength, Respiratory Functions, and Postural Control in Children With Hearing Loss: A Randomized Controlled Trial 3 October 2023 | Ear & Hearing, Vol. 45, No. 2
• Effect of extracorporeal shockwave therapy on adhesive capsulitis in patients with type 2 diabetes mellitus: A systematic review 8 July 2023 | International Journal of Diabetes in Developing Countries, Vol. 44, No. 1
• Does Regular Physical Activity Protect Sleep and Mental Health of University Students: A Systematic Review 23 February 2024 | Sleep and Vigilance, Vol. 29
• Real World Interstitial Glucose Profiles of a Large Cohort of Physically Active Men and Women 24 January 2024 | Sensors, Vol. 24, No. 3
• Integrating Science Concepts Through Poetry: A Study Comparing Online and Face-to-Face Teaching Methods Courses for Pre-Service Elementary Educators 6 September 2023 | Journal of Science Education and Technology, Vol. 33, No. 1
• Pflegebedürftigkeit im Alter: Angehörigen- und Versorgungssituation im Zusammenhang mit den Pflegegraden bei häuslicher Pflege 23 February 2024 | Das Gesundheitswesen, Vol. 86, No. S 01
• Counting Nemo: anemonefish Amphiprion ocellaris identify species by number of white bars 1 February 2024 | Journal of Experimental Biology, Vol. 227, No. 2
• Fascia Tissue Manipulations in Chronic Low Back Pain: A Pragmatic Comparative Randomized Clinical Trial of the 4xT Method® and Exercise Therapy 20 December 2023 | Life, Vol. 14, No. 1
• Selective short-term memory impairment for verbalizable visual objects in children with Developmental Language Disorder Research in Developmental Disabilities, Vol. 144
• Parental Beliefs About the Motor Development of Dutch Infants Born Very Preterm: A Cohort Study Pediatric Physical Therapy, Vol. 36, No. 1
• Brief rapport: Perceptual aberration in patients at ultra‐high risk for psychosis 31 May 2023 | Early Intervention in Psychiatry, Vol. 18, No. 1
• Symptom presentation and evolution in the first 48 hours after injury are associated with return to play after concussion in elite Rugby Union Journal of Sport and Health Science, Vol. 55
• Continuing Medical Education Outcomes are Much More Than Statistical Significance 21 July 2023 | Journal of CME, Vol. 12, No. 1
• Assessment of bruxism and temporomandibular disorder in mothers of children with cerebral palsy 31 December 2023 | Adıyaman Üniversitesi Sağlık Bilimleri Dergisi, Vol. 9, No. 3
• COMPARISON OF THE EFFECTS OF MULLIGAN MOBILIZATION TECHNIQUE COMBINED WITH CERVICAL STABILIZATION EXERCISES WITH THE EFFECTS OF CERVICAL STABILIZATION EXERCISES ALONE IN CHRONIC NECK PAIN: RANDOMIZED CONTROLLED STUDY 30 December 2023 | Karya Journal of Health Science, Vol. 4, No. 3
• Inhibition of focal adhesion kinase 2 results in a macrophage polarization shift to M2 which attenuates local and systemic inflammation and reduces heterotopic ossification after polysystem extremity trauma 5 December 2023 | Frontiers in Immunology, Vol. 14
• Impact of corticosteroid administration on the response of exposed dental pulp to capping with bioactive cements-experimental study on mongrel dogs 26 June 2023 | BMC Oral Health, Vol. 23, No. 1
• Systemic Racism and Implicit Bias Influencing US School Counseling 9 September 2023 | International Journal for the Advancement of Counselling, Vol. 45, No. 4
• Effects of developmental relationships on the well‐being of youth in high‐stress families 21 January 2023 | Family Relations, Vol. 72, No. 5
• Students' Perception of Servant Leadership by Physical Therapy Faculty Mentors Is Associated With Interprofessional Socialization 30 August 2023 | Journal of Physical Therapy Education, Vol. 37, No. 4
• Phosphodiesterase 4 inhibition after retrieval switches the memory fate favoring extinction instead of reconsolidation 21 November 2023 | Scientific Reports, Vol. 13, No. 1
• Walking into a Modern Painting: The Impacts of Using Virtual Reality on Student Learning Performance and Experiences in Art Appreciation 23 November 2023 | International Journal of Human–Computer Interaction, Vol. 41
• Effect of changing threat conditions on police and military commanders’ preferences for urgent and offensive actions: An analysis of decision making at the operational level of war 3 November 2023 | Military Psychology, Vol. 77
• Safety of exercise training in multiple sclerosis: An updated systematic review and meta-analysis 26 October 2023 | Multiple Sclerosis Journal, Vol. 29, No. 13
• I know It's false, but I keep thinking as if it were true: A replication study of Johnson and Seifert's (1994) continued influence effect Acta Psychologica, Vol. 241
• Impact of Modality on Workload among Engineering Undergraduates during the COVID-19 Pandemic Journal of Civil Engineering Education, Vol. 149, No. 4
• Increasing representation and diversity in health research: A protocol of the MYHealth research training program for high school students 28 September 2023 | PLOS ONE, Vol. 18, No. 9
• Exploratory Mixed-Methods Study on PLTL Leaders’ Perceptions of their Group-Inclusion Skills in General Chemistry 30 August 2023 | Journal of Chemical Education, Vol. 100, No. 9
• Influence of physical fitness of coronary disease patients on vagal reentry and heart rate recovery after exercise with and without fluid replacement 2 May 2023 | Clinical Physiology and Functional Imaging, Vol. 43, No. 5
• Einfluss einer 3D-gedruckten bionischen Fußorthese auf die Rückfußbelastung und Laufökonomie Fuß & Sprunggelenk, Vol. 21, No. 3
• Individual Dimensions of Pain Catastrophizing Do Not Mediate the Effect of Sociodemographic and Psychological Factors on Chronic Orofacial Pain Severity, Interference, and Jaw Limitation: A Structural Equation Modeling Approach The Journal of Pain, Vol. 24, No. 9
• Informing Users: Effects of Notification Properties and User Characteristics on Sharing Attitudes 27 June 2022 | International Journal of Human–Computer Interaction, Vol. 39, No. 14
• Cerebral blood velocity during concurrent supine cycling, lower body negative pressure, and head-up tilt challenges: implications for concussion rehabilitation 18 August 2023 | Physiological Measurement, Vol. 44, No. 8
• Measurement of transdermal alcohol concentration using a wrist-worn enzymatic transdermal monitor Alcohol, Vol. 110
• Students in Correctional Education: Developmental Education’s Forgotten Population 1 May 2023 | Journal of College Reading and Learning, Vol. 53, No. 3
• COVID-19 and postural sway: a comparison of individuals with a SARS-CoV-2 history and healthy sedentary women 14 February 2023 | Somatosensory & Motor Research, Vol. 40, No. 3
• Estimation of the size of the COVID-19 pandemic using the epidemiological wavelength model: results from OECD countries Public Health, Vol. 220
• Catalyzing Collaboration Among Interprofessional Birth Transfer Teams Through Simulation 28 April 2023 | Journal of Midwifery & Women's Health, Vol. 68, No. 4
• Water drinking during aerobic exercise improves the recovery of non-linear heart rate dynamics in coronary artery disease: crossover clinical trial 22 June 2023 | Frontiers in Neuroscience, Vol. 17
• Association entre la perte de force des fléchisseurs du genou et les douleurs postéro-médiales du genou opéré d’une reconstruction du ligament croisé antérieur avec une technique utilisant les ischio-jambiers : mythe ou réalité ? Journal de Traumatologie du Sport, Vol. 40, No. 2
• Safety and Feasibility of Cervical and Thoracic Transcutaneous Spinal Cord Stimulation to Improve Hand Motor Function in Children With Chronic Spinal Cord Injury Neuromodulation: Technology at the Neural Interface, Vol. 30
• Effects of speech intelligibility and reverberation time on the serial recall task in Chinese open-plan offices: A laboratory study Applied Acoustics, Vol. 208
• The myth and reality of familial resemblance in dietary intake: a systematic review and meta-analysis on the resemblance of dietary intake among parent and offspring eClinicalMedicine, Vol. 60
• Rehabilitation interventions after hand burn injury in adults: A systematic review Burns, Vol. 49, No. 3
• Associations of micronutrients exposure with cadmium body burden among population: A systematic review Ecotoxicology and Environmental Safety, Vol. 256
• Measuring alcohol consumption while watching sport events: a feasibility and validity study comparing ecological momentary assessments and transdermal alcohol monitors 20 April 2023 | Journal of Substance Use, Vol. 6
• Semiochemicals produced by fungal bark beetle symbiont Endoconidiophora rufipennis and the discovery of an anti-attractant for Ips typographus 6 April 2023 | PLOS ONE, Vol. 18, No. 4
• Reduction of scratch or dirt artifacts on intraoral radiographs using dual imaging plates in image processing 31 August 2022 | Oral Radiology, Vol. 39, No. 2
• Acute effect of Δ-9-tetrahydrocannabinol on neuromuscular transmission and locomotive behaviors in larval zebrafish Journal of Neurophysiology, Vol. 129, No. 4
• Case-based teaching of experimental design – contributions for meaningful learning 4 February 2023 | Laboratory Animals, Vol. 57, No. 2
• The association between sensory processing sensitivity, the five-factor model and university adjustment amongst South African university students 26 July 2021 | Current Psychology, Vol. 42, No. 10
• Stressors in health care and their association to symptoms experienced by gender diverse people Public Health, Vol. 217
• Examining middle school students’ attitudes toward computing after participating in a physical computing unit 28 March 2023 | Interactive Learning Environments, Vol. 5
• Association of egg consumption with colorectal polyp prevalence: findings from the Lanxi Pre-Colorectal Cancer Cohort (LP3C) in China 1 January 2023 | Food & Function, Vol. 14, No. 6
• A case study on genomic imprinting facilitates student learning Advances in Physiology Education, Vol. 47, No. 1
• Supporting self-regulated learning with learning analytics interventions – a systematic literature review 8 September 2022 | Education and Information Technologies, Vol. 28, No. 3
• Foliar Fungicides Containing FRAC 11 Mitigate Phomopsis Stem Canker in Sunflower ( Helianthus annuus ) Plant Disease, Vol. 107, No. 3
• Metabolic Syndrome and Nephrolithiasis; A Cross Sectional Population-based Study on the Baseline Data of the PERSIAN Kavar Cohort Study Urology, Vol. 173
• Effects of delta-9 tetrahydrocannabinol on fear memory labilization and reconsolidation: A putative role of GluN2B-NMDA receptor within the dorsal hippocampus Neuropharmacology, Vol. 225
• Understanding the role of physical activity on the pathway from intra-articular knee injury to post-traumatic osteoarthritis disease in young people: a scoping review protocol 3 March 2023 | BMJ Open, Vol. 13, No. 3
• Effects of Different Exercise Types on Chrna7 and Chrfam7a Expression in Healthy Normal Weight and Overweight Type 2 Diabetic Adults 15 February 2023 | Biomedicines, Vol. 11, No. 2
• Nitrate and sodium nitroprusside alter the development of Asian black-spined toads’ embryos by inducing nitric oxide production 1 November 2022 | Environmental Science and Pollution Research, Vol. 30, No. 9
• Measuring palliative care self-efficacy of intellectual and developmental disability staff using Rasch models 23 January 2023 | Palliative and Supportive Care, Vol. 3
• Consequences of anterior knee pain after anterior cruciate ligament reconstruction: A 2015–2020 cohort study 5 January 2023 | PLOS ONE, Vol. 18, No. 1
• Perception of rhythmic agency for conversational labeling 9 February 2021 | Human–Computer Interaction, Vol. 38, No. 1
• Validating the modified Rivermead Post-concussion Symptoms Questionnaire (mRPQ) 4 August 2021 | The Clinical Neuropsychologist, Vol. 37, No. 1
• Perceived Effectiveness and Responsibilities of the Forest Biological Disasters Control System of China: A Perspective of Government Administrators 20 December 2022 | Forests, Vol. 14, No. 1
• Impact of Moderate-to-Vigorous Sports Participation Combined with Resistance Training on Metabolic and Cardiovascular Outcomes among Lean Adolescents: ABCD Growth Study 27 December 2022 | International Journal of Environmental Research and Public Health, Vol. 20, No. 1
• Investigating Socio-scientific Reasoning of Biology Prospective Teacher: Focus on Genetic Modified Organism (GMO) 22 March 2023
• Application of Pyraclostrobin Fungicide at Miniature Floral Head Development Growth Stage Is Efficacious Against Phomopsis Stem Canker Fungi in Sunflower ( Helianthus annuus ) Plant Health Progress, Vol. 24, No. 1
• Predictors of Cyberchondria During the COVID-19 Pandemic: Cross-sectional Study Using Supervised Machine Learning 25 April 2023 | JMIR Formative Research, Vol. 7
• Elevated hippocampal copper in cases of type 2 diabetes eBioMedicine, Vol. 86
• Molecular mechanisms of exceptional lifespan increase of Drosophila melanogaster with different genotypes after combinations of pro-longevity interventions 9 June 2022 | Communications Biology, Vol. 5, No. 1
• Who gets access to an interprofessional team-based primary care programme for patients with complex health and social needs? A cross-sectional analysis 14 December 2022 | BMJ Open, Vol. 12, No. 12
• Comparison of bone mineral density according to domains of sedentary behavior in children and adolescents 1 February 2022 | BMC Pediatrics, Vol. 22, No. 1
• Relationship between vigorous physical activity and health care costs among adolescents: ABCD Growth Study 17 March 2022 | BMC Pediatrics, Vol. 22, No. 1
• Development of a simple and effective online training for health workers: results from a pilot in Nigeria 21 March 2022 | BMC Public Health, Vol. 22, No. 1
• A parallel classification strategy to simultaneous control elbow, wrist, and hand movements 28 January 2022 | Journal of NeuroEngineering and Rehabilitation, Vol. 19, No. 1
• A case study of a novel summer bridge program to prepare transfer students for research in biological sciences 20 December 2022 | Disciplinary and Interdisciplinary Science Education Research, Vol. 4, No. 1
• Testing the Efficiency of the Push-and-Pull Strategy during Severe Ips typographus Outbreak and Extreme Drought in Norway Spruce Stands 18 December 2022 | Forests, Vol. 13, No. 12
• Assessing the Impacts of Creating Active Schools on Organisational Culture for Physical Activity 16 December 2022 | International Journal of Environmental Research and Public Health, Vol. 19, No. 24
• Relating students’ social belonging and course performance across multiple assessment types in a calculus-based introductory physics 1 course 29 December 2022 | Physical Review Physics Education Research, Vol. 18, No. 2
• Climate change and the pricing of sovereign debt: Insights from European markets Research in International Business and Finance, Vol. 62
• A screen-based or physical computing unit? Examining secondary students’ attitudes toward coding International Journal of Child-Computer Interaction, Vol. 34
• Skin Displacement as fascia tissue manipulation at the lower back affects instantaneously the flexion-and extension spine, pelvis, and hip range of motion 23 November 2022 | Frontiers in Physiology, Vol. 13
• Perceived Teacher Discrimination and Academic Achievement Among Urban Caribbean Black and African American Youth: School Bonding and Family Support as Protective Factors 23 October 2018 | Urban Education, Vol. 57, No. 9
• Assessing how students value learning communication skills in an undergraduate anatomy and physiology course 3 December 2021 | Anatomical Sciences Education, Vol. 15, No. 6
• Peripheral Skeletal Muscle Impairment in Children After Treatment for Leukemia and Lymphoma 28 January 2022 | Journal of Pediatric Hematology/Oncology, Vol. 44, No. 8
• What’s up with WhatsApp? A Critical Analysis of Mobile Instant Messaging Research in Language Learning 30 October 2022 | International Journal of Contemporary Educational Research, Vol. 6, No. 2
• Litter quality and decomposition responses to drought in a northeastern US deciduous forest 21 September 2022 | Oecologia, Vol. 200, No. 1-2
• Abyssal epibenthic holothurians respond differently to food quantity and concentration fluctuations over a decade of daily observation (2007–2017) Deep Sea Research Part I: Oceanographic Research Papers, Vol. 188
• Effects of Meaningful Action Observation Therapy on Occupational Performance, Upper Limb Function, and Corticospinal Excitability Poststroke: A Double-Blind Randomized Control Trial Neural Plasticity, Vol. 2022
• Multifidus muscle cross-sectional area adaptations over two volleyball seasons and one off-season in athletes with and without low back pain Journal of Back and Musculoskeletal Rehabilitation, Vol. 35, No. 5
• Impact of school meals on educational outcomes in Addis Ababa, Ethiopia 28 March 2022 | Public Health Nutrition, Vol. 25, No. 9
• Association of Preexisting Mental Health Conditions With Increased Initial Symptom Count and Severity Score on SCAT5 When Assessing Concussion 19 September 2022 | Orthopaedic Journal of Sports Medicine, Vol. 10, No. 9
• A laboratory study correlating serial recall performance and speech intelligibility of Chinese language in open-plan offices Building and Environment, Vol. 223
• Parent-Mediated Interventions for Infants under 24 Months at Risk for Autism Spectrum Disorder: A Systematic Review of Randomized Controlled Trials 8 July 2021 | Journal of Autism and Developmental Disorders, Vol. 52, No. 6
• The Impact of an Internet Use Promotion Programme on Communication, Internet Use, and the Extent of Social Networks among Low-Income Older Adults 21 July 2021 | Ageing International, Vol. 47, No. 2
• Does length of time since diagnosis in Parkinson’s disease influence heart rate variability? A cross-sectional study 1 June 2022 | Arquivos de Neuro-Psiquiatria, Vol. 80, No. 6
• Masseter Muscle Thickness And Elasticity in Bruxism After Exercise Treatment: A Comparison Trial Journal of Manipulative and Physiological Therapeutics, Vol. 45, No. 4
• Anti‐attractant activity of (+)‐ trans ‐4‐thujanol for Eurasian spruce bark beetle Ips typographus : Novel potency for females 14 February 2022 | Pest Management Science, Vol. 78, No. 5
• Real‐Life Turning Movements Capture Subtle Longitudinal and Preataxic Changes in Cerebellar Ataxia 24 January 2022 | Movement Disorders, Vol. 37, No. 5
• The rise to power of the microbiome: power and sample size calculation for microbiome studies Mucosal Immunology, Vol. 15, No. 6
• Comparison of the sit-up test and head-up tilt test for assessing blood pressure and hemodynamic responses in healthy young individuals 7 October 2021 | Blood Pressure Monitoring, Vol. 27, No. 2
• Reflective journals in music production pedagogy: A case study Journal of Music, Technology & Education, Vol. 15, No. 1
• ErbB4 alternative splicing mediates fetal mouse alveolar type II cell differentiation in vitro 25 March 2022 | Pediatric Research, Vol. 7
• Mediating Effects of Coping Strategies on the Relationship Between Mental Health and Quality of Life Among Indonesian Female Migrant Workers in Taiwan 16 November 2021 | Journal of Transcultural Nursing, Vol. 33, No. 2
• Construction and Calibration of an Exposure Matrix for the Welding Trades 28 August 2021 | Annals of Work Exposures and Health, Vol. 66, No. 2
• Influence of selective immunosuppressive drug regimens on the healing of exposed dogs’ dental pulp capped with a recent calcium silicate-based cement 9 August 2021 | Clinical Oral Investigations, Vol. 26, No. 2
• Australasian Emergency Care, Vol. 25, No. 2
• Weak DCS causes a relatively strong cumulative boost of synaptic plasticity with spaced learning Brain Stimulation, Vol. 15, No. 1
• Using Distributed Practice to Improve Students’ Attitudes and Performance in Statistics 17 December 2020 | Teaching of Psychology, Vol. 49, No. 1
• Journal of Intensive Care Medicine, Vol. 37, No. 8
• Revista Latino-Americana de Enfermagem, Vol. 30
• Frontiers in Human Neuroscience, Vol. 16
• Applied Sciences, Vol. 12, No. 14
• Dogs Trained to Recognise a Bark Beetle Pheromone Locate Recently Attacked Spruces Faster than Human Experts SSRN Electronic Journal, Vol. 82
• The impact of aging on interhemispheric transfer time and respective sex differences Aging Brain, Vol. 2
• Preparing for Class: Actions and Resources of Introductory Biology Students Journal of Microbiology & Biology Education, Vol. 22, No. 3
• Intrarater reliability of the Abilitator—a self-report questionnaire on work ability and functioning aimed at the population in a weak labour market position: a test–retest study 8 December 2021 | BMJ Open, Vol. 11, No. 12
• Resistance-induced brain activity changes during cycle ergometer exercises 19 March 2021 | BMC Sports Science, Medicine and Rehabilitation, Vol. 13, No. 1
• Towards robust and replicable sex differences in the intrinsic brain function of autism 1 March 2021 | Molecular Autism, Vol. 12, No. 1
• Safety of exercise training in multiple sclerosis: a protocol for an updated systematic review and meta-analysis 20 July 2021 | Systematic Reviews, Vol. 10, No. 1
• Amateur ballet practicing, body image and eating behaviors: a comparative study of classical ballet dancers, gym users and sedentary women 28 August 2021 | Journal of Eating Disorders, Vol. 9, No. 1
• Novel quantification of the regional strain distribution in the anterior cruciate ligament in response to simulated loading using micro-CT imaging 22 October 2021 | Journal of Experimental Orthopaedics, Vol. 8, No. 1
• Vagal reactivation after a cardiac rehabilitation session associated with hydration in coronary artery disease patients: crossover clinical trial 18 May 2021 | Scientific Reports, Vol. 11, No. 1
• The contribution of group prenatal care to maternal psychological health outcomes: A systematic review Women and Birth, Vol. 34, No. 6
• Association between gait speed and the SHARE Frailty Instrument in a Falls and Syncope Clinic 15 May 2021 | European Geriatric Medicine, Vol. 12, No. 5
• The Effect of a Web-Based Deep Breathing App on Stress of Direct Care Workers: Uncontrolled Intervention Study The Journal of Alternative and Complementary Medicine, Vol. 27, No. 10
• Analytical investigation of career variables as correlate work stress of electrical/electronic trade teachers in technical colleges 5 March 2020 | The International Journal of Electrical Engineering & Education, Vol. 58, No. 4
• Effect of Online Palliative Care Training on Knowledge and Self-Efficacy of Direct Care Workers 22 September 2021 | Intellectual and Developmental Disabilities, Vol. 59, No. 5
• Large-scale comparison of biomass and reproductive phenology among native and non-native populations of the seagrass Zostera japonica Marine Ecology Progress Series, Vol. 675
• Internet-based cognitive behavioral therapy in children and adolescents with obsessive compulsive disorder: a feasibility study 25 August 2021 | Journal of Neural Transmission, Vol. 128, No. 9
• A Single Bout of Constant-Load Exercise Test for Estimating the Time Constant of Oxygen Uptake Kinetics in Individuals With Stroke Annals of Rehabilitation Medicine, Vol. 45, No. 4
• Métodos estadísticos para el análisis del tamaño del efecto 22 August 2021 | Revista Alergia México, Vol. 68, No. 2
• The Sputum Microbiome in Pulmonary Tuberculosis and Its Association With Disease Manifestations: A Cross-Sectional Study 20 August 2021 | Frontiers in Microbiology, Vol. 12
• The relationship between flourishing, injury status, and resilience in collegiate athletes 22 February 2021 | International Journal of Sports Science & Coaching, Vol. 16, No. 4
• The efficacy of the Senior South African Individual Scale Revised in distinguishing between attention deficit hyperactivity disorder, normal and sluggish cognitive tempo children 29 July 2021 | African Journal of Psychological Assessment, Vol. 3, No. 0
• Risk Factors and Treatment Needs of Batterer Intervention Program Participants with Substance Abuse Problems The European Journal of Psychology Applied to Legal Context, Vol. 13, No. 2
• Student Outcomes From a Large-Enrollment Introductory Course-Based Undergraduate Research Experience on Soil Microbiomes 27 July 2021 | Frontiers in Microbiology, Vol. 12
• Improving Learning Outcomes and Metacognitive Monitoring: Replacing Traditional Textbook Readings with Question-Embedded Videos 9 June 2021 | Journal of Chemical Education, Vol. 98, No. 7
• A Comparative Efficacy Study of Robotic Priming of Bilateral Approach in Stroke Rehabilitation 12 July 2021 | Frontiers in Neurology, Vol. 12
• SYMPERHEART: an intervention to support symptom perception in persons with heart failure and their informal caregiver: a feasibility quasi-experimental study protocol 27 July 2021 | BMJ Open, Vol. 11, No. 7
• Understanding the Link between Computer Science Instruction and Reading & Math Performance 26 June 2021
• Evaluating the impact of a classroom simulator training on graduate teaching assistants’ instructional practices and undergraduate student learning 29 June 2021 | Physical Review Physics Education Research, Vol. 17, No. 1
• Mass spectrometry‐based quantification and spatial localization of small organic acid exudates in plant roots under phosphorus deficiency and aluminum toxicity 16 May 2021 | The Plant Journal, Vol. 106, No. 6
• An Exploration of Three New Methods to Facilitate the Calculation of the Significance Levels of Periodogram Peaks by Simulation 26 May 2021 | The Astronomical Journal, Vol. 161, No. 6
• The effectiveness of two different exercise approaches in adolescent idiopathic scoliosis: A single-blind, randomized-controlled trial 15 April 2021 | PLOS ONE, Vol. 16, No. 4
• Cardiorespiratory mechanisms underlying the impaired oxygen uptake kinetics at exercise onset after stroke Annals of Physical and Rehabilitation Medicine, Vol. 64, No. 2
• Temporal distribution modelling reveals upstream habitat drying and downstream non‐native introgression are squeezing out an imperiled headwater fish 13 January 2021 | Diversity and Distributions, Vol. 27, No. 3
• Parkinson's disease effect on autonomic modulation: an analysis using geometric indices 1 February 2021 | Arquivos de Neuro-Psiquiatria, Vol. 79, No. 2
• The Effects of Spirituality and Moral Intensity on Ethical Business Decisions: A Cross-Sectional Study 2 August 2019 | Journal of Business Ethics, Vol. 168, No. 1
• Biomechanical evaluation of a hybrid suture and anchor-based hip capsular repair Clinical Biomechanics, Vol. 81
• Ecosystem structure and resilience of the Florida Bay Estuary: an original ecosystem model with implications for everglades restoration Marine and Freshwater Research, Vol. 72, No. 4
• Classes with selective intake in Finnish Comprehensive School: A Problem of Societally Equal Opportunity for Schooling or a Boost for Learning and Equity in Pedagogical arrangements? International Journal of Educational Research, Vol. 109
• Student perceptions towards introductory lessons in R 27 October 2021 | Natural Sciences Education, Vol. 50, No. 2
• Die Artenkenntnis von Berliner Schüler_innen am Beispiel einheimischer Vögel 11 September 2020 | Zeitschrift für Didaktik der Naturwissenschaften, Vol. 26, No. 1
• Computational data mining method for isotopomer analysis in the quantitative assessment of metabolic reprogramming 14 January 2020 | Scientific Reports, Vol. 10, No. 1
• The growing impact of older patients in the emergency department: a 5-year retrospective analysis in Brazil 11 June 2020 | BMC Emergency Medicine, Vol. 20, No. 1
• Characterizing science graduate teaching assistants’ instructional practices in reformed laboratories and tutorials 29 June 2020 | International Journal of STEM Education, Vol. 7, No. 1
• Reference intervals for parameters of health of eastern grey kangaroos Macropus giganteus and management implications across their geographic range 8 July 2020 | Wildlife Biology, Vol. 2020, No. 3
• A Biomechanical Comparison of 2 Hip Capsular Reconstruction Techniques: Iliotibial Band Autograft Versus Achilles Tendon Allograft 12 October 2020 | The American Journal of Sports Medicine, Vol. 48, No. 13
• Association of frailty with mortality in older inpatients with Covid-19: a cohort study 10 August 2020 | Age and Ageing, Vol. 49, No. 6
• Are signs and symptoms in cardiovascular rehabilitation correlated with heart rate variability? An observational longitudinal study 4 September 2020 | Geriatrics & Gerontology International, Vol. 20, No. 10
• A non-parametric effect-size measure capturing changes in central tendency and data distribution shape 24 September 2020 | PLOS ONE, Vol. 15, No. 9
• Frailty and incident depressive symptoms in a Chinese sample: the Rugao Longevity and Ageing Study 18 June 2020 | Psychogeriatrics, Vol. 20, No. 5
• Sports participation improves metabolic profile in adolescents: ABCD growth study 28 December 2019 | American Journal of Human Biology, Vol. 32, No. 5
• Real-life gait assessment in degenerative cerebellar ataxia Neurology, Vol. 95, No. 9
• The Impact of Nutrition-Specific and Nutrition-Sensitive Interventions on Hemoglobin Concentrations and Anemia: A Meta-review of Systematic Reviews 26 August 2020 | Advances in Nutrition, Vol. 1
• Enhancing Cellulase Production in Aureobasidium pullulans by Genome Shuffling Industrial Biotechnology, Vol. 16, No. 4
• Physical Activity, Physical Fitness and the Sense of Coherence—Their Role in Body Acceptance among Polish Adolescents 10 August 2020 | International Journal of Environmental Research and Public Health, Vol. 17, No. 16
• Mindfulness meditation is related to sensory‐affective uncoupling of pain in trained novice and expert practitioners 7 May 2020 | European Journal of Pain, Vol. 24, No. 7
• Early-career faculty practice learner-centered teaching up to 9 years after postdoctoral professional development Science Advances, Vol. 6, No. 25
• Kinematic integration during prey capture varies among individuals but not ecological contexts in bluegill sunfish, Lepomis macrochirus (Perciformes: Centrarchidae) 30 March 2020 | Biological Journal of the Linnean Society, Vol. 130, No. 1
• Early Vocabulary Profiles of Young Deaf Children Who Use Cochlear Implants Journal of Speech, Language, and Hearing Research, Vol. 63, No. 4
• Gallery Game: Smartphone-based assessment of long-term memory in adults at risk of Alzheimer’s disease 24 January 2020 | Journal of Clinical and Experimental Neuropsychology, Vol. 42, No. 4
• Framework for the natures of negativity in introductory physics 22 April 2020 | Physical Review Physics Education Research, Vol. 16, No. 1
• Neuroplasticity of Cortical Planning for Initiating Stepping Poststroke: A Case Series Journal of Neurologic Physical Therapy, Vol. 44, No. 2
• The Correlative Strength of Objective Physical Assessment Against the ECOG Performance Status Assessment in Individuals Diagnosed With Cancer 10 February 2020 | Physical Therapy, Vol. 100, No. 3
• Brie Tripp ,
• Sophia A. Voronoff , and
• Erin E. Shortlidge
• Tammy Long, Monitoring Editor
• Accumulation of Ascorbic Acid in Tomato Cell Culture: Influence of the Genotype, Source Explant and Time of In Vitro Cultivation 7 March 2020 | Antioxidants, Vol. 9, No. 3
• Career Impact of the Chief Resident in Quality and Safety Training Program: An Alumni Evaluation Academic Medicine, Vol. 95, No. 2
• BiochemAR: An Augmented Reality Educational Tool for Teaching Macromolecular Structure and Function 27 December 2019 | Journal of Chemical Education, Vol. 97, No. 1
• The Impact of Social Media Influencers on Children’s Dietary Behaviors 10 January 2020 | Frontiers in Psychology, Vol. 10
• The effect of methylphenidate on social cognition and oxytocin in children with attention deficit hyperactivity disorder 12 September 2019 | Neuropsychopharmacology, Vol. 45, No. 2
• A single bout of vigorous-intensity aerobic exercise affects reactive, but not proactive cognitive brain functions International Journal of Psychophysiology, Vol. 147
• A Modified CREATE Intervention Improves Student Cognitive and Affective Outcomes in an Upper-Division Genetics Course Journal of Microbiology & Biology Education, Vol. 21, No. 1
• The Meta-Analysis Review: A Valuable Resource for Special Educators 9 May 2019 | Intervention in School and Clinic, Vol. 55, No. 3
• COVID-19 AND DIGITAL TECHNOLOGY: MOBILE APPLICATIONS AVAILABLE FOR DOWNLOAD IN SMARTPHONES 1 January 2020 | Texto & Contexto - Enfermagem, Vol. 29
• 2D or 3D? How cell motility measurements are conserved across dimensions in vitro and translate in vivo 9 December 2019 | Bioengineering & Translational Medicine, Vol. 5, No. 1
• Report on a large animal study with Göttingen Minipigs where regenerates and controls for articular cartilage were created in a large number. Focus on the conditions of the operated stifle joints and suggestions for standardized procedures 26 December 2019 | PLOS ONE, Vol. 14, No. 12
• Tessa C. Andrews ,
• Anna Jo J. Auerbach , and
• Emily F. Grant
• Jennifer Knight, Monitoring Editor
• Land-Use Change Alters Host and Vector Communities and May Elevate Disease Risk 24 April 2018 | EcoHealth, Vol. 16, No. 4
• Metabolome and Microbiome Signatures in the Roots of Citrus Affected by Huanglongbing Phytopathology®, Vol. 109, No. 12
• HIV prevention intervention for substance users: a review of the literature 3 January 2019 | Substance Abuse Treatment, Prevention, and Policy, Vol. 14, No. 1
• Local authority instrumental music tuition as a form of neo-liberal parental investment: Findings from a deviant, idiographic case study 17 May 2019 | Power and Education, Vol. 11, No. 3
• A New Attitudinal Integral-Model to Explain Green Purchase Intention 8 November 2019 | Sustainability, Vol. 11, No. 22
• Cardiorespiratory factors related to the increase in oxygen consumption during exercise in individuals with stroke 9 October 2019 | PLOS ONE, Vol. 14, No. 10
• Exploratory comparison of auditory verbal hallucinations and other psychotic symptoms among youth with borderline personality disorder or schizophrenia spectrum disorder 28 November 2018 | Early Intervention in Psychiatry, Vol. 13, No. 5
• Impact of liberalisation on Indian life insurance industry: A truly multivariate approach IIMB Management Review, Vol. 31, No. 3
• The experience of flow during sensorimotor synchronization to musical rhythms 20 July 2019 | Musicae Scientiae, Vol. 23, No. 3
• Associations between adolescent media use, mental health, and risky sexual behaviors Children and Youth Services Review, Vol. 103
• Imputation Methods to Deal With Missing Responses in Computerized Adaptive Multistage Testing 29 October 2018 | Educational and Psychological Measurement, Vol. 79, No. 3
• Comparing Individual and Group-Negotiated Knowledge Structures in an Introductory Biology Course for Majors 16 May 2018 | Journal of Biological Education, Vol. 53, No. 3
• Preliminary Evidence for the Cognitive Model of Auditory Verbal Hallucinations in Youth With Borderline Personality Disorder 16 May 2019 | Frontiers in Psychiatry, Vol. 10
• Single finger movements in the aging hand: changes in finger independence, muscle activation patterns and tendon displacement in older adults 19 February 2019 | Experimental Brain Research, Vol. 237, No. 5
• Boys who work on the streets of Iraq are exposed to more traumatic events such as torture than schoolboys of the same age 10 September 2018 | Acta Paediatrica, Vol. 108, No. 4
• Stanley M. Lo ,
• Grant E. Gardner ,
• Joshua Reid ,
• Velta Napoleon-Fanis ,
• Penny Carroll ,
• Emily Smith , and
• Brian K. Sato
• David Feldon, Monitoring Editor
• Heart Failure Patients’ Perceptions of Received and Wanted Information: A Cross-Sectional Study 10 July 2018 | Clinical Nursing Research, Vol. 28, No. 3
• The efficacy of computerized cognitive drill and practice training for patients with a schizophrenia-spectrum disorder: A meta-analysis Schizophrenia Research, Vol. 204
• Stocking Density Affects Stress and Anxious Behavior in the Laying Hen Chick During Rearing 10 February 2019 | Animals, Vol. 9, No. 2
• Integration between swim speed and mouth size evolves repeatedly in Trinidadian guppies and aligns with suction-feeding fishes 16 January 2019 | Journal of Experimental Biology, Vol. 222, No. 2
• Mental health in familial caregivers of Alzheimer’s disease patients: are the effects of chronic stress on cognition inevitable? 1 November 2018 | Stress, Vol. 22, No. 1
• Insights and Teacher Perceptions Regarding Students’ Conceptions as They Enter Tertiary Chemistry Studies: a Comparative Study 22 September 2017 | International Journal of Science and Mathematics Education, Vol. 17, No. 1
• Proceedings of the National Academy of Sciences, Vol. 116, No. 7
• Évaluer le transfert des compétences infirmières : analyse des effets d’une formation en soins intermédiaires en milieu hospitalier 16 December 2020 | Mesure et évaluation en éducation, Vol. 42, No. 3
• Assessing the Transfer of Nursing Competencies: Analysis of the effects of intermediate care training in a hospital setting Mesure et évaluation en éducation, Vol. 42, No. spécial
• Measuring Effect Size: To Standardize on Not to Standardize? Comment on “Effects of Myofascial Release on Pressure Pain Thresholds in Patients With Neck Pain” American Journal of Physical Medicine & Rehabilitation, Vol. 98, No. 1
• Thushani Rodrigo-Peiris ,
• Lin Xiang , and
• Vincent M. Cassone
• Graham Hatfull, Monitoring Editor
• Comparison of proximal versus distal upper-limb robotic rehabilitation on motor performance after stroke: a cluster controlled trial 1 February 2018 | Scientific Reports, Vol. 8, No. 1
• Feasibility and assessment of outcome measures for yoga as self-care for minorities with arthritis: a pilot study 20 February 2018 | Pilot and Feasibility Studies, Vol. 4, No. 1
• Native rotational knee kinematics is restored after lateral UKA but not after medial UKA 26 March 2018 | Knee Surgery, Sports Traumatology, Arthroscopy, Vol. 26, No. 11
• Ambulation capacity and functional outcome in patients undergoing neuromuscular electrical stimulation after cardiac valve surgery Medicine, Vol. 97, No. 46
• Adapting to the Large-Scale Advanced Placement Chemistry Reform: An Examination of Teachers’ Challenges and Instructional Practices 19 July 2018 | Journal of Chemical Education, Vol. 95, No. 10
• Out-of-pocket payments in hospital bills: a challenge to management 1 October 2018 | Revista Brasileira de Enfermagem, Vol. 71, No. 5
• A Framework for Image-Based Modeling of Acute Myocardial Ischemia Using Intramurally Recorded Extracellular Potentials 21 May 2018 | Annals of Biomedical Engineering, Vol. 46, No. 9
• Preanalytical phase: Effects of water ingestion during fasting on routine hematological parameters in a small cohort of young women Clinica Chimica Acta, Vol. 483
• A Comparison of One-Step and Three-Step Approaches for Including Covariates in the Shared Parameter Growth Mixture Model 12 February 2018 | Structural Equation Modeling: A Multidisciplinary Journal, Vol. 25, No. 4
• Social context reveals gender differences in cooperative behavior 22 February 2018 | Journal of Bioeconomics, Vol. 20, No. 2
• Ambient aerosol increases minimum leaf conductance and alters the aperture–flux relationship as stomata respond to vapor pressure deficit ( VPD ) 30 March 2018 | New Phytologist, Vol. 219, No. 1
• Infusion of Quantitative and Statistical Concepts Into Biology Courses Does Not Improve Quantitative Literacy 10 October 2023 | Journal of College Science Teaching, Vol. 47, No. 5
• Sport-based physical activity recommendations and modifications in C-reactive protein and arterial thickness 27 January 2018 | European Journal of Pediatrics, Vol. 177, No. 4
• A Critical Review of Methodologies and Results in Recent Research on Belief in Free Will 28 October 2017 | Neuroethics, Vol. 11, No. 1
• Is Corneal Arcus Independently Associated With Incident Cardiovascular Disease in Asians? American Journal of Ophthalmology, Vol. 188
• The exclusive presence of the chronic pulmonary disease could be more important in affecting autonomic cardiac modulation than the severity of airflow obstruction: Analysis using heart rate variability Biomedical Signal Processing and Control, Vol. 42
• Symmetry of cortical planning for initiating stepping in sub-acute stroke Clinical Neurophysiology, Vol. 129, No. 4
• A. J. Auerbach ,
• M. Higgins ,
• P. Brickman , and
• T. C. Andrews
• Jay Labov, Monitoring Editor
• The Influence of Exercise Dosing on Outcomes in Patients With Knee Disorders: A Systematic Review Journal of Orthopaedic & Sports Physical Therapy, Vol. 48, No. 3
• Fasting conditions: Influence of water intake on clinical chemistry analytes 24 November 2017 | Biochemia Medica, Vol. 28, No. 1
• The influence of dosing on effect size of exercise therapy for musculoskeletal foot and ankle disorders: a systematic review Brazilian Journal of Physical Therapy, Vol. 22, No. 1
• Computers & Education, Vol. 125
• Effects of resistance training protocols on nonlinear analysis of heart rate variability in metabolic syndrome 1 January 2018 | Brazilian Journal of Medical and Biological Research, Vol. 51, No. 8
• Technology as a Tool in Teaching Quantitative Biology at the Secondary and Undergraduate Levels: A Review 1 January 2018 | Letters in Biomathematics, Vol. 5, No. 1
• Many Paths Toward Discovery: A Module for Teaching How Science Works 5 October 2023 | Journal of College Science Teaching, Vol. 47, No. 3
• Prelude as lifespan gauge 3 July 2018 | Scientific Study of Literature, Vol. 7, No. 2
• Cheryl A. Sensibaugh ,
• Nathaniel J. Madrid ,
• Hye-Jeong Choi ,
• William L. Anderson , and
• Marcy P. Osgood
• Jennifer Momsen, Monitoring Editor
• Anna Jo Auerbach and
• Elisabeth Schussler
• Marilyne Stains, Monitoring Editor
• A Coxiella mutualist symbiont is essential to the development of Rhipicephalus microplus 14 December 2017 | Scientific Reports, Vol. 7, No. 1
• Falacias sobre el valor p compartidas por profesores y estudiantes universitarios 14 November 2017 | Universitas Psychologica, Vol. 16, No. 3
• Standard versus physiologic bone preparation in total knee arthroplasty and the effect on joint space opening Clinical Biomechanics, Vol. 49
• Impact of Martial Arts (Judo, Karate, and Kung Fu) on Bone Mineral Density Gains in Adolescents of Both Genders: 9-Month Follow-Up Pediatric Exercise Science, Vol. 29, No. 4
• Lower cortisol response in high-resilient caregivers of people with autism: the role of anger 13 October 2016 | Stress and Health, Vol. 33, No. 4
• Lucas P. Wachsmuth ,
• Christopher R. Runyon ,
• John M. Drake , and
• Erin L. Dolan ,
• Abraham Joel K., Monitoring Editor
• Multi-Institutional, Multidisciplinary Study of the Impact of Course-Based Research Experiences Journal of Microbiology & Biology Education, Vol. 18, No. 2
• Anne-Marie Hoskinson ,
• Jessica Middlemis Maher ,
• Cody Bekkering , and
• Erin L. Dolan, Monitoring Editor
• Restoring effective sleep tranquility (REST): A feasibility and pilot study 5 April 2017 | British Journal of Occupational Therapy, Vol. 80, No. 6
• Responses of an Endangered Songbird to an Extreme Drought Event 8 June 2017 | Southeastern Naturalist, Vol. 16, No. 2
• A Systematic Examination of the Association between Parental and Child Obesity across Countries 15 May 2017 | Advances in Nutrition: An International Review Journal, Vol. 8, No. 3
• The infra-meniscal fibers of the anterolateral ligament are stronger and stiffer than the supra-meniscal fibers despite similar histological characteristics 25 January 2017 | Knee Surgery, Sports Traumatology, Arthroscopy, Vol. 25, No. 4
• Perturbation effects in men’s and women’s international sevens 29 March 2017 | International Journal of Performance Analysis in Sport, Vol. 17, No. 1-2
• The Rate of Epistaxis Incidence in New-Generation Anticoagulants and Perioperative Approach in Otorhinolaryngological Practices Journal of Craniofacial Surgery, Vol. 28, No. 2
• Noncompletion of Evidence-Based Parent Training: An Empirical Examination Among Families of Children in Foster Care 30 September 2016 | Journal of Social Service Research, Vol. 43, No. 1
• A Practical Guide to Teaching Measures of Effect Size in Physical Therapist Education Journal of Physical Therapy Education, Vol. 31, No. 2
• Effect of Soft Tissue Releases on Joint Space Opening in Total Knee Arthroplasty The Journal of Arthroplasty, Vol. 31, No. 12
• The relationship between post-operative time and cardiac autonomic modulation in breast cancer survivors International Journal of Cardiology, Vol. 224
• An Avida-ED digital evolution curriculum for undergraduate biology 29 July 2016 | Evolution: Education and Outreach, Vol. 9, No. 1
• Sports Practice and Bone Mass in Prepubertal Adolescents and Young Adults: A Cross-sectional Analysis Motriz: Revista de Educação Física, Vol. 22, No. 4
• Clinical frailty adds to acute illness severity in predicting mortality in hospitalized older adults: An observational study European Journal of Internal Medicine, Vol. 35
• Failure to Report Effect Sizes 13 September 2016 | Health Education & Behavior, Vol. 43, No. 5
• Dancers and followers in a honeybee colony differently prioritize individual and colony nutritional needs Animal Behaviour, Vol. 119
• Insights, Pearls, and Guidance on Successfully Producing and Publishing Educational Research American Journal of Pharmaceutical Education, Vol. 80, No. 5
• Psychosocial Characteristics of Student Veterans with Service-Related Disabilities: Implications for Further Research and Occupational Therapy Involvement 11 April 2016 | Occupational Therapy in Mental Health, Vol. 32, No. 2
• Assessing faculty professional development in STEM higher education: Sustainability of outcomes Science Advances, Vol. 2, No. 3
• Kristy L. Kenyon ,
• Morgan E. Onorato ,
• Alan J. Gottesman ,
• Jamila Hoque , and
• Sally G. Hoskins
• James Hewlett, Monitoring Editor:
• A Subjectively-Based Definition of Life Balance using Personal Meaning in Occupation 18 September 2014 | Journal of Occupational Science, Vol. 23, No. 1
• Protective and Vulnerability Factors Contributing to Resilience in Post-9/11 Veterans With Service-Related Injuries in Postsecondary Education 14 December 2015 | The American Journal of Occupational Therapy, Vol. 70, No. 1
• Effect of geographic region and seasonality on Clostridium difficile incidence and hospital mortality American Journal of Infection Control, Vol. 43, No. 12
• Late effects of adjuvant chemotherapy for breast cancer on fine motor function 9 March 2015 | Psycho-Oncology, Vol. 24, No. 12
• The association between childhood trauma and facial emotion recognition in adults with bipolar disorder Psychiatry Research, Vol. 229, No. 3
• Matthew P. Rowe ,
• B. Marcus Gillespie ,
• Kevin R. Harris ,
• Steven D. Koether ,
• Li-Jen Y. Shannon , and
• Lori A. Rose
• Ross Nehm, Monitoring Editor
• Adolescents' Sedentary Behaviors in Two European Cities 26 May 2015 | Research Quarterly for Exercise and Sport, Vol. 86, No. 3
• Sports practice is related to parasympathetic activity in adolescents Revista Paulista de Pediatria (English Edition), Vol. 33, No. 2
• Commentary on Dutta and Pullig (2011): Corrective action is more effective than downplaying harm for restoring brand equity Journal of Business Research, Vol. 68, No. 6
• Prática esportiva está relacionada à atividade parassimpática em adolescentes Revista Paulista de Pediatria, Vol. 33, No. 2
• Reduction of Pain Sensitivity is Associated with the Response to Treatment in Women with Chronic Pelvic Pain 1 May 2015 | Pain Medicine, Vol. 16, No. 5
• Strontium ranelate improves the interaction of osteoblastic cells with titanium substrates: Increase in cell proliferation, differentiation and matrix mineralization 15 July 2015 | Biomatter, Vol. 5, No. 1
• A. Malcolm Campbell ,
• Todd Eckdahl ,
• Brian Cronk ,
• Corinne Andresen ,
• Paul Frederick ,
• Samantha Huckuntod ,
• Claire Shinneman ,
• Annie Wacker , and
• Mary Lee Ledbetter, Monitoring Editor
• Leslie M. Stevens , and
• Nancy Pelaez, Monitoring Editor

© 2013 J. Middlemis Maher et al. CBE—Life Sciences Education © 2013 The American Society for Cell Biology. This article is distributed by The American Society for Cell Biology under license from the author(s). It is available to the public under an Attribution–Noncommercial–Share Alike 3.0 Unported Creative Commons License (http://creativecommons.org/licenses/by-nc-sa/3.0).

Archer Library

Quantitative research: literature review .

• Archer Library This link opens in a new window
• Research Resources handout This link opens in a new window
• Locating Books
• Library eBook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Statistics
• Literature Review Resources
• Citations & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

Archer Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: scholarly & practitioner journals.

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Basic Searching with EBSCO
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Statistics
• Next: Literature Review Resources >>
• Last Updated: Apr 23, 2024 3:36 PM
• URL: https://libguides.ashland.edu/quantitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical Literature
• Classical Reception
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Archaeology
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Agriculture
• History of Education
• History of Emotions
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Variation
• Language Families
• Language Evolution
• Language Reference
• Language Acquisition
• Lexicography
• Linguistic Theories
• Linguistic Typology
• Linguistic Anthropology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Modernism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Culture
• Music and Media
• Music and Religion
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Society
• Law and Politics
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Oncology
• Medical Toxicology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Medical Ethics
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Games
• Computer Security
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Neuroscience
• Cognitive Psychology
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business History
• Business Ethics
• Business Strategy
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Methodology
• Economic History
• Economic Systems
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Theory
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

5 Effect Size Metrics and Pooling Methods

• Published: March 2008
• Cite Icon Cite
• Permissions Icon Permissions

This chapter covers the meanings and measures of effect size (ES), focusing on common effect size measures. Confidence intervals are used to capture the precision of ES estimates. Corrections for clustering, dependencies, and other problems are considered. The use of Forrest plots and different pooling methods are described. Finally, measures of heterogeneity of effects across studies — and ways to handle this heterogeneity — are explored.

Signed in as

Institutional accounts.

• Google Scholar Indexing
• GoogleCrawler [DO NOT DELETE]

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code

Institutional access

• Sign in with a library card Sign in with username/password Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Sign in through your institution

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Sign in with a library card

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.