literature review on health and medicine

Literature and Medicine

Michael Blackie, University of Illinois at Chicago

Journal Details

Call for papers and guidelines for contributors.

Literature and Medicine  is a peer-reviewed journal publishing scholarship that explores representational and cultural practices concerning health care and the body. Areas of interest include disease, illness, and health; the cultures of biomedical science and technology and of the clinic; disability; and violence, trauma, and power relations as these are represented and interpreted in broadly-defined archives of verbal, visual, and material texts.  Literature and Medicine  features one thematic and one general issue each year. Past theme issues have explored identity and difference; contagion and infection; cancer pathography; the representations of genomics; and the narration of pain.

Literature and Medicine  is published semiannually. Theme issues are announced in calls for papers in the journal and on the journal website.  Literature and Medicine  editors will consider essay clusters devoted to a particular topic or written on a specific occasion. Submissions on any aspect of literature and medicine will be considered, but the journal rarely publishes short notes, personal essays, or creative writing. Authors are advised to look carefully at past issues of the journal (available on the journal website) before submitting their work. We welcome submissions by graduate students, but encourage authors to rework term papers into publishable manuscripts (as one does in turning dissertation into book) before submission.

Manuscripts should be between 5,000 and 9,000 words in length, approximately twenty-five to forty pages inclusive of double-spaced notes. Please include an abstract of 100–150 words and 3–5 keywords.

All submissions should have text, endnotes, and bibliography double-spaced and prepared according to guidelines in  The Chicago Manual of Style , current edition. Authors will be responsible for securing permission to include visual images, figures, or verbal quotations that exceed fair use.

Literature and Medicine  is a peer-reviewed journal. Authors’ names should appear only on a cover sheet, and any identifiers in the text should be masked so manuscripts can be reviewed anonymously.  Literature and Medicine  reviews only unpublished manuscripts that are not simultaneously under review for publication elsewhere.

Manuscripts must be submitted in digital form here . 

Correspondence should be sent to: Michael Blackie, PhD Executive Editor,  Literature and Medicine
 808 S Wood St Department of Medical Education Chicago, Illinois 60612-7309

Email:  [email protected]

Executive Editor

Michael Blackie, The University of Illinois at Chicago

Managing Editor

Anna Fenton-Hathaway

Senior Consulting Editors

Catherine Belling  Rita Charon  Anne Hudson Jones  Maura Spiegel

Associate Editors

Sari Altschuler  Tod Chambers  Sayantani DasGupta  Lisa Diedrich  Rebecca Garden  Martha Stoddard Holmes  Ann Jurecic  Thomas Long  Juliet McMullin  Kirsten Ostherr  Lorenzo Servitje  Susan Squier  Jaipreet Virdi  Priscilla Wald

Book Review Editor

Travis Chi Wing Lau

Comics Editor

MK Czerwiec

Contributing Editors

Felice Aull  Gert H. Brieger  D. Heyward Brock  Ronald A. Carson  Larry R. Churchill  Arthur W. Frank  Rosemarie Garland-Thomson  Sander Gilman  Peter W. Graham  Steven Marcus  Marilyn Chandler McEntyre  Jonathan M. Metzl  Kathryn Montgomery  David B. Morris  Kathryn Allen Rabuzzi  Margrit Shildrick

International Contributing Editors

Robin S. Downie  John Wiltshire  Faith McLellan

Health Humanities, Department of Medical Education, University of Illinois at Chicago

The Hopkins Press Journals Ethics and Malpractice Statement can be found at the ethics-and-malpractice  page.

Peer Review Policy

Literature and Medicine  accepts for consideration original scholarly manuscripts, not under consideration at any other publication, that explore the connections between literature (broadly defined, in any media) and medical health care. The journal does not consider unsolicited submissions of short notes, informal pieces, personal essays, or creative writing. Original translations are considered, if accompanied by substantial scholarly introduction or annotation. We welcome proposals for guest-edited theme issues or essay clusters devoted to a particular topic or marking a specific occasion (contact the editor directly regarding these).

Literature and Medicine  is a double-blind peer-reviewed journal.  Preliminary desk review rejects clearly unsuitable or weak submissions. Submissions that prima facie meet the review criteria—original and relevant contribution to the field;  humanities methodology; evidence in support of claims; quality of expression—are assigned two reviewers familiar with the subject matter and the field. Reviewers are asked to recommend rejection, acceptance with major revisions, or acceptance with minor revisions. The editor adjudicates, and provides a final decision: reject, accept with major revisions or with minor revisions, or reject with option to revise and resubmit. Resubmitted manuscripts are sent for re-review to at least one of the original reviewers, along with their original report. Accepted submissions are returned with required and suggested revisions, and revised manuscripts are reviewed by the editor, who makes the final acceptance decision. The manuscript will then be copy-edited in collaboration with the author.

Timetable:  ​varies depending on alacrity of peer reviewers and the fact that the journal publishes one thematic and one general issue yearly.

Send books for review to: Dr. Travis Chi Wing Lau 50 W. Broad St. #3205 Columbus, OH 43215 Email  [email protected]

Please send book review copies to the contact above. Review copies received by the Johns Hopkins University Press office will be discarded.

Abstracting & Indexing Databases

• Reactions Weekly (Online)
• Arts & Humanities Citation Index
• Current Contents
• Web of Science
• Dietrich's Index Philosophicus
• IBZ - Internationale Bibliographie der Geistes- und Sozialwissenschaftlichen Zeitschriftenliteratur
• Internationale Bibliographie der Rezensionen Geistes- und Sozialwissenschaftlicher Literatur
• Academic Search Alumni Edition, 4/1/2005-
• Academic Search Complete, 4/1/2005-
• Academic Search Elite, 4/1/2005-
• Academic Search Premier, 4/1/2005-
• Current Abstracts, 4/1/2005-
• Humanities International Complete, 4/1/2005-
• Humanities International Index, 4/1/2005-
• Humanities Source, 4/1/2005-
• Humanities Source Ultimate, 4/1/2005-
• International Bibliography of Theatre & Dance with Full Text, 1/1/1993-
• MLA International Bibliography (Modern Language Association)
• RILM Abstracts of Music Literature (Repertoire International de Litterature Musicale)
• TOC Premier (Table of Contents), 4/1/2005-
• Scopus, 1986-
• ArticleFirst, vol.11, no.2, 1992-vol.29, no.1, 2011
• Electronic Collections Online, vol.14, no.1, 1995-vol.29, no.1, 2011
• Periodical Abstracts, v.19, n.1, 2000-v.28, n.2, 2009
• Personal Alert (E-mail)
• Art, Design & Architecture Collection, 04/01/1996-
• Arts & Humanities Database, 04/01/1996-
• Arts Premium Collection, 4/1/1996-
• Health & Medical Collection, 4/1/1996-
• Health Research Premium Collection, 4/1/1996-
• Hospital Premium Collection, 4/1/1996-
• Medical Database, 4/1/1996-
• Periodicals Index Online
• Professional ProQuest Central, 04/01/1996-
• ProQuest 5000, 04/01/1996-
• ProQuest 5000 International, 04/01/1996-
• ProQuest Central, 04/01/1996-
• Research Library, 04/01/1996-

Abstracting & Indexing Sources

• Index Medicus (Ceased) (Print)
• MLA Abstracts of Articles in Scholarly Jourals (Ceased) (Print)

Source: Ulrichsweb Global Serials Directory.

0.2 (2022) 0.3 (Five-Year Impact Factor) 0.00027(Eigenfactor™ Score) Rank in Category (by Journal Impact Factor): Note:  While journals indexed in AHCI and ESCI are receiving a JIF for the first time in June 2023, they will not receive ranks, quartiles, or percentiles until the release of 2023 data in June 2024.  

© Clarivate Analytics 2023

Published twice a year

Readers include: Scholars in the medical humanities, practitioners, philosophers, historians, writers, and students of literature

Print circulation: 104

Print Advertising Rates

Full Page: (4.75 x 7.5") - $375.00

Half Page: (4.75 x 3.5") - $281.00

2 Page Spread - $563.00

Print Advertising Deadlines

May Issue - March 15

November Issue - September 15

Online Advertising Rates (per month)

Promotion (400x200 pixels) - $281.00

Online Advertising Deadline

Online advertising reservations are placed on a month-to-month basis.

All online ads are due on the 20th of the month prior to the reservation.

General Advertising Info

For more information on advertising or to place an ad, please visit the Advertising page.  

Institute for Bioethics & Health Humanities The Institute for Bioethics & Health Humanities is committed to moral inquiry, research, teaching, and professional service in medicine and health care. In today's often bewildering world of scientific, technological, cultural, and political changes, medicine faces human problems and possibilities that transcend traditional academic disciplines.

Literature, Arts, and Medicine Database At NYU School of Medicine: A multi-media annotated bibliographic Web resource of prose, poetry, art and film for teaching and scholarship in the Medical Humanities. Written by a multi-institutional board of editor/annotators.

Roster of Physician Writers A list of creative writers who also practiced medicine. It includes information such as nationality, dates of birth and death, and a list of written works.

Journal of Medical Humanities This journal focuses on interdisciplinary inquiry in medicine and medical education. It publishes original essays of a theoretical and critical nature, combining explorations of the traditional humanities--notably literature, history, philosophy, and bioethics--along with related inquiry from sociology, anthropology, pedagogy, and other branches of the social and behavioral sciences that have strong humanistic traditions.

The University of Houston Health Law and Policy Institute Contains hundreds of annotated links from the Health Law and Policy Institute's Web site to informative Web sites pertaining to health law, health policy, and general health.

Awards and Milestones

The  Literature and Medicine  Spring 2004 special issue titled "Difference and Identity" was chosen as a runner-up for the Council of Editors of Learned Journals (CELJ) 2004 Best Special Issue award.

Praise for the Journal

"In medicine's high-tech environment,  Literature & Medicine  is a tangible way of expressing the totality of needs of both patients and medical staff. As a scholarly, peer-reviewed journal, it fills an important subject domain in a growing and important specialty. At the Mayo Clinic, we have two institutional subscriptions to  Literature and Medicine ." J. Michael Homan | Director of Libraries, The Mayo Clinic

" Literature and Medicine?  Almost as necessary for a medical education as a stethoscope." Dannie Abse, M.D. | author of  White Coat, Purple Coat and A Poet in the Family

" Literature and Medicine  was one of the earliest journals to give a public voice to the pioneers in bioethics. Nobody, it seems to me, has done better since." Leslie Fiedler, Ph.D. | Samuel L. Clemens Chair Professor, SUNY Buffalo,  author of  Freaks and The Tyranny of the Normal

" Literature and Medicine  should have a role in the teaching of every medical educator and a place on the shelves of every medical school library." Larry R. Churchill | Professor and Chair, Social Medicine, author of  Rationing Health Care in America and Self-Interest and Universal Health Care

"No medical library can be considered complete without a subscription to the essential journal  Literature and Medicine.  It is, in its way, as important as the  New England Journal of Medicine  or  Lancet. " Richard Selzer, M.D. | author of  Mortal Lessons and Letters to a Young Doctor

eTOC (Electronic Table of Contents) alerts can be delivered to your inbox when this or any Hopkins Press journal is published via your ProjectMUSE MyMUSE account. Visit the eTOC instructions page for detailed instructions on setting up your MyMUSE account and alerts. 

Also of Interest

James M. DuBois, Washington University in St. Louis; Ana S. Iltis, Wake Forest University; and  Heidi Walsh, Washington University in St. Louis

Jeremy A. Greene, M.D., Ph.D., Johns Hopkins University; Alisha Rankin, Ph.D., Tufts University; Gabriela Soto Laveaga, Ph.D., Harvard University

 A. Hal Strelnick, MD; Albert Einstein College of Medicine and Karen Calhoun, MA University of Michigan Institute for Clinical & Health Research

Martha Montello, Harvard Medical School

Virginia M. Brennan, PhD, MA, Meharry Medical College

Hopkins Press Journals

Health (Nursing, Medicine, Allied Health)

• Find Articles/Databases
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images and Streaming Video
• Grey Literature
• Mobile Apps & "Point of Care" Tools
• Tests & Measures This link opens in a new window
• Citing Sources
• Selecting Databases
• Framing Research Questions
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Saving Searches
• Term Glossary
• Critical Appraisal Resources
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Finding Full Text

Choosing a Review Type

For guidance related to choosing a review type, see:

• "What Type of Review is Right for You?" - Decision Tree (PDF) This decision tree, from Cornell University Library, highlights key difference between narrative, systematic, umbrella, scoping and rapid reviews.
• Reviewing the literature: choosing a review design Noble, H., & Smith, J. (2018). Reviewing the literature: Choosing a review design. Evidence Based Nursing, 21(2), 39–41. https://doi.org/10.1136/eb-2018-102895
• What synthesis methodology should I use? A review and analysis of approaches to research synthesis Schick-Makaroff, K., MacDonald, M., Plummer, M., Burgess, J., & Neander, W. (2016). What synthesis methodology should I use? A review and analysis of approaches to research synthesis. AIMS Public Health, 3 (1), 172-215. doi:10.3934/publichealth.2016.1.172 More information less... ABSTRACT: Our purpose is to present a comprehensive overview and assessment of the main approaches to research synthesis. We use "research synthesis" as a broad overarching term to describe various approaches to combining, integrating, and synthesizing research findings.
• Right Review - Decision Support Tool Not sure of the most suitable review method? Answer a few questions and be guided to suitable knowledge synthesis methods. Updated in 2022 and featured in the Journal of Clinical Epidemiology 10.1016/j.jclinepi.2022.03.004

Types of Evidence Synthesis / Literature Reviews

Literature reviews are are comprehensive summaries and syntheses of the previous research on a given topic.  While narrative reviews are common across all academic disciplines, reviews that focus on appraising and synthesizing research evidence are increasingly important in the health and social sciences.  

Most evidence synthesis methods use formal and explicit methods to identify, select and combine results from multiple studies, making evidence synthesis a form of meta-research.  

The review purpose, methods used and the results produced vary among different kinds of literature reviews; some of the common types of literature review are detailed below.

Common Types of Literature Reviews 1

Narrative (literature) review.

• A broad term referring to reviews with a wide scope and non-standardized methodology
• Search strategies, comprehensiveness of literature search, time range covered and method of synthesis will vary and do not follow an established protocol

Integrative Review

• A type of literature review based on a systematic, structured literature search
• Often has a broadly defined purpose or review question
• Seeks to generate or refine and theory or hypothesis and/or develop a holistic understanding of a topic of interest
• Relies on diverse sources of data (e.g. empirical, theoretical or methodological literature; qualitative or quantitative studies)

Systematic Review

• Systematically and transparently collects and categorize existing evidence on a question of scientific, policy or management importance
• Follows a research protocol that is established a priori
• Some sub-types of systematic reviews include: SRs of intervention effectiveness, diagnosis, prognosis, etiology, qualitative evidence, economic evidence, and more.
• Time-intensive and often takes months to a year or more to complete 
• The most commonly referred to type of evidence synthesis; sometimes confused as a blanket term for other types of reviews

Meta-Analysis

• Statistical technique for combining the findings from disparate quantitative studies
• Uses statistical methods to objectively evaluate, synthesize, and summarize results
• Often conducted as part of a systematic review

Scoping Review

• Systematically and transparently collects and categorizes existing evidence on a broad question of scientific, policy or management importance
• Seeks to identify research gaps, identify key concepts and characteristics of the literature and/or examine how research is conducted on a topic of interest
• Useful when the complexity or heterogeneity of the body of literature does not lend itself to a precise systematic review
• Useful if authors do not have a single, precise review question
• May critically evaluate existing evidence, but does not attempt to synthesize the results in the way a systematic review would 
• May take longer than a systematic review

Rapid Review

• Applies a systematic review methodology within a time-constrained setting
• Employs methodological "shortcuts" (e.g., limiting search terms and the scope of the literature search), at the risk of introducing bias
• Useful for addressing issues requiring quick decisions, such as developing policy recommendations

Umbrella Review

• Reviews other systematic reviews on a topic
• Often defines a broader question than is typical of a traditional systematic review
• Most useful when there are competing interventions to consider

1. Adapted from:

Eldermire, E. (2021, November 15). A guide to evidence synthesis: Types of evidence synthesis. Cornell University LibGuides. https://guides.library.cornell.edu/evidence-synthesis/types

Nolfi, D. (2021, October 6). Integrative Review: Systematic vs. Scoping vs. Integrative. Duquesne University LibGuides. https://guides.library.duq.edu/c.php?g=1055475&p=7725920

Delaney, L. (2021, November 24). Systematic reviews: Other review types. UniSA LibGuides. https://guides.library.unisa.edu.au/SystematicReviews/OtherReviewTypes

Further Reading: Exploring Different Types of Literature Reviews

• A typology of reviews: An analysis of 14 review types and associated methodologies Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26 (2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x More information less... ABSTRACT: The expansion of evidence-based practice across sectors has lead to an increasing variety of review types. However, the diversity of terminology used means that the full potential of these review types may be lost amongst a confusion of indistinct and misapplied terms. The objective of this study is to provide descriptive insight into the most common types of reviews, with illustrative examples from health and health information domains.
• Clarifying differences between review designs and methods Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1 , 28. doi:10.1186/2046-4053-1-28 More information less... ABSTRACT: This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews....It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation.
• Are we talking the same paradigm? Considering methodological choices in health education systematic review Gordon, M. (2016). Are we talking the same paradigm? Considering methodological choices in health education systematic review. Medical Teacher, 38 (7), 746-750. doi:10.3109/0142159X.2016.1147536 More information less... ABSTRACT: Key items discussed are the positivist synthesis methods meta-analysis and content analysis to address questions in the form of "whether and what" education is effective. These can be juxtaposed with the constructivist aligned thematic analysis and meta-ethnography to address questions in the form of "why." The concept of the realist review is also considered. It is proposed that authors of such work should describe their research alignment and the link between question, alignment and evidence synthesis method selected.
• Meeting the review family: Exploring review types and associated information retrieval requirements Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information & Libraries Journal, 36(3), 202–222. doi: 10.1111/hir.12276

Integrative Reviews

"The integrative review method is an approach that allows for the inclusion of diverse methodologies (i.e. experimental and non-experimental research)." (Whittemore & Knafl, 2005, p. 547).

• The integrative review: Updated methodology Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52 (5), 546–553. doi:10.1111/j.1365-2648.2005.03621.x More information less... ABSTRACT: The aim of this paper is to distinguish the integrative review method from other review methods and to propose methodological strategies specific to the integrative review method to enhance the rigour of the process....An integrative review is a specific review method that summarizes past empirical or theoretical literature to provide a more comprehensive understanding of a particular phenomenon or healthcare problem....Well-done integrative reviews present the state of the science, contribute to theory development, and have direct applicability to practice and policy.

• Conducting integrative reviews: A guide for novice nursing researchers Dhollande, S., Taylor, A., Meyer, S., & Scott, M. (2021). Conducting integrative reviews: A guide for novice nursing researchers. Journal of Research in Nursing, 26(5), 427–438. https://doi.org/10.1177/1744987121997907
• Rigour in integrative reviews Whittemore, R. (2007). Rigour in integrative reviews. In C. Webb & B. Roe (Eds.), Reviewing Research Evidence for Nursing Practice (pp. 149–156). John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470692127.ch11

Scoping Reviews

Scoping reviews are evidence syntheses that are conducted systematically, but begin with a broader scope of question than traditional systematic reviews, allowing the research to 'map' the relevant literature on a given topic.

• Scoping studies: Towards a methodological framework Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19-32. doi:10.1080/1364557032000119616 More information less... ABSTRACT: We distinguish between different types of scoping studies and indicate where these stand in relation to full systematic reviews. We outline a framework for conducting a scoping study based on our recent experiences of reviewing the literature on services for carers for people with mental health problems.
• Scoping studies: Advancing the methodology Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 69. doi:10.1186/1748-5908-5-69 More information less... ABSTRACT: We build upon our experiences conducting three scoping studies using the Arksey and O'Malley methodology to propose recommendations that clarify and enhance each stage of the framework.
• Methodology for JBI scoping reviews Peters, M. D. J., Godfrey, C. M., McInerney, P., Baldini Soares, C., Khalil, H., & Parker, D. (2015). The Joanna Briggs Institute reviewers’ manual: Methodology for JBI scoping reviews [PDF]. Retrieved from The Joanna Briggs Institute website: http://joannabriggs.org/assets/docs/sumari/Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf More information less... ABSTRACT: Unlike other reviews that address relatively precise questions, such as a systematic review of the effectiveness of a particular intervention based on a precise set of outcomes, scoping reviews can be used to map the key concepts underpinning a research area as well as to clarify working definitions, and/or the conceptual boundaries of a topic. A scoping review may focus on one of these aims or all of them as a set.

Systematic vs. Scoping Reviews: What's the Difference? 

YouTube Video 4 minutes, 45 seconds

Rapid Reviews

Rapid reviews are systematic reviews that are undertaken under a tighter timeframe than traditional systematic reviews. 

• Evidence summaries: The evolution of a rapid review approach Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: The evolution of a rapid review approach. Systematic Reviews, 1 (1), 10. doi:10.1186/2046-4053-1-10 More information less... ABSTRACT: Rapid reviews have emerged as a streamlined approach to synthesizing evidence - typically for informing emergent decisions faced by decision makers in health care settings. Although there is growing use of rapid review "methods," and proliferation of rapid review products, there is a dearth of published literature on rapid review methodology. This paper outlines our experience with rapidly producing, publishing and disseminating evidence summaries in the context of our Knowledge to Action (KTA) research program.
• What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments Harker, J., & Kleijnen, J. (2012). What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. International Journal of Evidence‐Based Healthcare, 10 (4), 397-410. doi:10.1111/j.1744-1609.2012.00290.x More information less... ABSTRACT: In recent years, there has been an emergence of "rapid reviews" within Health Technology Assessments; however, there is no known published guidance or agreed methodology within recognised systematic review or Health Technology Assessment guidelines. In order to answer the research question "What is a rapid review and is methodology consistent in rapid reviews of Health Technology Assessments?", a study was undertaken in a sample of rapid review Health Technology Assessments from the Health Technology Assessment database within the Cochrane Library and other specialised Health Technology Assessment databases to investigate similarities and/or differences in rapid review methodology utilised.
• Rapid Review Guidebook Dobbins, M. (2017). Rapid review guidebook. Hamilton, ON: National Collaborating Centre for Methods and Tools.
• NCCMT Summary and Tool for Dobbins' Rapid Review Guidebook National Collaborating Centre for Methods and Tools. (2017). Rapid review guidebook. Hamilton, ON: McMaster University. Retrieved from http://www.nccmt.ca/knowledge-repositories/search/308
• << Previous: Literature Reviews
• Next: Conducting & Reporting Systematic Reviews >>
• Last Updated: May 2, 2024 10:52 AM
• URL: https://guides.nyu.edu/health

The Sheridan Libraries

• Medicine, Nursing, and Other Health Fields
• Sheridan Libraries

Literature Reviews

• How to Access Full Text
• PUBMED and EMBASE
• Google Scholar and Google Books
• Consumer Health Information
• Drugs and Pharmacology
• Medical Campus Information
• For Pre-health Students
• Videos and Images
• Avoiding Plagiarism
• Citing Sources This link opens in a new window
• Copyright This link opens in a new window
• Evaluating Information This link opens in a new window
• RefWorks Guide and Help This link opens in a new window
• Organizing/Synthesizing
• Peer Review
• Ulrich's -- One More Way To Find Peer-reviewed Papers

"Literature review," "systematic literature review," "integrative literature review" -- these are terms used in different disciplines for basically the same thing -- a rigorous examination of the scholarly literature about a topic (at different levels of rigor, and with some different emphases).  

1. Our library's guide to Writing a Literature Review

2. Other helpful sites

• Writing Center at UNC (Chapel Hill) -- A very good guide about lit reviews and how to write them
• Literature Review: Synthesizing Multiple Sources (LSU, June 2011 but good; PDF) -- Planning, writing, and tips for revising your paper

3. Welch Library's list of the types of expert reviews

Doing a good job of organizing your information makes writing about it a lot easier.

You can organize your sources using a citation manager, such as refworks , or use a matrix (if you only have a few references):.

• Use Google Sheets, Word, Excel, or whatever you prefer to create a table
• The column headings should include the citation information, and the main points that you want to track, as shown

Synthesizing your information is not just summarizing it. Here are processes and examples about how to combine your sources into a good piece of writing:

• Purdue OWL's Synthesizing Sources
• Synthesizing Sources (California State University, Northridge)

Annotated Bibliography  

An "annotation" is a note or comment. An "annotated bibliography" is a "list of citations to books, articles, and [other items]. Each citation is followed by a brief...descriptive and evaluative paragraph, [whose purpose is] to inform the reader of the relevance, accuracy, and quality of the sources cited."*

• Sage Research Methods (database) --> Empirical Research and Writing (ebook) -- Chapter 3: Doing Pre-research  
• Purdue's OWL (Online Writing Lab) includes definitions and samples of annotations  
• Cornell's guide * to writing annotated bibliographies  

* Thank you to Olin Library Reference, Research & Learning Services, Cornell University Library, Ithaca, NY, USA https://guides.library.cornell.edu/annotatedbibliography

What does "peer-reviewed" mean?

• If an article has been peer-reviewed before being published, it means that the article has been read by other people in the same field of study ("peers").
• The author's reviewers have commented on the article, not only noting typos and possible errors, but also giving a judgment about whether or not the article should be published by the journal to which it was submitted.

How do I find "peer-reviewed" materials?

• Most of the the research articles in scholarly journals are peer-reviewed.
• Many databases allow you to check a box that says "peer-reviewed," or to see which results in your list of results are from peer-reviewed sources. Some of the databases that provide this are Academic Search Ultimate, CINAHL, PsycINFO, and Sociological Abstracts.

What kinds of materials are *not* peer-reviewed?

• open web pages
• most newspapers, newsletters, and news items in journals
• letters to the editor
• press releases
• columns and blogs
• book reviews
• anything in a popular magazine (e.g., Time, Newsweek, Glamour, Men's Health)

If a piece of information wasn't peer-reviewed, does that mean that I can't trust it at all?

No; sometimes you can. For example, the preprints submitted to well-known sites such as  arXiv  (mainly covering physics) and  CiteSeerX (mainly covering computer science) are probably trustworthy, as are the databases and web pages produced by entities such as the National Library of Medicine, the Smithsonian Institution, and the American Cancer Society.

Is this paper peer-reviewed? Ulrichsweb will tell you.

1) On the library home page , choose "Articles and Databases" --> "Databases" --> Ulrichsweb

2) Put in the title of the JOURNAL (not the article), in quotation marks so all the words are next to each other

3) Mouse over the black icon, and you'll see that it means "refereed" (which means peer-reviewed, because it's been looked at by referees or reviewers). This journal is not peer-reviewed, because none of the formats have a black icon next to it:

• << Previous: Evaluating Information
• Next: RefWorks Guide and Help >>
• Last Updated: Apr 23, 2024 6:48 PM
• URL: https://guides.library.jhu.edu/medicine

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Performing a...

Performing a literature review

• Related content
• Peer review
• Gulraj S Matharu , academic foundation doctor ,
• Christopher D Buckley , Arthritis Research UK professor of rheumatology
• 1 Institute of Biomedical Research, College of Medical and Dental Sciences, School of Immunity and Infection, University of Birmingham, UK

A necessary skill for any doctor

What causes disease, which drug is best, does this patient need surgery, and what is the prognosis? Although experience helps in answering these questions, ultimately they are best answered by evidence based medicine. But how do you assess the evidence? As a medical student, and throughout your career as a doctor, critical appraisal of published literature is an important skill to develop and refine. At medical school you will repeatedly appraise published literature and write literature reviews. These activities are commonly part of a special study module, research project for an intercalated degree, or another type of essay based assignment.

Formulating a question

Literature reviews are most commonly performed to help answer a particular question. While you are at medical school, there will usually be some choice regarding the area you are going to review.

Once you have identified a subject area for review, the next step is to formulate a specific research question. This is arguably the most important step because a clear question needs to be defined from the outset, which you aim to answer by doing the review. The clearer the question, the more likely it is that the answer will be clear too. It is important to have discussions with your supervisor when formulating a research question as his or her input will be invaluable. The research question must be objective and concise because it is easier to search through the evidence with a clear question. The question also needs to be feasible. What is the point in having a question for which no published evidence exists? Your supervisor’s input will ensure you are not trying to answer an unrealistic question. Finally, is the research question clinically important? There are many research questions that may be answered, but not all of them will be relevant to clinical practice. The research question we will use as an example to work through in this article is, “What is the evidence for using angiotensin converting enzyme (ACE) inhibitors in patients with hypertension?”

Collecting the evidence

After formulating a specific research question for your literature review, the next step is to collect the evidence. Your supervisor will initially point you in the right direction by highlighting some of the more relevant papers published. Before doing the literature search it is important to agree a list of keywords with your supervisor. A source of useful keywords can be obtained by reading Cochrane reviews or other systematic reviews, such as those published in the BMJ . 1 2 A relevant Cochrane review for our research question on ACE inhibitors in hypertension is that by Heran and colleagues. 3 Appropriate keywords to search for the evidence include the words used in your research question (“angiotensin converting enzyme inhibitor,” “hypertension,” “blood pressure”), details of the types of study you are looking for (“randomised controlled trial,” “case control,” “cohort”), and the specific drugs you are interested in (that is, the various ACE inhibitors such as “ramipril,” “perindopril,” and “lisinopril”).

Once keywords have been agreed it is time to search for the evidence using the various electronic medical databases (such as PubMed, Medline, and EMBASE). PubMed is the largest of these databases and contains online information and tutorials on how to do literature searches with worked examples. Searching the databases and obtaining the articles are usually free of charge through the subscription that your university pays. Early consultation with a medical librarian is important as it will help you perform your literature search in an impartial manner, and librarians can train you to do these searches for yourself.

Literature searches can be broad or tailored to be more specific. With our example, a broad search would entail searching all articles that contain the words “blood pressure” or “ACE inhibitor.” This provides a comprehensive list of all the literature, but there are likely to be thousands of articles to review subsequently (fig 1). ⇓ In contrast, various search restrictions can be applied on the electronic databases to filter out papers that may not be relevant to your review. Figure 2 gives an example of a specific search. ⇓ The search terms used in this case were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in the English language over the last 10 years, with the search terms appearing in the title of the study only. Thus the more specific the search strategy, the more manageable the number of articles to review (fig 3), and this will save you time. ⇓ However, this method risks your not identifying all the evidence in the particular field. Striking a balance between a broad and a specific search strategy is therefore important. This will come with experience and consultation with your supervisor. It is important to note that evidence is continually becoming available on these electronic databases and therefore repeating the same search at a later date can provide new evidence relevant to your review.

Fig 1 Results from a broad literature search using the term “angiotensin converting enzyme inhibitor”

• Download figure
• Open in new tab
• Download powerpoint

Fig 2 Example of a specific literature search. The search terms used were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in English over the past 10 years, with the search terms appearing in the title of the study only

Fig 3 Results from a specific literature search (using the search terms and limits from figure 2)

Reading the abstracts (study summary) of the articles identified in your search may help you decide whether the study is applicable for your review—for example, the work may have been carried out using an animal model rather than in humans. After excluding any inappropriate articles, you need to obtain the full articles of studies you have identified. Additional relevant articles that may not have come up in your original search can also be found by searching the reference lists of the articles you have already obtained. Once again, you may find that some articles are still not applicable for your review, and these can also be excluded at this stage. It is important to explain in your final review what criteria you used to exclude articles as well as those criteria used for inclusion.

The National Institute for Health and Clinical Excellence (NICE) publishes evidence based guidelines for the United Kingdom and therefore provides an additional resource for identifying the relevant literature in a particular field. 4 NICE critically appraises the published literature with recommendations for best clinical practice proposed and graded based on the quality of evidence available. Similarly, there are internationally published evidence based guidelines, such as those produced by the European Society of Cardiology and the American College of Chest Physicians, which can be useful when collecting the literature in a particular field. 5 6

Appraising the evidence

Once you have collected the evidence, you need to critically appraise the published material. Box 1 gives definitions of terms you will encounter when reading the literature. A brief guide of how to critically appraise a study is presented; however, it is advisable to consult the references cited for further details.

Box 1: Definitions of common terms in the literature 7

Prospective—collecting data in real time after the study is designed

Retrospective—analysis of data that have already been collected to determine associations between exposure and outcome

Hypothesis—proposed association between exposure and outcome. If presented in the negative it is called the null hypothesis

Variable—a quantity or quality that changes during the study and can be measured

Single blind—subjects are unaware of their treatment, but clinicians are aware

Double blind—both subjects and clinicians are unaware of treatment given

Placebo—a simulated medical intervention, with subjects not receiving the specific intervention or treatment being studied

Outcome measure/endpoint—clinical variable or variables measured in a study subsequently used to make conclusions about the original interventions or treatments administered

Bias—difference between reported results and true results. Many types exist (such as selection, allocation, and reporting biases)

Probability (P) value—number between 0 and 1 providing the likelihood the reported results occurred by chance. A P value of 0.05 means there is a 5% likelihood that the reported result occurred by chance

Confidence intervals—provides a range between two numbers within which one can be certain the results lie. A confidence interval of 95% means one can be 95% certain the actual results lie within the reported range

The study authors should clearly define their research question and ideally the hypothesis to be tested. If the hypothesis is presented in the negative, it is called the null hypothesis. An example of a null hypothesis is smoking does not cause lung cancer. The study is then performed to assess the significance of the exposure (smoking) on outcome (lung cancer).

A major part of the critical appraisal process is to focus on study methodology, with your key task being an assessment of the extent to which a study was susceptible to bias (the discrepancy between the reported results and the true results). It should be clear from the methods what type of study was performed (box 2).

Box 2: Different study types 7

Systematic review/meta-analysis—comprehensive review of published literature using predefined methodology. Meta-analyses combine results from various studies to give numerical data for the overall association between variables

Randomised controlled trial—random allocation of patients to one of two or more groups. Used to test a new drug or procedure

Cohort study—two or more groups followed up over a long period, with one group exposed to a certain agent (drug or environmental agent) and the other not exposed, with various outcomes compared. An example would be following up a group of smokers and a group of non-smokers with the outcome measure being the development of lung cancer

Case-control study—cases (those with a particular outcome) are matched as closely as possible (for age, sex, ethnicity) with controls (those without the particular outcome). Retrospective data analysis is performed to determine any factors associated with developing the particular outcomes

Cross sectional study—looks at a specific group of patients at a single point in time. Effectively a survey. An example is asking a group of people how many of them drink alcohol

Case report—detailed reports concerning single patients. Useful in highlighting adverse drug reactions

There are many different types of bias, which depend on the particular type of study performed, and it is important to look for these biases. Several published checklists are available that provide excellent resources to help you work through the various studies and identify sources of bias. The CONSORT statement (which stands for CONsolidated Standards Of Reporting Trials) provides a minimum set of recommendations for reporting randomised controlled trials and comprises a rigorous 25 item checklist, with variations available for other study types. 8 9 As would be expected, most (17 of 25) of the items focus on questions relating to the methods and results of the randomised trial. The remaining items relate to the title, abstract, introduction, and discussion of the study, in addition to questions on trial registration, protocol, and funding.

Jadad scoring provides a simple and validated system to assess the methodological quality of a randomised clinical trial using three questions. 10 The score ranges from zero to five, with one point given for a “yes” in each of the following questions. (1) Was the study described as randomised? (2) Was the study described as double blind? (3) Were there details of subject withdrawals, exclusions, and dropouts? A further point is given if (1) the method of randomisation was appropriate, and (2) the method of blinding was appropriate.

In addition, the Critical Appraisal Skills Programme provides excellent tools for assessing the evidence in all study types (box 2). 11 The Oxford Centre for Evidence-Based Medicine levels of evidence is yet another useful resource for assessing the methodological quality of all studies. 12

Ensure all patients have been accounted for and any exclusions, for whatever reason, are reported. Knowing the baseline demographic (age, sex, ethnicity) and clinical characteristics of the population is important. Results are usually reported as probability values or confidence intervals (box 1).

This should explain the major study findings, put the results in the context of the published literature, and attempt to account for any variations from previous work. Study limitations and sources of bias should be discussed. Authors’ conclusions should be supported by the study results and not unnecessarily extrapolated. For example, a treatment shown to be effective in animals does not necessarily mean it will work in humans.

The format for writing up the literature review usually consists of an abstract (short structured summary of the review), the introduction or background, methods, results, and discussion with conclusions. There are a number of good examples of how to structure a literature review and these can be used as an outline when writing your review. 13 14

The introduction should identify the specific research question you intend to address and briefly put this into the context of the published literature. As you have now probably realised, the methods used for the review must be clear to the reader and provide the necessary detail for someone to be able to reproduce the search. The search strategy needs to include a list of keywords used, which databases were searched, and the specific search limits or filters applied. Any grading of methodological quality, such as the CONSORT statement or Jadad scoring, must be explained in addition to any study inclusion or exclusion criteria. 6 7 8 The methods also need to include a section on the data collected from each of the studies, the specific outcomes of interest, and any statistical analysis used. The latter point is usually relevant only when performing meta-analyses.

The results section must clearly show the process of filtering down from the articles obtained from the original search to the final studies included in the review—that is, accounting for all excluded studies. A flowchart is usually best to illustrate this. Next should follow a brief description of what was done in the main studies, the number of participants, the relevant results, and any potential sources of bias. It is useful to group similar studies together as it allows comparisons to be made by the reader and saves repetition in your write-up. Boxes and figures should be used appropriately to illustrate important findings from the various studies.

Finally, in the discussion you need to consider the study findings in light of the methodological quality—that is, the extent of potential bias in each study that may have affected the study results. Using the evidence, you need to make conclusions in your review, and highlight any important gaps in the evidence base, which need to be dealt with in future studies. Working through drafts of the literature review with your supervisor will help refine your critical appraisal skills and the ability to present information concisely in a structured review article. Remember, if the work is good it may get published.

Originally published as: Student BMJ 2012;20:e404

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

• ↵ The Cochrane Library. www3.interscience.wiley.com/cgibin/mrwhome/106568753/HOME?CRETRY=1&SRETRY=0 .
• ↵ British Medical Journal . www.bmj.com/ .
• ↵ Heran BS, Wong MMY, Heran IK, Wright JM. Blood pressure lowering efficacy of angiotensin converting enzyme (ACE) inhibitors for primary hypertension. Cochrane Database Syst Rev 2008 ; 4 : CD003823 , doi: 10.1002/14651858.CD003823.pub2. OpenUrl PubMed
• ↵ National Institute for Health and Clinical Excellence. www.nice.org.uk .
• ↵ European Society of Cardiology. www.escardio.org/guidelines .
• ↵ Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians evidence-based clinical practice guidelines (8th ed). Chest 2008 ; 133 : 381 -453S. OpenUrl CrossRef
• ↵ Wikipedia. http://en.wikipedia.org/wiki .
• ↵ Moher D, Schulz KF, Altman DG, Egger M, Davidoff F, Elbourne D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001 ; 357 : 1191 -4. OpenUrl CrossRef PubMed Web of Science
• ↵ The CONSORT statement. www.consort-statement.org/ .
• ↵ Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996 ; 17 : 1 -12. OpenUrl CrossRef PubMed Web of Science
• ↵ Critical Appraisal Skills Programme (CASP). www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme .
• ↵ Oxford Centre for Evidence-based Medicine—Levels of Evidence. www.cebm.net .
• ↵ Van den Bruel A, Thompson MJ, Haj-Hassan T, Stevens R, Moll H, Lakhanpaul M, et al . Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review. BMJ 2011 ; 342 : d3082 . OpenUrl Abstract / FREE Full Text
• ↵ Awopetu AI, Moxey P, Hinchliffe RJ, Jones KG, Thompson MM, Holt PJ. Systematic review and meta-analysis of the relationship between hospital volume and outcome for lower limb arterial surgery. Br J Surg 2010 ; 97 : 797 -803. OpenUrl CrossRef PubMed

Review Writing in Health and Medicine

• Developing a Search Strategy
• Choosing the Right Review
• Choosing Databases
• Reporting Guidelines
• Free Submission Options
• Literature/Narrative Reviews
• Rapid Reviews
• Integrative Reviews
• Scoping Reviews
• Systematic Review Process
• Umbrella Reviews
• Additional Resources

Librarians utilize their knowledge and professional skills to provide expert information services to faculty, staff, and students in academic departments and our Wegner Partners. Contact your librarian before and during your review process. Find your librarian by your department .

Overview: Literature/Narrative Reviews

A literature review is “a thematic synthesis of sources used to provide readers with an up-to-date summary of theoretical and empirical findings on a particular topic.”

Cisco, J. (2014). Teaching the Literature Review: A Practical Approach for College Instructors. Teaching and Learning Inquiry: The ISSOTL Journal, 2(2), 41-57.

Steps for writing a literature review:

Find & Evaluate

Perform a literature search - you librarian can help you with this.  Make sure you narrow your topic to make it easier to find a manageable number of sources and to get a good survey of the material.

Spend time reading and managing the information in the literature you found.

Summarize the information you find in each of your sources. Look at each source: What are the findings, the methodology, theories, etc.? 

After reviewing your summaries, you will start to notice common themes or ideas within your resources.  Sometimes putting this information into a matrix can be helpful to organize your resources and group them by their themes; you can start weaving them together.

Now you will take all of that information and integrate it.  Organize the literature.  Literature reviews are typically organized in one of the following ways:

Chronologically (show a progression of a particular methodology), or

Thematically - by idea/theme (progression of time may still be important in this organization as well).

Additional Resources:

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1471-1842.2009.00848.x.  doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Free Submission Options
• Next: Rapid Reviews >>
• Last Updated: May 7, 2024 10:01 AM
• URL: https://libguides.usd.edu/reviews

University Library, University of Illinois at Urbana-Champaign

Literature Reviews in Medicine and Health

• Types of Literature Reviews
• Narrative Review
• Meta-Analysis
• Steps of a Systematic Review
• Systematic Review Standards
• Systematic Review Tools
• Systematic Review in the Social Sciences
• Rapid Review
• Scoping Review Process
• Formulate the Question
• Develop a Protocol
• Search Terms (Mesh, etc.)
• Search Tips
• Citation Management
• Appraisal by Study Type
• Analyze Data
• Library Services

Medical, Biomedical, and Health Librarians

• Megan Sapp-Nelson - Medical And Bioengineering Librarian, [email protected]
• Yali Feng - Behavioral Sciences Research and Data Services Librarian, [email protected]
• Erin Kerby - Veterinary Medicine  [email protected]
• Kelli Trei - Biosciences  [email protected]

Research Consultation Scheduler   - Schedule an appointment with a subject specialist librarian.

About Literature Reviews

The term Literature Review refers to both a process and an output product.

As a PROCESS, a literature review is a survey of scholarly publications related to a specific research question or topic.

As a PRODUCT, a literature review can be integrated into a research study publication or it can a free-standing publication.

 A literature review offers a snapshot of published literature that addresses a specific issue, topic or question. Different types of literature reviews involve varying degrees of rigor, scope, and focus. A Narrative Review is often part of a research publication and serves to  situate a research study within the broader landscape of a topic area. Systematic Reviews, Meta-Analyses, and Scoping Reviews are free-standing publications that offer a deeper exploration of literature to address specific research questions.

Depending on the type of review...

SEARCH may or may not be comprehensive

APPRAISAL may or may not include quality assessment

SYNTHESIS is narrative

ANALYSIS may be chronological, conceptual, thematic, etc.

• Literature Reviews Comparison Chart

What kind of review should I consider?

• Right Review This tool is designed to provide guidance and supporting material to reviewers on methods for the conduct and reporting of knowledge synthesis.
• SR Toolbox The Systematic Review Toolbox is an online catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process.

Related Guides

• Demystifuing the Literature Review This guide provides a general overview of the literature review and its place in a research project, thesis, or dissertation, and demonstrates some strategies and resources for finding the information you need using the U of I Library.
• Doing Research in Medicine, Biomedicine & Health This guide is designed to help you get started with your research and provides information about management citations, managing data, finding funding, publishing and more.
• Gray Literature in Health and Biomedical Sciences by Carle Illinois College of Medicine Last Updated Nov 9, 2023 25 views this year
• Next: Types of Literature Reviews >>
• Last Updated: Mar 5, 2024 2:36 PM
• URL: https://guides.library.illinois.edu/healthscilitreviews

• University of Detroit Mercy
• Health Professions

Health Services Administration

• Writing a Literature Review
• Find Articles (Databases)
• Evidence-based Practice
• eBooks & Articles
• General Writing Support
• Creating & Printing Posters
• Research Project Web Resources
• Statistics: Health / Medical
• Searching Tips
• Streaming Video
• Database & Library Help
• Medical Apps & Mobile Sites
• Faculty Publications

Literature Review Overview

What is a Literature Review? Why Are They Important?

A literature review is important because it presents the "state of the science" or accumulated knowledge on a specific topic. It summarizes, analyzes, and compares the available research, reporting study strengths and weaknesses, results, gaps in the research, conclusions, and authors’ interpretations.

Tips and techniques for conducting a literature review are described more fully in the subsequent boxes:

• Literature review steps
• Strategies for organizing the information for your review
• Literature reviews sections
• In-depth resources to assist in writing a literature review
• Templates to start your review
• Literature review examples

Literature Review Steps

Graphic used with permission: Torres, E. Librarian, Hawai'i Pacific University

1. Choose a topic and define your research question

• Try to choose a topic of interest. You will be working with this subject for several weeks to months.
• Ideas for topics can be found by scanning medical news sources (e.g MedPage Today), journals / magazines, work experiences, interesting patient cases, or family or personal health issues.
• Do a bit of background reading on topic ideas to familiarize yourself with terminology and issues. Note the words and terms that are used.
• Develop a focused research question using PICO(T) or other framework (FINER, SPICE, etc - there are many options) to help guide you.
• Run a few sample database searches to make sure your research question is not too broad or too narrow.
• If possible, discuss your topic with your professor. 

2. Determine the scope of your review

The scope of your review will be determined by your professor during your program. Check your assignment requirements for parameters for the Literature Review.

• How many studies will you need to include?
• How many years should it cover? (usually 5-7 depending on the professor)
• For the nurses, are you required to limit to nursing literature?

3. Develop a search plan

• Determine which databases to search. This will depend on your topic. If you are not sure, check your program specific library website (Physician Asst / Nursing / Health Services Admin) for recommendations.
• Create an initial search string using the main concepts from your research (PICO, etc) question. Include synonyms and related words connected by Boolean operators
• Contact your librarian for assistance, if needed.

4. Conduct searches and find relevant literature

• Keep notes as you search - tracking keywords and search strings used in each database in order to avoid wasting time duplicating a search that has already been tried
• Read abstracts and write down new terms to search as you find them
• Check MeSH or other subject headings listed in relevant articles for additional search terms
• Scan author provided keywords if available
• Check the references of relevant articles looking for other useful articles (ancestry searching)
• Check articles that have cited your relevant article for more useful articles (descendancy searching). Both PubMed and CINAHL offer Cited By links
• Revise the search to broaden or narrow your topic focus as you peruse the available literature
• Conducting a literature search is a repetitive process. Searches can be revised and re-run multiple times during the process.
• Track the citations for your relevant articles in a software citation manager such as RefWorks, Zotero, or Mendeley

5. Review the literature

• Read the full articles. Do not rely solely on the abstracts. Authors frequently cannot include all results within the confines of an abstract. Exclude articles that do not address your research question.
• While reading, note research findings relevant to your project and summarize. Are the findings conflicting? There are matrices available than can help with organization. See the Organizing Information box below.
• Critique / evaluate the quality of the articles, and record your findings in your matrix or summary table. Tools are available to prompt you what to look for. (See Resources for Appraising a Research Study box on the HSA, Nursing , and PA guides )
• You may need to revise your search and re-run it based on your findings.

6. Organize and synthesize

• Compile the findings and analysis from each resource into a single narrative.
• Using an outline can be helpful. Start broad, addressing the overall findings and then narrow, discussing each resource and how it relates to your question and to the other resources.
• Cite as you write to keep sources organized.
• Write in structured paragraphs using topic sentences and transition words to draw connections, comparisons, and contrasts.
• Don't present one study after another, but rather relate one study's findings to another. Speak to how the studies are connected and how they relate to your work.

Organizing Information

Options to assist in organizing sources and information :

1. Synthesis Matrix

• helps provide overview of the literature
• information from individual sources is entered into a grid to enable writers to discern patterns and themes
• article summary, analysis, or results
• thoughts, reflections, or issues
• each reference gets its own row
• mind maps, concept maps, flowcharts
• at top of page record PICO or research question
• record major concepts / themes from literature
• list concepts that branch out from major concepts underneath - keep going downward hierarchically, until most specific ideas are recorded
• enclose concepts in circles and connect the concept with lines - add brief explanation as needed

3. Summary Table

• information is recorded in a grid to help with recall and sorting information when writing
• allows comparing and contrasting individual studies easily
• purpose of study
• methodology (study population, data collection tool)

Efron, S. E., & Ravid, R. (2019). Writing the literature review : A practical guide . Guilford Press.

Literature Review Sections

• Lit reviews can be part of a larger paper / research study or they can be the focus of the paper
• Lit reviews focus on research studies to provide evidence
• New topics may not have much that has been published

* The sections included may depend on the purpose of the literature review (standalone paper or section within a research paper)

Standalone Literature Review (aka Narrative Review):

• presents your topic or PICO question
• includes the why of the literature review and your goals for the review.
• provides background for your the topic and previews the key points
• Narrative Reviews: tmay not have an explanation of methods.
• include where the search was conducted (which databases) what subject terms or keywords were used, and any limits or filters that were applied and why - this will help others re-create the search
• describe how studies were analyzed for inclusion or exclusion
• review the purpose and answer the research question
• thematically - using recurring themes in the literature
• chronologically - present the development of the topic over time
• methodological - compare and contrast findings based on various methodologies used to research the topic (e.g. qualitative vs quantitative, etc.)
• theoretical - organized content based on various theories
• provide an overview of the main points of each source then synthesize the findings into a coherent summary of the whole
• present common themes among the studies
• compare and contrast the various study results
• interpret the results and address the implications of the findings
• do the results support the original hypothesis or conflict with it
• provide your own analysis and interpretation (eg. discuss the significance of findings; evaluate the strengths and weaknesses of the studies, noting any problems)
• discuss common and unusual patterns and offer explanations
•  stay away from opinions, personal biases and unsupported recommendations
• summarize the key findings and relate them back to your PICO/research question
• note gaps in the research and suggest areas for further research
• this section should not contain "new" information that had not been previously discussed in one of the sections above
• provide a list of all the studies and other sources used in proper APA 7

Literature Review as Part of a Research Study Manuscript:

• Compares the study with other research and includes how a study fills a gap in the research.
• Focus on the body of the review which includes the synthesized Findings and Discussion

Literature Reviews vs Systematic Reviews

Systematic Reviews are NOT the same as a Literature Review:

Literature Reviews:

• Literature reviews may or may not follow strict systematic methods to find, select, and analyze articles, but rather they selectively and broadly review the literature on a topic
• Research included in a Literature Review can be "cherry-picked" and therefore, can be very subjective

Systematic Reviews:

• Systemic reviews are designed to provide a comprehensive summary of the evidence for a focused research question
• rigorous and strictly structured, using standardized reporting guidelines (e.g. PRISMA, see link below)
• uses exhaustive, systematic searches of all relevant databases
• best practice dictates search strategies are peer reviewed
• uses predetermined study inclusion and exclusion criteria in order to minimize bias
• aims to capture and synthesize all literature (including unpublished research - grey literature) that meet the predefined criteria on a focused topic resulting in high quality evidence

Literature Review Examples

• Breastfeeding initiation and support: A literature review of what women value and the impact of early discharge (2017). Women and Birth : Journal of the Australian College of Midwives
• Community-based participatory research to promote healthy diet and nutrition and prevent and control obesity among African-Americans: A literature review (2017). Journal of Racial and Ethnic Health Disparities

• Vitamin D deficiency in individuals with a spinal cord injury: A literature review (2017). Spinal Cord

Resources for Writing a Literature Review

These sources have been used in developing this guide.

Resources Used on This Page

Aveyard, H. (2010). Doing a literature review in health and social care : A practical guide . McGraw-Hill Education.

Purdue Online Writing Lab. (n.d.). Writing a literature review . Purdue University. https://owl.purdue.edu/owl/research_and_citation/conducting_research/writing_a_literature_review.html

Torres, E. (2021, October 21). Nursing - graduate studies research guide: Literature review. Hawai'i Pacific University Libraries. Retrieved January 27, 2022, from https://hpu.libguides.com/c.php?g=543891&p=3727230

• << Previous: General Writing Support
• Next: Creating & Printing Posters >>
• Last Updated: Apr 19, 2024 3:11 PM
• URL: https://udmercy.libguides.com/hsa

Writing in the Health Sciences: Research and Lit Reviews

• Research and Lit Reviews
• Tables and Figures
• Citation Management
• Further Reference

What Is a Literature Review?

In simple terms, a literature review investigates the available information on a certain topic. It may be only a knowledge survey with an intentional focus. However, it is often a well-organized examination of the existing research which evaluates each resource in a systematic way. Often a lit review will involve a series of inclusion/exclusion criteria or an assessment rubric which examines the research in-depth. Below are some interesting sources to consider.

The Writing Center's Literature Reviews - UNC-Chapel Hill's writing center explains some of the key criteria involved in doing a literature review.

Literature Review vs. Systematic Review - This recent article details the difference between a literature review and a systematic review. Though the two share similar attributes, key differences are identified here.

Literature Review Steps

1. Identify a research question. For example: "Does the use of warfarin in elderly patients recovering from myocardial infarction help prevent stroke?"

2. Consider which databases might provide information for your topic. Often PubMed or CINAHL will cover a wide spectrum of biomedical issues. However, other databases and grey literature sources may specialize in certain disciplines. Embase is generally comprehensive but also specializes in pharmacological interventions.

3. Select the major subjects or ideas from your question.  Focus in on the particular concepts involved in your research. Then brainstorm synonyms and related terminology for these topics.

4. Look for the  preferred indexing terms for each concept in your question. This is especially important with databases such as PubMed, CINAHL, or Scopus where headings within the MeSH database or under the Emtree umbrella are present.  For example, the above question's keywords such as " warfarin " or "myocardial infarction" can involve related terminology or subject headings such as "anti-coagulants" or "cardiovascular disease."

5. Build your search using boolean operators. Combine the synonyms in your database using boolean operators such as AND or OR. Sometimes it is necessary to research parts of a question rather than the whole. So you might link searches for things like the preventive effects of anti-coagulants with stroke or embolism, then AND these results with the therapy for patients with cardiovascular disease.

6. Filter and save your search results from the first database (do this for all databases). This may be a short list because of your topic's limitations, but it should be no longer than 15 articles for an initial search. Make sure your list is saved or archived and presents you with what's needed to access the full text.

7. Use the same process with the next databases on your list. But pay attention to how certain major headings may alter the terminology. "Stroke" may have a suggested term of "embolism" or even "cerebrovascular incident" depending on the database.

8. Read through the material for inclusion/exclusion . Based on your project's criteria and objective, consider which studies or reviews deserve to be included and which should be discarded. Make sure the information you have permits you to go forward. 

9. Write the literature review. Begin by summarizing why your research is important and explain why your approach will help fill gaps in current knowledge. Then incorporate how the information you've selected will help you to do this. You do not need to write about all of the included research you've chosen, only the most pe rtinent.

10. Select the most relevant literature for inclusion in the body of your report. Choose the articles and data sets that are most particularly relevant to your experimental approach. Consider how you might arrange these sources in the body of your draft. 

Library Books

Call #: WZ 345 G192h 2011

ISBN #: 9780763771867

This book details a practical, step-by-step method for conducting a literature review in the health sciences. Aiming to  synthesize the information while also analyzing it, the Matrix Indexing System enables users to establish a  structured process for tracking, organizing and integrating the knowledge within a collection.

Key Research Databases

PubMed -  The premier medical database for review articles in medicine, nursing, healthcare, other related biomedical disciplines. PubMed contains over 20 million citations and can be navigated through multiple database capabilities and searching strategies.

CINAHL Ultimate - Offers comprehensive coverage of health science literature. CINAHL is particularly useful for those researching the allied disciplines of nursing, medicine, and pharmaceutical sciences.

Scopus - Database with over 12 million abstracts and citations which include peer-reviewed titles from international and Open Access journals. Also includes interactive bibliometrics and researcher profiling.

Embase - Elsevier's fully interoperable database of both Medline and Emtree-indexed articles. Embase also specializes in pharmacologic interventions.

Cochrane - Selected evidence-based medicine resources from the Cochrane Collaboration that includes peer-reviewed systematic reviews and randomized controlled trials. Access this database through OVID with TTUHSC Libraries.

DARE - Literally the Datatase of Abstracts of Reviews of Effectiveness, this collection of systematic reviews and other evidence-based research contains critical assessments from a wide variety of medical journals.

TRIP - This TRIP database is structured according to the level of evidence for its EBM content. It allows users to quickly and easily locate high-quality, accredited medical literature for clinical and research purposes.

Web of Science - Contains bibliographic articles and data from a wide variety of publications in the life sciences and other fields. Also, see this link for conducting a lit review exclusively within Web of Science.

• << Previous: Welcome
• Next: Drafting >>
• Last Updated: Sep 29, 2023 10:07 AM
• URL: https://ttuhsc.libguides.com/Writing_HealthSciences

This website uses cookies to ensure you get the best experience on our website. Without cookies your experience may not be seamless.

• Literature and Medicine

About this Journal

Founded in 1982, Literature and Medicine is a peer-reviewed journal publishing scholarship that explores representational and cultural practices concerning health care and the body. Areas of interest include disease, illness, health, and disability; violence, trauma, and power relations; and the cultures of biomedical science and technology and of the clinic, as these are represented and interpreted in verbal, visual, and material texts. Literature and Medicine features one thematic and one general issue each year. Past theme issues have explored identity and difference; contagion and infection; cancer pathography; the representations of genomics; and the narration of pain.

Literature and Medicine is co-sponsored by the Department of Medical Education, College of Medicine at the University of Illinois at Chicago.

published by

Available issues, table of contents, volume 41, 2023.

• Volume 41, Number 2, Fall 2023
• Volume 41, Number 1, Spring 2023

Volume 40, 2022

• Volume 40, Number 2, Fall 2022

Volume 39, 2021

• Volume 39, Number 2, Fall 2021
• Volume 39, Number 1, Spring 2021

Volume 38, 2020

• Volume 38, Number 2, Fall 2020
• Volume 38, Number 1, Spring 2020

Volume 37, 2019

• Volume 37, Number 2, Fall 2019
• Volume 37, Number 1, Spring 2019

Volume 36, 2018

• Volume 36, Number 2, Fall 2018
• Volume 36, Number 1, Spring 2018

Volume 35, 2017

• Volume 35, Number 2, Fall 2017
• Volume 35, Number 1, Spring 2017

Volume 34, 2016

• Volume 34, Number 2, Fall 2016
• Volume 34, Number 1, Spring 2016

Volume 33, 2015

• Volume 33, Number 2, Fall 2015
• Volume 33, Number 1, Spring 2015

Volume 32, 2014

• Volume 32, Number 2, Fall 2014
• Volume 32, Number 1, Spring 2014

Volume 31, 2013

• Volume 31, Number 2, Fall 2013
• Volume 31, Number 1, Spring 2013

Volume 30, 2012

• Volume 30, Number 2, Fall 2012
• Volume 30, Number 1, Spring 2012

Volume 29, 2011

• Volume 29, Number 2, Fall 2011
• Volume 29, Number 1, Spring 2011

Volume 28, 2009

• Volume 28, Number 2, Fall 2009
• Volume 28, Number 1, Spring 2009

Volume 27, 2008

• Volume 27, Number 2, Fall 2008
• Volume 27, Number 1, Spring 2008

Volume 26, 2007

• Volume 26, Number 2, Fall 2007
• Volume 26, Number 1, Spring 2007

Volume 25, 2006

• Volume 25, Number 2, Fall 2006
• Volume 25, Number 1, Spring 2006

Volume 24, 2005

• Volume 24, Number 2, Fall 2005
• Volume 24, Number 1, Spring 2005

Volume 23, 2004

• Volume 23, Number 2, Fall 2004
• Volume 23, Number 1, Spring 2004

Volume 22, 2003

• Volume 22, Number 2, Fall 2003
• Volume 22, Number 1, Spring 2003

Volume 21, 2002

• Volume 21, Number 2, Fall 2002
• Volume 21, Number 1, Spring 2002

Volume 20, 2001

• Volume 20, Number 2, Fall 2001
• Volume 20, Number 1, Spring 2001

Volume 19, 2000

• Volume 19, Number 2, Fall 2000
• Volume 19, Number 1, Spring 2000

Volume 18, 1999

• Volume 18, Number 2, Fall 1999
• Volume 18, Number 1, Spring 1999

Volume 17, 1998

• Volume 17, Number 2, Fall 1998
• Volume 17, Number 1, Spring 1998

Volume 16, 1997

• Volume 16, Number 2, Fall 1997
• Volume 16, Number 1, Spring 1997

Volume 15, 1996

• Volume 15, Number 2, Fall 1996
• Volume 15, Number 1, Spring 1996

Volume 14, 1995

• Volume 14, Number 2, Fall 1995
• Volume 14, Number 1, Spring 1995

Volume 13, 1994

• Volume 13, Number 2, Fall 1994
• Volume 13, Number 1, Spring 1994

Volume 12, 1993

• Volume 12, Number 2, Fall 1993
• Volume 12, Number 1, Spring 1993

Volume 11, 1992

• Volume 11, Number 2, Fall 1992
• Volume 11, Number 1, Spring 1992
• Volume 10, 1991
• Volume 9, 1990
• Volume 8, 1989
• Volume 7, 1988
• Volume 6, 1987
• Volume 5, 1986
• Volume 4, 1985
• Volume 3, 1984
• Volume 2, 1983
• Volume 1, 1982

Additional Information

Additional materials.

Additional Issue Materials

• Editorial Board -- Volume 38, Number 1, Spring 2020
• Editorial Board -- Volume 37, Number 2, Fall 2019
• Editorial Board -- Volume 37, Number 1, Spring 2019
• Editorial Board -- Volume 36, Number 2, Fall 2018
• Editorial Board -- Volume 36, Number 1, Spring 2018
• Editorial Board -- Volume 35, Number 2, Fall 2017
• Editorial Board -- Volume 35, Number 1, Spring 2017
• Editorial Board -- Volume 34, Number 2, Fall 2016
• Editorial Board -- Volume 34, Number 1, Spring 2016
• Editorial Board -- Volume 33, Number 2, Fall 2015
• Editorial Board -- Volume 33, Number 1, Spring 2015
• Editorial Board -- Volume 32, Number 2, Fall 2014
• Editorial Board -- Volume 32, Number 1, Spring 2014
• Editorial Board -- Volume 22, Number 2, Fall 2003
• Editorial Board -- Volume 22, Number 1, Spring 2003
• Editorial Board -- Volume 21, Number 2, Fall 2002
• Editorial Board -- Volume 21, Number 1, Spring 2002
• Editorial Board -- Volume 20, Number 2, Fall 2001
• Editorial Board -- Volume 20, Number 1, Spring 2001
• Editorial Board -- Volume 19, Number 2, Fall 2000
• Editorial Board -- Volume 19, Number 1, Spring 2000
• Editorial Board -- Volume 18, Number 2, Fall 1999
• Editorial Board -- Volume 18, Number 1, Spring 1999
• Editorial Board -- Volume 17, Number 2, Fall 1998
• Editorial Board -- Volume 17, Number 1, Spring 1998
• Editorial Board -- Volume 16, Number 2, Fall 1997
• Editorial Board -- Volume 16, Number 1, Spring 1997
• Editorial Board -- Volume 15, Number 2, Fall 1996
• Editorial Board -- Volume 15, Number 1, Spring 1996
• Editorial Board -- Volume 14, Number 2, Fall 1995
• Editorial Board -- Volume 14, Number 1, Spring 1995

Project MUSE Mission

Project MUSE promotes the creation and dissemination of essential humanities and social science resources through collaboration with libraries, publishers, and scholars worldwide. Forged from a partnership between a university press and a library, Project MUSE is a trusted part of the academic and scholarly community it serves.

2715 North Charles Street Baltimore, Maryland, USA 21218

+1 (410) 516-6989 [email protected]

©2024 Project MUSE. Produced by Johns Hopkins University Press in collaboration with The Sheridan Libraries.

Now and Always, The Trusted Content Your Research Requires

Built on the Johns Hopkins University Campus

This paper is in the following e-collection/theme issue:

Published on 8.5.2024 in Vol 10 (2024)

The Scope of Virtual Reality Simulators in Radiology Education: Systematic Literature Review

Authors of this article:

• Shishir Shetty 1 , PhD ; 
• Supriya Bhat 2 , MDS ; 
• Saad Al Bayatti 1 , MSc ; 
• Sausan Al Kawas 1 , PhD ; 
• Wael Talaat 1 , PhD ; 
• Mohamed El-Kishawi 3 , PhD ; 
• Natheer Al Rawi 1 , PhD ; 
• Sangeetha Narasimhan 1 , PhD ; 
• Hiba Al-Daghestani 1 , MSc ; 
• Medhini Madi 4 , MDS ; 
• Raghavendra Shetty 5 , PhD

1 Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, , Sharjah, , United Arab Emirates

2 Department of Oral Medicine and Radiology, AB Shetty Memorial Institute of Dental Sciences, Nitte (Deemed to be University), , Mangalore, , India

3 Department of Preventive and Restorative Dentistry, College of Dental Medicine, University of Sharjah, , Sharjah, , United Arab Emirates

4 Department of Oral Medicine and Radiology, Manipal College of Dental Sciences, Manipal Academy of Higher Education, , Manipal, , India

5 Department of Clinical Sciences, College of Dentistry, Ajman University, , Ajman, , United Arab Emirates

Corresponding Author:

Supriya Bhat, MDS

Background: In recent years, virtual reality (VR) has gained significant importance in medical education. Radiology education also has seen the induction of VR technology. However, there is no comprehensive review in this specific area. This review aims to fill this knowledge gap.

Objective: This systematic literature review aims to explore the scope of VR use in radiology education.

Methods: A literature search was carried out using PubMed, Scopus, ScienceDirect, and Google Scholar for articles relating to the use of VR in radiology education, published from database inception to September 1, 2023. The identified articles were then subjected to a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)–defined study selection process.

Results: The database search identified 2503 nonduplicate articles. After PRISMA screening, 17 were included in the review for analysis, of which 3 (18%) were randomized controlled trials, 7 (41%) were randomized experimental trials, and 7 (41%) were cross-sectional studies. Of the 10 randomized trials, 3 (30%) had a low risk of bias, 5 (50%) showed some concerns, and 2 (20%) had a high risk of bias. Among the 7 cross-sectional studies, 2 (29%) scored “good” in the overall quality and the remaining 5 (71%) scored “fair.” VR was found to be significantly more effective than traditional methods of teaching in improving the radiographic and radiologic skills of students. The use of VR systems was found to improve the students’ skills in overall proficiency, patient positioning, equipment knowledge, equipment handling, and radiographic techniques. Student feedback was also reported in the included studies. The students generally provided positive feedback about the utility, ease of use, and satisfaction of VR systems, as well as their perceived positive impact on skill and knowledge acquisition.

Conclusions: The evidence from this review shows that the use of VR had significant benefit for students in various aspects of radiology education. However, the variable nature of the studies included in the review reduces the scope for a comprehensive recommendation of VR use in radiology education.

Introduction

The use of technology in education helps students achieve improved acquisition of professional knowledge and practical skills [ 1 - 3 ]. Virtual reality (VR) is a modern technology that simulates experience by producing 3D interactive situations and presenting objects in a virtual world with spatial dimensions [ 4 , 5 ]. VR technology can be classified as nonimmersive or immersive [ 6 ]. In a nonimmersive VR, the simulated 3D environment is experienced through a computer monitor [ 6 ]. On the other hand, an immersive VR provides a sense of presence in a computer-generated environment, created by producing realistic sights, sounds, and other sensations that replicate a user’s physical presence in a virtual environment [ 6 , 7 ]. Using VR technology, a person can look about the artificial world, navigate around in it, and interact with simulated objects or items [ 5 , 8 ]. Due to the broad nature of VR technology, it has many applications, some of which are in the field of medicine [ 9 , 10 ].

The use of VR in medicine started in the 1990s when medical researchers were trying to create 3D models of patients’ internal organs [ 11 - 13 ]. Since then, VR use in the field of medicine and general health care has increased substantially to cover many areas including medical education. Radiology education has also come to see the use of VR technology in the recent past [ 14 ]. The use of VR in radiology education enables students to practice radiography in a virtual environment, which is radiation free [ 15 ]. Additionally, the use of VR enables effective and repeatable training. This allows trainees to recognize and correct errors as they occur [ 16 , 17 ]. The aim of this review is to explore the scope of VR in radiology education.

This systematic review has been performed using the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) guidelines [ 18 ] [ Checklist 1 ]).

Information Sources and Study Selection

The bibliographic databases used were PubMed, Scopus, ScienceDirect, and Google Scholar. A systematic literature search was conducted for articles published from database inception to September 1, 2023. Topic keywords were used to generate search strings. The search strings that were used are provided in Table 1 . Only the first 10 pages of Google Scholar results were exported. The identified studies were then subjected to a study selection process. The search string for ScienceDirect was shorter because the database only allows a maximum of 8 Boolean operators, hence the sting had to be shortened. The search in PubMed was limited to the title and abstract. The searches in Scopus and ScienceDirect were limited to title, abstract, and keywords.

Inclusion and Exclusion Criteria

Original research articles written in the English language were included in the review. Studies conducted on medical, dental, and allied health sciences students (undergraduate and postgraduate) from any part of the world were included in the review. Studies exploring the use of VR learning in radiology education were included.

Narrative reviews, scoping reviews, systematic reviews, meta-analyses, editorials, and commentaries were excluded. Studies that did not align with the required study objective were excluded.

Method of Quality Assessment

Randomized controlled trials (RCTs) and randomized experimental studies were appraised using the RoB 2 tool from the Cochrane Collaboration [ 19 ]. A visualization of the risk-of-bias assessment was done using the web-based robvis tool [ 20 ]. Cross-sectional studies were appraised using the appraisal checklist for analytical cross-sectional studies from the Joanna Briggs Institute [ 21 ].

Data Extraction

Each article included in the review was summarized in a table, including basic study characteristics. The extracted attributes were study author(s), publication year, study design, type and number of participants, type of radiology education under study, and the outcome being assessed. The extracted data are provided in Table 2 .

a RCT: randomized controlled trial.

b CT: computed tomography.

Search Results

The database search identified a total of 2877 studies; 374 (13%) studies were from PubMed, 2169 (75.4%) were from Scopus, 234 (8.1%) were from ScienceDirect, and 100 (3.5%) were from Google Scholar. Before the screening procedure, 37 duplicates were removed. During title and abstract screening, 2808 articles were excluded since they did not align with the eligibility criteria. The remaining 32 articles were then subjected to a full-text review, and 15 were excluded for reasons provided in Figure 1 , which shows the study selection process [ 38 ]. At the end of the process, 17 studies were found eligible for inclusion in the review.

Characteristics of Included Studies

Among the 17 studies, 3 (18%) RCTs, 7 (41%) randomized experimental trials, and 7 (41%) cross-sectional studies were included. The studies encompassed various aspects of radiology education, including dental radiology [ 28 , 29 ], diagnostic radiology [ 22 , 24 ], and interventional radiology [ 25 , 31 ].

Results of Quality Assessment

Among the 7 cross-sectional studies, 2 (29%) scored “good” in overall quality and the remaining 5 (71%) scored “fair.” The results for the quality appraisal of cross-sectional studies are shown in Table 3 . Studies were appraised using the checklist for analytical cross-sectional studies from the Joanna Briggs Institute [ 21 ].

Among the 10 randomized trials, 3 (30%) had a low risk of bias, 5 (50%) showed some concerns, and 2 (20%) had a high risk of bias. These results are shown in Table 4 . RCTs were appraised using the RoB 2 tool from the Cochrane Collaboration [ 19 ]. A risk-of-bias graph ( Figure 2 ) and a risk-of-bias summary ( Figure 3 ) are also provided.

a Item 1: were the criteria for inclusion in the sample clearly defined?

b Item 2: were the study subjects and the setting described in detail?

c Item 3: was the exposure measured in a valid and reliable way?

d Item 4: were objective, standard criteria used for measurement of the condition?

e Item 5: were confounding factors identified?

f Item 6: were strategies to deal with confounding factors stated?

g Item 7: were the outcomes measured in a valid and reliable way?

h Item 8: was appropriate statistical analysis used?

i N/A: not assessable.

a D1: risk of bias arising from the randomization process.

b D2: risk of bias due to deviations from the intended interventions (effect of assignment to intervention).

c D3: risk of bias due to missing outcome data.

d D4: risk of bias in measurement of the outcome.

e D5: risk of bias in selection of the reported result.

Type of VR Hardware and Software Used in the Studies

The studies used a wide range of VR software and hardware. Some of the studies used 3D simulation software packages displayed on 2D desktop computers [ 22 , 24 , 25 , 36 ], whereas others used headsets for an immersive VR environment [ 15 , 23 , 26 , 35 , 37 ]. The most used VR teaching software were the CETSOL VR Clinic software [ 33 , 35 ], Virtual Medical Coaching VR software [ 15 , 30 , 32 ], Projection VR (Shaderware) software [ 36 ], SieVRt VR system (Luxsonic Technologies) [ 37 ], medical imaging training immersive environment software [ 23 ], VR CT Sim software [ 25 ], VitaSim ApS software [ 26 ], VR X-Ray (Skilitics and Virtual Medical Coaching) software [ 27 ], and radiation dosimetry VR software (Virtual Medical Coaching Ltd) [ 31 ].

Effect of VR Teaching on Skill Acquisition

Ahlqvist et al [ 22 ] looked at how virtual simulation can be used as an effective tool to teach quality assessment of radiographic images. They also compared how it faired in comparison to traditional teaching. The study reported a statistically significant improvement in proficiency from before training to after training. Additionally, the study reported that the proficiency score improvement for the VR-trained students was higher than that for the students trained using conventional method.

In the study conducted by Sapkaroski et al [ 34 ], students in the VR group demonstrated significantly better patient positioning skills compared to those in the conventional role-play group. The positioning parameters that were assessed were digit separation and palm flatness (the VR group scored 11% better), central ray positioning onto the third metacarpophalangeal joint (the VR group scored 23% better), and a control position projection of an oblique hand. The results for the control position projection indicated no significant difference in positioning between the 2 groups [ 34 ].

Bridge et al [ 23 ] also performed a performance comparison between students trained by VR and traditional methods. They assessed skills about patient positioning, equipment positioning, and time taken to complete a performative role-play. Students in the VR group performed better than those in the control group, with 91% of them receiving an overall score of above average (>3). The difference in mean group performance was statistically significant ( P =.0366). Similarly, Gunn et al [ 24 ] reported improved and higher role-play skill scores for students trained using VR software simulation compared to those trained on traditional laboratory simulation. The mean role-play score for the VR group was 30.67 and that for the control group was 28.8 [ 24 ].

Another study reported that students trained using VR performed significantly better (ranked as “very good” or “excellent”) than the control group (conventional learning) in skills such as patient positioning, selecting exposure factors, centering and collimating the x-ray beam, placing the anatomical marker, appraisal of image quality, equipment positioning, and procedure explanation to the patient [ 30 ]. Another recently conducted study found that the VR-taught group achieved better test duration and fewer errors in moving equipment and positioning a patient. There was no significant difference in the frequency of errors in the radiographic exposure setting such as source-to-image distance between the VR and the physical simulation groups [ 32 ].

Nilsson et al [ 28 ] developed a test to evaluate the student’s ability to interpret 3D information in radiographs using parallax. This test was applied to students before and after training. There was a significantly larger ( P <.01) pre-post intervention mean score for the VR group (3.11 to 4.18) compared to the control group (3.24 to 3.72). A subgroup analysis was also performed, and students with low visuospatial ability in the VR group had a significantly higher improvement in the proficiency test compared to those in the control group. The same authors conducted another follow-up study to test skill retention [ 29 ]. Net skill improvement was calculated as the difference in test scores after 8 months. The results from the proficiency test showed that the ability to interpret spatial relations in radiographs 8 months after the completion of VR training was significantly better than before VR training. The students who trained conventionally showed almost the same positive trend in improvement. The group difference was smaller and not statistically significant. This meant that, 8 months after training, the VR group and the traditionally trained group had the same skill level [ 29 ].

Among the included studies, only 1 reported that the VR group had lower performance in proficiency tests and radiographic skill tests, compared to a conventionally trained group. The study, conducted in 2022, showed that the proficiency of the VR group was significantly lower than that of the conventional technique group in performing lateral elbow and posterior-anterior chest radiography [ 27 ]. An itemized rubric evaluation used in the study revealed that the VR group also had lower performance in most of the radiographic skills, such as locating and centering of the x-ray beam, side marker placement, positioning the x-ray image detector, patient interaction, and process control and safety [ 27 ]. The study concluded that VR simulation can be less effective than real-world training in radiographic techniques, which requires palpation and patient interaction. These results may be different from those of other studies due to different outcome evaluation methods and since they used head-mounted display VR coaching, whereas the other studies, except O’Connor et al [ 15 ], used VR on a PC monitor.

All of the studies except Kato et al [ 27 ] agreed that VR use was more effective for students in developing radiographic and radiologic skills. Despite this general agreement, there were slight in-study variations in learning outcomes, which made some of the studies look at factors that may influence skill and knowledge acquisition during VR use. In studies such as Bridge et al [ 23 ], it was noted that the arrangement of equipment had the greatest influence on the overall score. After performing a multivariable analysis, Gunn et al [ 24 ] reported that there was no effect of age, gender, and gaming skills or activity on the outcome of VR learning. In the study by Shanahan [ 36 ], a few students (19/84, 23%) had previously used VR simulation software. This had no bearing on the learning outcomes. Another observation in the same study was that student age was found to significantly affected the student’s confidence about skill acquisition after VR training [ 36 ].

Students’ Perception of VR Uses for Learning

The findings from the study by Gunn et al [ 25 ] revealed that 68% of students agreed or strongly agreed that VR simulation was significantly helpful in learning about computed tomography (CT) scanning. In another study by Jensen et al [ 26 ], 90% of the students strongly agreed that VR simulators could contribute to learning radiography, with 90% reporting that the x-ray equipment in the VR simulation was realistic. In the study by Wu et al [ 37 ], most of the students (55.6%) agreed or somewhat agreed that VR use was useful in radiology education. Similarly, 83% of the students in Shanahan’s [ 36 ] study regarded VR learning with an ease of use. In the same study, students also reported that one of the major benefits of VR learning include using the simulation to repeat activities until being satisfied with the results (95% of respondents). Students also stated that VR enabled them to quickly see images and understand if changes needed to be made (94%) [ 36 ]. In the study by Gunn et al [ 25 ], 75% of medical imaging students agreed on the ease of use and software enjoyment in VR simulated learning. In the same study, 57% of the students reported a positive perceived usefulness of VR. Most respondents (80%) in the study by Rainford et al [ 31 ] favored the in-person VR experience over web-based VR. Similarly, 58% of the respondents in the study conducted by O’Connor et al [ 15 ] reported enjoying learning using VR simulation. In the study by Wu et al [ 37 ], 83.3% of students agreed or strongly agreed that they enjoyed using VR for learning. Similarly, the studies by Rainford et al [ 31 ] and O’Connor et al [ 15 ] reported student recommendation of 87% and 94%, respectively, for VR as a learning tool.

Students’ Perceived Skill and Knowledge Acquisition

In the study by Bridge et al [ 23 ], students who trained using VR reported an increase in perceived skill acquisition and high levels of satisfaction. The study authors attributed this feedback to the availability of “gold standards” that showed correct positioning techniques, as well as instant feedback provided by the VR simulators. Gunn et al [ 25 ] examined students’ confidence in performing a CT scan in a real clinical environment after using VR simulations as a learning tool. The study reported an increase (from before to after training) in the students’ perceived confidence in performing diagnostic CT scans. Similarly, the study by Jensen et al [ 26 ] reported that the use of VR had influenced students’ self-perceived readiness to perform wrist x-ray radiographs. The study, however, found no significant difference in pre- and posttraining (perceived preparedness) scores. The pre- and posttraining scores were 75 (95% CI 54-96) and 77 (95% CI 59-95), respectively. The study by O’Connor et al [ 15 ] looked at the effect of VR on perceived skill adoption. Most of the students in the study reported high levels of perceived knowledge acquisition in the areas of beam collimation, anatomical marker placement, centering of the x-ray tube, image evaluation, anatomical knowledge, patient positioning, and exposure parameter selection to their VR practice. However, most students felt that VR did not contribute to their knowledge of patient dose tracking and radiation safety [ 15 ]. In the study by Rainford et al [ 31 ], 73% of radiography and medical students felt that VR learning increased their confidence across all relevant learning outcomes. The biggest increase in confidence level was regarding their understanding of radiation safety matters [ 31 ]. Sapkaroski et al [ 33 ] performed a self-perception test to see how students viewed their clinical and technical skills after using VR for learning. In their study, students reported a perceived improvement in their hand and patient positioning skills. Their study also compared 2 software, CETSOL VR Clinic and Shaderware. The cohort who used CETSOL VR Clinic had higher scores on perceived improvement [ 33 ]. Sapkaroski et al [ 35 ] compared the student’s perception scores on the educational enhancement of their radiographic hand positioning skills, after VR or clinical role-play scenario training. Although the VR group scored higher, there was no significant difference between the scores for the 2 groups [ 35 ]. In the study by Shanahan [ 36 ], when the perception of skill development was evaluated, most of the students reported that the simulation positively developed their technical (78%), radiographic image evaluation (85%), problem-solving (85%), and self-evaluation (88%) abilities. However, in the study by Kato et al [ 27 ], there was no difference in the perceived acquisition of knowledge among students using traditional teaching and VR-based teaching.

Principal Findings

The results presented in this review reveal strong evidence for the effectiveness of VR teaching in radiology education, particularly in the context of skill acquisition and development [ 22 , 24 , 27 , 30 , 32 , 34 ].

In this review, quality appraisal of the cross-sectional studies revealed that the strategies for deal with confounding factors was one of the factors directly affecting the reliability of the results. Similarly, the appraisal of the randomized trials revealed that the bias arising due to missing outcome data was one of the factors directly affecting the reliability of the results.

All the studies found that VR-based teaching had a positive impact on various areas of radiographic and radiologic skill development. In comparison to the traditional way of teaching, only 1 study by Kato et al [ 27 ] reported VR teaching as inferior to traditional teaching. The studies consistently reported better improvements in proficiency, patient positioning outcomes, equipment handling, and radiographic techniques among students trained using VR. According to Nilsson et al [ 29 ], O’Connor et al [ 15 ], and Wu et al [ 37 ], the improvements were due to the immersive and interactive nature of VR simulations, which allowed learners to engage with radiological scenarios in a dynamic and hands-on manner. The studies also revealed that VR learning has the ability to easily and effectively introduce students to new skills. It was also found that existing skills could be improved, mainly through simulation feedback that happens in real time during training [ 22 , 24 , 28 , 30 , 36 ].

The improvement of skills after VR training have been noted in different domains, including patient positioning, equipment positioning, equipment knowledge, assessment of radiographic image quality, and patient interaction. Improvement was also observed in other skills such as as central ray positioning, source-to-image distance, image receptor placement, and side marker placement [ 22 , 24 , 30 , 32 , 34 ]. Two studies, Nilsson et al [ 28 ] and Nilsson et al [ 29 ], looked at how VR affected the students’ ability to interpret 3D information in radiographs using parallax. They both reported a positive effect. Nilsson et al [ 29 ] also gave insights into the long-term benefits of VR training in radiology. Eight months after training, the control (traditionally taught) group in Nilsson et al [ 29 ] showed a slight increase in skills, but the VR-trained group still maintained a significantly higher skill level. This finding shows the enduring impact of VR-based education on skill acquisition in radiology. Although most studies supported the effectiveness of VR in radiology education, 1 study reported contrasting results [ 27 ]. VR-trained students were found to perform worse than traditionally trained students in conducting lateral elbow and posterior-anterior chest radiography in Kato et al [ 27 ]. This difference in results was, according to the authors, attributed to the use of a different rubric evaluation method and the use of a head-mounted display–based immersive VR system, which was not used in other studies. These 2 reasons may be the reason for the variation in study findings.

A wide range of VR software with different functions were used in the studies. In addition to acquiring radiographic images, the CETSOL VR Clinic software facilitated students to interact with their learning environment [ 33 , 35 ]. Students using the Virtual Medical Coaching VR software performed imaging exercise on a virtual patient with VR headsets and hand controllers [ 15 , 30 , 32 ]. The SieVRt VR system displayed Digital Imaging and Communications in Medicine format images in a virtual environment, thus facilitating teaching [ 37 ]. The medical imaging training immersive environment simulation software provided automated feedback to the learners including a rerun of procedures, thus highlighting procedural errors [ 23 ]. The VR CT Sim software allowed the student virtually to perform the complete CT workflow [ 25 ]. Students could manipulate patient positioning and get feedback from the VitaSim ApS software [ 26 ]. The VR X-Ray software allowed students to manipulate radiographic equipment and patient’s position with a high level of immersive experience [ 27 ]. Radiation dosimetry VR software facilitated virtual movement of the staff and equipment to radiation-free areas, thus optimizing radiation protection [ 31 ].

The included studies also looked at factors that could influence skill acquisition when VR is used in radiology education. Bridge et al [ 23 ], Gunn et al [ 24 ], Kato et al [ 27 ], and Shanahan [ 36 ] investigated factors such as age, gender, prior gaming experience, and familiarity with VR technology. However, these factors were shown to have no significant effect on VR learning outcomes. This shows that VR education can equally accommodate a wide range of learners, regardless of experience or existing attributes.

Across several studies, positive feedback emerged regarding the utility, ease of use, enjoyment, and perceived impact on skill and knowledge acquisition. The included studies consistently reported positive perceptions of VR use among students [ 25 , 26 , 37 ]. Gunn et al [ 25 ] reported that a significant proportion of medical imaging and radiation therapy students found the use of VR simulation to be significantly helpful in learning about CT scanning. Similarly, Jensen et al [ 26 ] and Wu et al [ 37 ] reported that a majority of students agreed on the usefulness of VR in radiology education. Another aspect that received positive feedback was the ease of use. Students liked the ability to repeat tasks until they were satisfied with the results and the ability to quickly visualize radiographs to determine the need for revisions [ 36 ]. Rainford et al [ 31 ] and O’Connor and Rainford [ 30 ] found that most students would recommend VR as a learning tool to other students.

Several studies investigated student’s perceptions of skill and knowledge acquisition when using VR for radiology education. Bridge et al [ 15 ] and O’Connor et al [ 23 ] discovered an increase in students’ perceived acquisition of radiographic skills. Gunn et al [ 25 ] reported an increase in students’ perceived confidence to perform CT scans after learning using VR simulations. According to Rainford et al [ 31 ], a large percentage of radiography and medical students felt that VR learning boosted their confidence across all relevant learning outcomes, with the highest levels of confidence recorded in radiation safety. Sapkaroski et al [ 33 ] discovered that after using VR for learning, students experienced an improvement in their hand and patient placement skills. In summary, the positive feedback from the students shows that VR use in radiology education is a useful, engaging, and effective teaching tool. This perceived acquisition of skills is backed by the results from the proficiency tests.

The VR modalities used in some of the studies allowed remote assistance from an external agent (teacher), as the VR training is conducted in front of a screen while being part of a team, with the teacher making constant corrections and indications [ 22 , 24 , 27 ]. However, researchers are looking into VR systems with artificial intelligence–supported tutoring, which includes the assessment of learners, generation of learning content, and automated feedback [ 39 ].

Findings from the included studies show that VR-based teaching offers substantial benefits in various aspects of radiographic and radiologic skill development. The studies consistently reported that students educated using VR systems improved significantly in overall proficiency, patient positioning, equipment knowledge, equipment handling, and radiographic techniques. However, the variable nature of the studies included in the review reduces the scope for a comprehensive recommendation of VR use in radiology education. A key contributing factor to relatively better learning outcomes was the immersive and interactive nature of VR systems, which provided real-time feedback and dynamic learning experiences to students. Factors such as age, gender, gaming experience, and familiarity with VR systems did not significantly influence learning outcomes. This shows that VR can be used for diverse groups of students when teaching radiology. Students generally provided positive feedback about the utility, ease of use, and satisfaction of VR, as well as its perceived impact on skill and knowledge acquisition. These students’ reports show the value of VR as an important, interesting, and effective tool in radiology education.

Conflicts of Interest

None declared.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

• Ding X, Li Z. A review of the application of virtual reality technology in higher education based on Web of Science literature data as an example. Front Educ. Nov 17, 2022;7:1048816. [ CrossRef ]
• Oyelere SS, Bouali N, Kaliisa R, Obaido G, Yunusa AA, Jimoh ER. Exploring the trends of educational virtual reality games: a systematic review of empirical studies. Smart Learning Environments. Oct 19, 2020;7:31. [ CrossRef ]
• Zamyatina NV, Ushakova IA, Mandrikov VB. Medical education in digital transformation. In: Proceedings of the International Scientific Conference “Digitalization of Education: History, Trends and Prospects” (DETP 2020). Atlantis Press; 2020;324-327. [ CrossRef ]
• Kardong-Edgren S, Farra SL, Alinier G, Young HM. A call to unify definitions of virtual reality. Clin Simul Nurs. Jun 2019;31:28-34. [ CrossRef ]
• Sousa Santos B, Dias P, Pimentel A, et al. Head-mounted display versus desktop for 3D navigation in virtual reality: a user study. Multimed Tools Appl. Jan 2009;41(1):161-181. [ CrossRef ]
• Hamad A, Jia B. How virtual reality technology has changed our lives: an overview of the current and potential applications and limitations. Int J Environ Res Public Health. Sep 8, 2022;19(18):11278. [ CrossRef ] [ Medline ]
• Kaplan-Rakowski R, Meseberg K. Immersive media and their future. In: Branch RM, Lee H, Tseng S, editors. Educational Media and Technology Yearbook. Vol 42. Springer; 2019;143-153. [ CrossRef ]
• Kuliga SF, Thrash T, Dalton RC, Hölscher C. Virtual reality as an empirical research tool — exploring user experience in a real building and a corresponding virtual model. Comput Environ Urban Syst. Nov 2015;54:363-375. [ CrossRef ]
• Guan H, Xu Y, Zhao D. Application of virtual reality technology in clinical practice, teaching, and research in complementary and alternative medicine. Evid Based Complement Alternat Med. Aug 11, 2022;2022:1373170. [ CrossRef ] [ Medline ]
• Pottle J. Virtual reality and the transformation of medical education. Future Healthc J. Oct 2019;6(3):181-185. [ CrossRef ] [ Medline ]
• Satava RM, Jones SB. Current and future applications of virtual reality for medicine. Proc IEEE. Mar 1998;86(3):484-489. [ CrossRef ]
• Satava RM. Medical virtual reality. the current status of the future. Stud Health Technol Iform. 1996;29:100-106. [ Medline ]
• Rosenberg LB, Stredney D. A haptic interface for virtual simulation of endoscopic surgery. Stud Health Technol Inform. 1996;29:371-387. [ Medline ]
• Chytas D, Salmas M, Demesticha T, et al. A review of the use of virtual reality for teaching radiology in conjunction with anatomy. Cureus. Dec 5, 2021;13(12):e20174. [ CrossRef ] [ Medline ]
• O’Connor M, Stowe J, Potocnik J, Giannotti N, Murphy S, Rainford L. 3D virtual reality simulation in radiography education: the students' experience. Radiography (Lond). Feb 2021;27(1):208-214. [ CrossRef ] [ Medline ]
• Hart R, Karthigasu K. The benefits of virtual reality simulator training for laparoscopic surgery. Curr Opin Obstet Gynecol. Aug 2007;19(4):297-302. [ CrossRef ] [ Medline ]
• Thomas DJ, Singh D. Letter to the editor: virtual reality in surgical training. Int J Surg. May 2021;89:105935. [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372(71):n71. [ CrossRef ] [ Medline ]
• Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. Aug 28, 2019;366:l4898. [ CrossRef ] [ Medline ]
• McGuinness LA, Higgins JPT. Risk-of-bias visualization (robvis): an R package and shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. Jan 2021;12(1):55-61. [ CrossRef ] [ Medline ]
• Critical appraisal tools. JBI. URL: https://jbi.global/critical-appraisal-tools [Accessed 2024-04-25]
• Ahlqvist JB, Nilsson TA, Hedman LR, et al. A randomized controlled trial on 2 simulation-based training methods in radiology: effects on radiologic technology student skill in assessing image quality. Simul Healthc. Dec 2013;8(6):382-387. [ CrossRef ] [ Medline ]
• Bridge P, Gunn T, Kastanis L, et al. The development and evaluation of a medical imaging training immersive environment. J Med Radiat Sci. Sep 2014;61(3):159-165. [ CrossRef ] [ Medline ]
• Gunn T, Jones L, Bridge P, Rowntree P, Nissen L. The use of virtual reality simulation to improve technical skill in the undergraduate medical imaging student. Interactive Learning Environments. 2018;26(5):613-620. [ CrossRef ]
• Gunn T, Rowntree P, Starkey D, Nissen L. The use of virtual reality computed tomography simulation within a medical imaging and a radiation therapy undergraduate programme. J Med Radiat Sci. Mar 2021;68(1):28-36. [ CrossRef ] [ Medline ]
• Jensen J, Graumann O, Jensen RO, et al. Using virtual reality simulation for training practical skills in musculoskeletal wrist x-ray - a pilot study. J Clin Imaging Sci. Jul 11, 2023;13:20. [ CrossRef ] [ Medline ]
• Kato K, Kon D, Ito T, Ichikawa S, Ueda K, Kuroda Y. Radiography education with VR using head mounted display: proficiency evaluation by rubric method. BMC Med Educ. Jul 28, 2022;22(1):579. [ CrossRef ] [ Medline ]
• Nilsson TA, Hedman LR, Ahlqvist JB. A randomized trial of simulation-based versus conventional training of dental student skill at interpreting spatial information in radiographs. Simul Healthc. 2007;2(3):164-169. [ CrossRef ] [ Medline ]
• Nilsson TA, Hedman LR, Ahlqvist JB. Dental student skill retention eight months after simulator-supported training in oral radiology. J Dent Educ. May 2011;75(5):679-684. [ Medline ]
• O’Connor M, Rainford L. The impact of 3D virtual reality radiography practice on student performance in clinical practice. Radiography (Lond). Jan 2023;29(1):159-164. [ CrossRef ] [ Medline ]
• Rainford L, Tcacenco A, Potocnik J, et al. Student perceptions of the use of three-dimensional (3-D) virtual reality (VR) simulation in the delivery of radiation protection training for radiography and medical students. Radiography (Lond). Jul 2023;29(4):777-785. [ CrossRef ] [ Medline ]
• Rowe D, Garcia A, Rossi B. Comparison of virtual reality and physical simulation training in first-year radiography students in South America. J Med Radiat Sci. Jun 2023;70(2):120-126. [ CrossRef ] [ Medline ]
• Sapkaroski D, Baird M, McInerney J, Dimmock MR. The implementation of a haptic feedback virtual reality simulation clinic with dynamic patient interaction and communication for medical imaging students. J Med Radiat Sci. Sep 2018;65(3):218-225. [ CrossRef ] [ Medline ]
• Sapkaroski D, Baird M, Mundy M, Dimmock MR. Quantification of student radiographic patient positioning using an immersive virtual reality simulation. Simul Healthc. Aug 2019;14(4):258-263. [ CrossRef ] [ Medline ]
• Sapkaroski D, Mundy M, Dimmock MR. Virtual reality versus conventional clinical role-play for radiographic positioning training: a students' perception study. Radiography (Lond). Feb 2020;26(1):57-62. [ CrossRef ] [ Medline ]
• Shanahan M. Student perspective on using a virtual radiography simulation. Radiography (Lond). Aug 2016;22(3):217-222. [ CrossRef ]
• Wu Y, Mondal P, Stewart M, Ngo R, Burbridge B. Bringing radiology education to a new reality: a pilot study of using virtual reality as a remote educational tool. Can Assoc Radiol J. May 2023;74(2):251-263. [ CrossRef ] [ Medline ]
• Haddaway NR, Page MJ, Pritchard CC, McGuinness LA. PRISMA2020: an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and open synthesis. Campbell Syst Rev. Mar 27, 2022;18(2):e1230. [ CrossRef ] [ Medline ]
• King S, Boyer J, Bell T, Estapa A. An automated virtual reality training system for teacher-student interaction: a randomized controlled trial. JMIR Serious Games. Dec 8, 2022;10(4):e41097. [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Hasan Sapci, Taiane de Azevedo Cardoso; submitted 23.09.23; peer-reviewed by FernandezHerrero Jorge, Stacey Kassutto; final revised version received 01.02.24; accepted 31.03.24; published 08.05.24.

© Shishir Shetty, Supriya Bhat, Saad Al Bayatti, Sausan Al Kawas, Wael Talaat, Mohamed El-Kishawi, Natheer Al Rawi, Sangeetha Narasimhan, Hiba Al-Daghestani, Medhini Madi, Raghavendra Shetty. Originally published in JMIR Medical Education (https://mededu.jmir.org), 8.5.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/ , as well as this copyright and license information must be included.

Department of Medicine

A community resource for faculty and staff

• Digestive Diseases

Statins in chronic liver disease: review of literature and future role

Pham N, Benhammou JN. Statins in chronic liver disease: review of literature and future role. Semin Liver Dis. 2024 May 3. doi: 10.1055/a-2319-0694. Epub ahead of print. PMID: 38701856.

Tags: Digestive Diseases , General Internal Medicine , Publications , Resident , Veterans Administration

Related Posts

Singh J, Ibrahim B, Jackson NJ, Khalil H, Valenzuela J, Yanny B, Saab S. Lower FIB-4 threshold in patients with diabetes improves diagnostic accuracy of[...]

Sharafeddin F, Sierra J, Ghaly M, Simon TB, Ontiveros-Ángel P, Edelbach B, Febo M, Labus J, Figueroa JD. Role of the prefrontal cortical protease TACE/ADAM17[...]

Brahmania M, Rogal S, Serper M, Patel A, Goldberg D, Mathur A, Wilder J, Vittorio J, Yeoman A, Rich NE, Lazo M, Kardashian A, Asrani[...]

• Open access
• Published: 09 May 2024

Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research

• Jingcheng Du 1 ,
• Ekin Soysal 1 , 3 ,
• Dong Wang 2 ,
• Long He 1 ,
• Bin Lin 1 ,
• Jingqi Wang 1 ,
• Frank J. Manion 1 ,
• Yeran Li 2 ,
• Elise Wu 2 &
• Lixia Yao 2  

BMC Medical Research Methodology volume  24 , Article number:  108 ( 2024 ) Cite this article

Metrics details

Systematic literature reviews (SLRs) are critical for life-science research. However, the manual selection and retrieval of relevant publications can be a time-consuming process. This study aims to (1) develop two disease-specific annotated corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases (PAPD), and (2) optimize machine- and deep-learning models to facilitate automation of the SLR abstract screening.

This study constructed two disease-specific SLR screening corpora for HPV and PAPD, which contained citation metadata and corresponding abstracts. Performance was evaluated using precision, recall, accuracy, and F1-score of multiple combinations of machine- and deep-learning algorithms and features such as keywords and MeSH terms.

Results and conclusions

The HPV corpus contained 1697 entries, with 538 relevant and 1159 irrelevant articles. The PAPD corpus included 2865 entries, with 711 relevant and 2154 irrelevant articles. Adding additional features beyond title and abstract improved the performance (measured in Accuracy) of machine learning models by 3% for HPV corpus and 2% for PAPD corpus. Transformer-based deep learning models that consistently outperformed conventional machine learning algorithms, highlighting the strength of domain-specific pre-trained language models for SLR abstract screening. This study provides a foundation for the development of more intelligent SLR systems.

Peer Review reports

Introduction

Systematic literature reviews (SLRs) are an essential tool in many areas of health sciences, enabling researchers to understand the current knowledge around a topic and identify future research and development directions. In the field of health economics and outcomes research (HEOR), SLRs play a crucial role in synthesizing evidence around unmet medical needs, comparing treatment options, and preparing the design and execution of future real-world evidence studies. SLRs provide a comprehensive and transparent analysis of available evidence, allowing researchers to make informed decisions and improve patient outcomes.

Conducting a SLR involves synthesizing high-quality evidence from biomedical literature in a transparent and reproducible manner, and seeks to include all available evidence on a given research question, and provides some assessment regarding quality of the evidence [ 1 , 2 ]. To conduct an SLR one or more bibliographic databases are queried based on a given research question and a corresponding set of inclusion and exclusion criteria, resulting in the selection of a relevant set of abstracts. The abstracts are reviewed, further refining the set of articles that are used to address the research question. Finally, appropriate data is systematically extracted from the articles and summarized [ 1 , 3 ].

The current approach to conducting a SLR is through manual review, with data collection, and summary done by domain experts against pre-specified eligibility criteria. This is time-consuming, labor-intensive, expensive, and non-scalable given the current more-than linear growth of the biomedical literature [ 4 ]. Michelson and Reuter estimate that each SLR costs approximately $141,194.80 and that on average major pharmaceutical companies conduct 23.36 SLRs, and major academic centers 177.32 SLRs per year, though the cost may vary based on the scope of different reviews [ 4 ]. Clearly automated methods are needed, both from a cost/time savings perspective, and for the ability to effectively scan and identify increasing amounts of literature, thereby allowing the domain experts to spend more time analyzing the data and gleaning the insights.

One major task of SLR project that involves large amounts of manual effort, is the abstract screening task. For this task, selection criteria are developed and the citation metadata and abstract for articles tentatively meeting these criteria are retrieved from one or more bibliographic databases (e.g., PubMed). The abstracts are then examined in more detail to determine if they are relevant to the research question(s) and should be included or excluded from further consideration. Consequently, the task of determining whether articles are relevant or not based on their titles, abstracts and metadata can be treated as a binary classification task, which can be addressed by natural language processing (NLP). NLP involves recognizing entities and relationships expressed in text and leverages machine-learning (ML) and deep-learning (DL) algorithms together with computational semantics to extract information. The past decade has witnessed significant advances in these areas for biomedical literature mining. A comprehensive review on how NLP techniques in particular are being applied for automatic mining and knowledge extraction from biomedical literature can be found in Zhao et al. [ 5 ].

Materials and methods

The aims of this study were to: (1) identify and develop two disease-specific corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases suitable for training the ML and DL models underlying the necessary NLP functions; (2) investigate and optimize the performance of the ML and DL models using different sets of features (e.g., keywords, Medical Subject Heading (MeSH) terms [ 6 ]) to facilitate automation of the abstract screening tasks necessary to construct a SLR. Note that these screening corpora can be used as training data to build different NLP models. We intend to freely share these two corpora with the entire scientific community so they can serve as benchmark corpora for future NLP model development in this area.

SLR corpora preparation

Two completed disease-specific SLR studies by Merck & Co., Inc., Rahway, NJ, USA were used as the basis to construct corpora for abstract-level screening. The two SLR studies were both relevant to health economics and outcome research, including one for human papillomavirus (HPV) associated diseases (referred to as the HPV corpus), and one for pneumococcal-associated pediatric diseases (which we refer to as the PAPD corpus). Both of the original SLR studies contained literature from PubMed/MEDLINE and EMBASE. Since we intended for the screening corpora to be released to the community, we only kept citations found from PubMed/MEDLINE in the finalized corpora. Because the original SLR studies did not contain the PubMed ID (PMID) for each article, we matched each article’s citation information (if available) against PubMed and then collected meta-data such as authors, journals, keywords, MeSH terms, publication types, etc., using PubMed Entrez Programming Utilities (E-utilities) Application Programming Interface (API). The detailed description of the two corpora can be seen in Table  1 . Both of the resulting corpora are publicly available at [ https://github.com/Merck/NLP-SLR-corpora ].

Machine learning algorithms

Although deep learning algorithms have demonstrated superior performance on many NLP tasks, conventional machine learning algorithms have certain advantages, such as low computation costs and faster training and prediction speed.

We evaluated four traditional ML-based document classification algorithms, XGBoost [ 7 ], Support Vector Machines (SVM) [ 8 ], Logistic regression (LR) [ 9 ], and Random Forest [ 10 ] on the binary inclusion/exclusion classification task for abstract screening. Salient characteristics of these models are as follows:

XGBoost: Short for “eXtreme Gradient Boosting”, XGBoost is a boosting-based ensemble of algorithms that turn weak learners into strong learners by focusing on where the individual models went wrong. In Gradient Boosting, individual weak models train upon the difference between the prediction and the actual results [ 7 ]. We set max_depth at 3, n_estimators at 150 and learning rate at 0.7.

Support vector machine (SVM): SVM is one of the most robust prediction methods based on statistical learning frameworks. It aims to find a hyperplane in an N-dimensional space (where N = the number of features) that distinctly classifies the data points [ 8 ]. We set C at 100, gamma at 0.005 and kernel as radial basis function.

Logistic regression (LR): LR is a classic statistical model that in its basic form uses a logistic function to model a binary dependent variable [ 9 ]. We set C at 5 and penalty as l2.

Random forest (RF): RF is a machine learning technique that utilizes ensemble learning to combine many decision trees classifiers through bagging or bootstrap aggregating [ 10 ]. We set n_estimators at 100 and max_depth at 14.

These four algorithms were trained for both the HPV screening task and the PAPD screening task using the corresponding training corpus.

For each of the four algorithms, we examined performance using (1) only the baseline feature criteria (title and abstract of each article), and (2) with five additional meta-data features (MeSH, Authors, Keywords, Journal, Publication types.) retrieved from each article using the PubMed E-utilities API. Conventionally, title and abstract are the first information a human reviewer would depend on when making a judgment for inclusion or exclusion of an article. Consequently, we used title and abstract as the baseline features to classify whether an abstract should be included at the abstract screening stage. We further evaluated the performance with additional features that can be retrieved by PubMed E-utilities API, including MeSH terms, authors, journal, keywords and publication type. For baseline evaluation, we concatenated the titles and abstracts and extracted the TF-IDF (term frequency-inverse document frequency) vector for the corpus. TF-IDF evaluates how relevant a word is to a document in a collection of documents. For additional features, we extracted TF-IDF vector using each feature respectively and then concatenated the extracted vectors with title and abstract vector. XGBoost was selected for the feature evaluation process, due to its relatively quick computational running time and robust performance.

Deep learning algorithms

Conventional ML methods rely heavily on manually designed features and suffer from the challenges of data sparsity and poor transportability when applied to new use cases. Deep learning (DL) is a set of machine learning algorithms based on deep neural networks that has advanced performance of text classification along with many other NLP tasks. Transformer-based deep learning models, such as BERT (Bidirectional encoder representations from transformers), have achieved state-of-the-art performance in many NLP tasks [ 11 ]. A Transformer is an emerging architecture of deep learning models designed to handle sequential input data such as natural language by adopting the mechanisms of attention to differentially weigh the significance of each part of the input data [ 12 ]. The BERT model and its variants (which use Transformer as a basic unit) leverage the power of transfer learning by first pre-training the models over 100’s of millions of parameters using large volumes of unlabeled textual data. The resulting model is then fine-tuned for a particular downstream NLP application, such as text classification, named entity recognition, relation extraction, etc. The following three BERT models were evaluated against both the HPV and Pediatric pneumococcal corpus using two sets of features (title and abstract versus adding all additional features into the text). For all BERT models, we used Adam optimizer with weight decay. We set learning rate at 1e-5, batch size at 8 and number of epochs at 20.

BERT base: this is the original BERT model released by Google. The BERT base model was pre-trained on textual data in the general domain, i.e., BooksCorpus (800 M words) and English Wikipedia (2500 M words) [ 11 ].

BioBERT base: as the biomedical language is different from general language, the BERT models trained on general textual data may not work well on biomedical NLP tasks. BioBERT was further pre-trained (based on original BERT models) in the large-scale biomedical corpora, including PubMed abstracts (4.5B words) and PubMed Central Full-text articles (13.5B words) [ 13 ].

PubMedBERT: PubMedBERT was pre-trained from scratch using abstracts from PubMed. This model has achieved state-of-the-art performance on several biomedical NLP tasks on Biomedical Language Understanding and Reasoning Benchmark [ 14 ].

Text pre-processing and libraries that were used

We have removed special characters and common English words as a part of text pre-processing. Default tokenizer from scikit-learn was adopted for tokenization. Scikit-learn was also used for TF-IDF feature extraction and machine learning algorithms implementation. Transformers libraries from Hugging Face were used for deep learning algorithms implementation.

Evaluation datasets were constructed from the HPV and Pediatric pneumococcal corpora and were split into training, validation and testing sets with a ratio of 8:1:1 for the two evaluation tasks: (1) ML algorithms performance assessment; and (2) DL algorithms performance assessment. Models were fitted on the training sets, and model hyperparameters were optimized on the validation sets and the performance were evaluated on the testing sets. The following major metrics are expressed by the noted calculations:

Where True positive is an outcome where the model correctly predicts the positive (e.g., “included” in our tasks) class. Similarly, a True negative is an outcome where the model correctly predicts the negative class (e.g., “excluded” in our tasks). False positive is an outcome where the model incorrectly predicts the positive class, and a False negative is an outcome where the model incorrectly predicts the negative class. We have repeated all experiments five times and reported the mean scores with standard deviation.

Table  2 shows the baseline comparison using different feature combinations for the SLR text classification tasks using XGBoost. As noted, adding additional features in addition to title and abstract was effective in further improving the classification accuracy. Specifically, using all available features for the HPV classification increased accuracy by ? ∼  3% and F1 score by ? ∼  3%; using all available features for Pediatric pneumococcal classification increased accuracy by ? ∼  2% and F1 score by ? ∼  4%. As observed, adding additional features provided a stronger boost in precision, which contributed to the overall performance improvement.

The comparison of the article inclusion/exclusion classification task for four machine learning algorithms with all features is shown in Table  3 . XGBoost achieved the highest accuracy and F-1 scores in both tasks. Table  4 shows the comparison between XGBoost and deep learning algorithms on the classification tasks for each disease. Both XGBoost and deep learning models consistently have achieved higher accuracy scores when using all features as input. Among all models, BioBERT has achieved the highest accuracy at 0.88, compared with XGBoost at 0.86. XGBoost has the highest F1 score at 0.8 and the highest recall score at 0.9 for inclusion prediction.

Discussions and conclusions

Abstract screening is a crucial step in conducting a systematic literature review (SLR), as it helps to identify relevant citations and reduces the effort required for full-text screening and data element extraction. However, screening thousands of abstracts can be a time-consuming and burdensome task for scientific reviewers. In this study, we systematically investigated the use of various machine learning and deep learning algorithms, using different sets of features, to automate abstract screening tasks. We evaluated these algorithms using disease-focused SLR corpora, including one for human papillomavirus (HPV) associated diseases and another for pneumococcal-associated pediatric diseases (PADA). The publicly available corpora used in this study can be used by the scientific community for advanced algorithm development and evaluation. Our findings suggest that machine learning and deep learning algorithms can effectively automate abstract screening tasks, saving valuable time and effort in the SLR process.

Although machine learning and deep learning algorithms trained on the two SLR corpora showed some variations in performance, there were also some consistencies. Firstly, adding additional citation features significantly improved the performance of conventional machine learning algorithms, although the improvement was not as strong in transformer-based deep learning models. This may be because transformer models were mostly pre-trained on abstracts, which do not include additional citation information like MeSH terms, keywords, and journal names. Secondly, when using only title and abstract as input, transformer models consistently outperformed conventional machine learning algorithms, highlighting the strength of subject domain-specific pre-trained language models. When all citation features were combined as input, conventional machine learning algorithms showed comparable performance to deep learning models. Given the much lower computation costs and faster training and prediction time, XGBoost or support vector machines with all citation features could be an excellent choice for developing an abstract screening system.

Some limitations remain for this study. Although we’ve evaluated cutting-edge machine learning and deep learning algorithms on two SLR corpora, we did not conduct much task-specific customization to the learning algorithms, including task-specific feature engineering and rule-based post-processing, which could offer additional benefits to the performance. As the focus of this study is to provide generalizable strategies for employing machine learning to abstract screening tasks, we leave the task-specific customization to future improvement. The corpora we evaluated in this study mainly focus on health economics and outcome research, the generalizability of learning algorithms to another domain will benefit from formal examination.

Extensive studies have shown the superiority of transformer-based deep learning models for many NLP tasks [ 11 , 13 , 14 , 15 , 16 ]. Based on our experiments, however, adding features to the pre-trained language models that have not seen these features before may not significantly boost their performance. It would be interesting to find a better way of encoding additional features to these pre-trained language models to maximize their performance. In addition, transfer learning has proven to be an effective technique to improve the performance on a target task by leveraging annotation data from a source task [ 17 , 18 , 19 ]. Thus, for a new SLR abstract screening task, it would be worthwhile to investigate the use of transfer learning by adapting our (publicly available) corpora to the new target task.

When labeled data is available, supervised machine learning algorithms can be very effective and efficient for article screening. However, as there is increasing need for explainability and transparency in NLP-assisted SLR workflow, supervised machine learning algorithms are facing challenges in explaining why certain papers fail to fulfill the criteria. The recent advances in large language models (LLMs), such as ChatGPT [ 20 ] and Gemini [ 21 ], show remarkable performance on NLP tasks and good potentials in explainablity. Although there are some concerns on the bias and hallucinations that LLMs could bring, it would be worthwhile to evaluate further how LLMs could be applied to SLR tasks and understand the performance of using LLMs to take free-text article screening criteria as the input and provide explainanation for article screening decisions.

Data availability

The annotated corpora underlying this article are available at https://github.com/Merck/NLP-SLR-corpora .

Bullers K, Howard AM, Hanson A, et al. It takes longer than you think: librarian time spent on systematic review tasks. J Med Libr Assoc. 2018;106:198–207. https://doi.org/10.5195/jmla.2018.323 .

Article   PubMed   PubMed Central   Google Scholar  

Carver JC, Hassler E, Hernandes E et al. Identifying Barriers to the Systematic Literature Review Process. In: 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement . 2013. 203–12. https://doi.org/10.1109/ESEM.2013.28 .

Lame G. Systematic literature reviews: an introduction. Proc Des Society: Int Conf Eng Des. 2019;1:1633–42. https://doi.org/10.1017/dsi.2019.169 .

Article   Google Scholar  

Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun. 2019;16:100443. https://doi.org/10.1016/j.conctc.2019.100443 .

Recent advances in. biomedical literature mining | Briefings in Bioinformatics | Oxford Academic. https://academic.oup.com/bib/article/22/3/bbaa057/5838460?login=true (accessed 30 May 2022).

Medical Subject Headings - Home Page. https://www.nlm.nih.gov/mesh/meshhome.html (accessed 30 May 2022).

Chen T, Guestrin C, XGBoost:. A Scalable Tree Boosting System. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . New York, NY, USA: Association for Computing Machinery 2016. 785–94. https://doi.org/10.1145/2939672.2939785 .

Noble WS. What is a support vector machine? Nat Biotechnol. 2006;24:1565–7. https://doi.org/10.1038/nbt1206-1565 .

Article   CAS   PubMed   Google Scholar  

Logistic Regression . https://doi.org/10.1007/978-1-4419-1742-3 (accessed 30 May 2022).

Random forest classifier for remote sensing classification. International Journal of Remote Sensing: Vol 26, No 1. https://www.tandfonline.com/doi/abs/10.1080/01431160412331269698 (accessed 30 May 2022).

Devlin J, Chang M-W, Lee K, et al. BERT: pre-training of Deep Bidirectional transformers for Language understanding. arXiv. 2019. https://doi.org/10.48550/arXiv.1810.04805 .

Vaswani A, Shazeer N, Parmar N et al. Attention is All you Need. In: Advances in Neural Information Processing Systems . Curran Associates, Inc. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html (accessed 30 May 2022).

BioBERT. a pre-trained biomedical language representation model for biomedical text mining | Bioinformatics | Oxford Academic. https://academic.oup.com/bioinformatics/article/36/4/1234/5566506 (accessed 3 Jun 2020).

Gu Y, Tinn R, Cheng H, et al. Domain-specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans Comput Healthc. 2021;3(2):1–2. https://doi.org/10.1145/3458754 .

Article   CAS   Google Scholar  

Chen Q, Du J, Allot A, et al. LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curation. arXiv. 2022. https://doi.org/10.48550/arXiv.2204.08649 .

Chen Q, Allot A, Leaman R, et al. Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations. arXiv. 2022. https://doi.org/10.48550/arXiv.2204.09781 .

Kermany DS, Goldbaum M, Cai W, et al. Identifying Medical diagnoses and Treatable diseases by Image-based deep learning. Cell. 2018;172:1122–e11319. https://doi.org/10.1016/j.cell.2018.02.010 .

Howard J, Ruder S. Universal Language Model fine-tuning for text classification. arXiv. 2018. https://doi.org/10.48550/arXiv.1801.06146 .

Do CB, Ng AY. Transfer learning for text classification. In: Advances in Neural Information Processing Systems . MIT Press. 2005. https://proceedings.neurips.cc/paper/2005/hash/bf2fb7d1825a1df3ca308ad0bf48591e-Abstract.html (accessed 30 May 2022).

Achiam J et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

https:// gemini.google.com/app/a4dcd2e2d7672354 . (accessed 01 Feb 2024).

Download references

Acknowledgements

We thank Dr. Majid Rastegar-Mojarad for conducting some additional experiments during revision.

This research was supported by Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA.

Author information

Authors and affiliations.

Intelligent Medical Objects, Houston, TX, USA

Jingcheng Du, Ekin Soysal, Long He, Bin Lin, Jingqi Wang & Frank J. Manion

Merck & Co., Inc, Rahway, NJ, USA

Dong Wang, Yeran Li, Elise Wu & Lixia Yao

McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA

Ekin Soysal

You can also search for this author in PubMed   Google Scholar

Contributions

Study concept and design: JD and LY Corpus preparation: DW, YL and LY Experiments: JD and ES Draft of the manuscript: JD, DW, FJM and LY Acquisition, analysis, or interpretation of data: JD, ES, DW and LY Critical revision of the manuscript for important intellectual content: JD, ES, DW, LH, BL, JW, FJM, YL, EW, LY Study supervision: LY.

Corresponding author

Correspondence to Lixia Yao .

Ethics declarations

Disclaimers.

The content is the sole responsibility of the authors and does not necessarily represent the official views of Merck & Co., Inc., Rahway, NJ, USA or Intelligent Medical Objects.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Competing interests.

DW is an employee of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA. EW, YL, and LY were employees of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA for this work. JD, LH, JW, and FJM are employees of Intelligent Medical Objects. ES was an employee of Intelligent Medical Objects during his contributions, and is currently an employee of EBSCO Information Services. All the other authors declare no competing interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Du, J., Soysal, E., Wang, D. et al. Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research. BMC Med Res Methodol 24 , 108 (2024). https://doi.org/10.1186/s12874-024-02224-3

Download citation

Received : 19 May 2023

Accepted : 18 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s12874-024-02224-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Machine learning
• Deep learning
• Text classification
• Article screening
• Systematic literature review

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

The Women's Health Initiative Randomized Trials and Clinical Practice: A Review

Affiliations.

• 1 Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts.
• 2 Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles.
• 3 National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland.
• 4 Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, California.
• 5 Division of Public Health Sciences, Fred Hutchinson Cancer Center, Seattle, Washington.
• 6 Stanford Prevention Research Center, Department of Medicine, Stanford University, Stanford, California.
• 7 Department of Epidemiology, University of Pittsburgh School of Public Health|Epidemiology, Pittsburgh, Pennsylvania.
• 8 Department of Medicine, University of Alabama, Birmingham.
• 9 Division of Epidemiology, Herbert Wertheim School of Public Health and Human Longevity Science, University of California, San Diego, La Jolla, California.
• 10 Department of Health Promotion Science, University of Arizona, Tucson.
• 11 Department of Preventive Medicine, Northwestern University, Chicago, Illinois.
• 12 MedStar Health Research Institute and Department of Medicine, Georgetown University School of Medicine, Washington, DC.
• 13 Department of Epidemiology and Environmental Health, University at Buffalo-SUNY, Buffalo, New York.
• 14 Department of Gerontology and Geriatric Medicine, Wake Forest University School of Medicine, Winston-Salem, North Carolina.
• PMID: 38691368
• DOI: 10.1001/jama.2024.6542

Importance: Approximately 55 million people in the US and approximately 1.1 billion people worldwide are postmenopausal women. To inform clinical practice about the health effects of menopausal hormone therapy, calcium plus vitamin D supplementation, and a low-fat dietary pattern, the Women's Health Initiative (WHI) enrolled 161 808 postmenopausal US women (N = 68 132 in the clinical trials) aged 50 to 79 years at baseline from 1993 to 1998, and followed them up for up to 20 years.

Observations: The WHI clinical trial results do not support hormone therapy with oral conjugated equine estrogens plus medroxyprogesterone acetate for postmenopausal women or conjugated equine estrogens alone for those with prior hysterectomy to prevent cardiovascular disease, dementia, or other chronic diseases. However, hormone therapy is effective for treating moderate to severe vasomotor and other menopausal symptoms. These benefits of hormone therapy in early menopause, combined with lower rates of adverse effects of hormone therapy in early compared with later menopause, support initiation of hormone therapy before age 60 years for women without contraindications to hormone therapy who have bothersome menopausal symptoms. The WHI results do not support routinely recommending calcium plus vitamin D supplementation for fracture prevention in all postmenopausal women. However, calcium and vitamin D are appropriate for women who do not meet national guidelines for recommended intakes of these nutrients through diet. A low-fat dietary pattern with increased fruit, vegetable, and grain consumption did not prevent the primary outcomes of breast or colorectal cancer but was associated with lower rates of the secondary outcome of breast cancer mortality during long-term follow-up.

Conclusions and relevance: For postmenopausal women, the WHI randomized clinical trials do not support menopausal hormone therapy to prevent cardiovascular disease or other chronic diseases. Menopausal hormone therapy is appropriate to treat bothersome vasomotor symptoms among women in early menopause, without contraindications, who are interested in taking hormone therapy. The WHI evidence does not support routine supplementation with calcium plus vitamin D for menopausal women to prevent fractures or a low-fat diet with increased fruits, vegetables, and grains to prevent breast or colorectal cancer. A potential role of a low-fat dietary pattern in reducing breast cancer mortality, a secondary outcome, warrants further study.

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Preventing the Next Big Cyberattack on U.S. Health Care

• Erik Decker,
• John Glaser,
• Janet Guptill

Five actions that can help avoid a repeat of the Change Healthcare debacle.

The cyberattack on Change Healthcare that devastated the U.S. health care sector made painfully clear that much more needs to be done to address vulnerabilities that exist throughout the ecosystem. This article offers five actions that can go a long way to improving cybersecurity throughout the sector and make it much more resilient.

This past February, a ransomware attack on a company called Change Healthcare brought medical billing in the United States to a standstill and propelled hundreds of financially strapped health systems and medical practices to the brink of bankruptcy. The breach paralyzed the cash flow of many of the organizations that collectively account for a fifth of the U.S. economy, potentially compromised as many as 85 million patient records, and cost billions of dollars. Recovery is still in progress as we write, and it may be months or years before the final toll is known.

• Erik Decker is a vice president and the chief information security officer at Intermountain Health. He chairs the Health Sector Coordinating Council’s Cybersecurity Working Group, an industry-led council of more than 400 healthcare organizations that advises the government and health sector on how to protect against and recover from cyberthreats. He also co-leads the 405(d) Task Group, a collaborative effort between the Health Sector Coordinating Council and the U.S. government to align the health care sector’s security practices.
• John Glaser is an executive in residence at Harvard Medical School. He previously served as the CIO of Partners Healthcare (now Mass General Brigham), a senior vice president at Cerner, and the CEO of Siemens Health Services. He is co-chair of the HL7 Advisory Council and a board member of the National Committee for Quality Assurance.
• Janet Guptill is president and CEO of the Scottsdale Institute, a not-for-profit organization dedicated to helping its more than 60 large, integrated health systems leverage information and technology to create effective, affordable, and equitable health care centered on whole person care.

Partner Center

This paper is in the following e-collection/theme issue:

Published on 1.5.2024 in Vol 12 (2024)

Cross-Cutting mHealth Behavior Change Techniques to Support Treatment Adherence and Self-Management of Complex Medical Conditions: Systematic Review

Authors of this article:

• Cyd K Eaton 1 , PhD ; 
• Emma McWilliams 2 , BA ; 
• Dana Yablon 3 , BS ; 
• Irem Kesim 3 , BS ; 
• Renee Ge 3 , BS ; 
• Karissa Mirus 3 , MHSA ; 
• Takeera Sconiers 3 , BS ; 
• Alfred Donkoh 3 , BS ; 
• Melanie Lawrence 4 , BS ; 
• Cynthia George 5 , MSN ; 
• Mary Leigh Morrison 4 , MA ; 
• Emily Muther 6 , PhD ; 
• Gabriela R Oates 7 , PhD ; 
• Meghana Sathe 8 , MD ; 
• Gregory S Sawicki 2 , MD, MPH ; 
• Carolyn Snell 2 , PhD ; 
• Kristin Riekert 3 , PhD

1 Division of General Pediatrics, Department of Pediatrics, Johns Hopkins School of Medicine, , Baltimore, MD, , United States

2 Boston Children’s Hospital, Harvard Medical School, , Boston, MA, , United States

3 Division of Pulmonary and Critical Care Medicine, Department of Medicine, Johns Hopkins School of Medicine, , Baltimore, MD, , United States

4 Success with Therapies Research Consortium CF Community Member Advisory Board, , Bethesda, MD, , United States

5 Cystic Fibrosis Foundation, , Bethesda, MD, , United States

6 Children’s Hospital Colorado, University of Colorado School of Medicine, , Aurora, CO, , United States

7 Division of Pediatric Pulmonary & Sleep Medicine, Preventive Medicine, University of Alabama at Birmingham, , Birmingham, AL, , United States

8 Children’s Health Dallas, University of Texas Southwestern Medical Center, , Dallas, TX, , United States

Corresponding Author:

Cyd K Eaton, PhD

Background: Mobile health (mHealth) interventions have immense potential to support disease self-management for people with complex medical conditions following treatment regimens that involve taking medicine and other self-management activities. However, there is no consensus on what discrete behavior change techniques (BCTs) should be used in an effective adherence and self-management–promoting mHealth solution for any chronic illness. Reviewing the extant literature to identify effective, cross-cutting BCTs in mHealth interventions for adherence and self-management promotion could help accelerate the development, evaluation, and dissemination of behavior change interventions with potential generalizability across complex medical conditions.

Objective: This study aimed to identify cross-cutting, mHealth-based BCTs to incorporate into effective mHealth adherence and self-management interventions for people with complex medical conditions, by systematically reviewing the literature across chronic medical conditions with similar adherence and self-management demands.

Methods: A registered systematic review was conducted to identify published evaluations of mHealth adherence and self-management interventions for chronic medical conditions with complex adherence and self-management demands. The methodological characteristics and BCTs in each study were extracted using a standard data collection form.

Results: A total of 122 studies were reviewed; the majority involved people with type 2 diabetes (28/122, 23%), asthma (27/122, 22%), and type 1 diabetes (19/122, 16%). mHealth interventions rated as having a positive outcome on adherence and self-management used more BCTs (mean 4.95, SD 2.56) than interventions with no impact on outcomes (mean 3.57, SD 1.95) or those that used >1 outcome measure or analytic approach (mean 3.90, SD 1.93; P =.02). The following BCTs were associated with positive outcomes: self-monitoring outcomes of behavior (39/59, 66%), feedback on outcomes of behavior (34/59, 58%), self-monitoring of behavior (34/59, 58%), feedback on behavior (29/59, 49%), credible source (24/59, 41%), and goal setting (behavior; 14/59, 24%). In adult-only samples, prompts and cues were associated with positive outcomes (34/45, 76%). In adolescent and young adult samples, information about health consequences (1/4, 25%), problem-solving (1/4, 25%), and material reward (behavior; 2/4, 50%) were associated with positive outcomes. In interventions explicitly targeting medicine taking, prompts and cues (25/33, 76%) and credible source (13/33, 39%) were associated with positive outcomes. In interventions focused on self-management and other adherence targets, instruction on how to perform the behavior (8/26, 31%), goal setting (behavior; 8/26, 31%), and action planning (5/26, 19%) were associated with positive outcomes.

Conclusions: To support adherence and self-management in people with complex medical conditions, mHealth tools should purposefully incorporate effective and developmentally appropriate BCTs. A cross-cutting approach to BCT selection could accelerate the development of much-needed mHealth interventions for target populations, although mHealth intervention developers should continue to consider the unique needs of the target population when designing these tools.

Trial Registration: PROSPERO CRD42021224407; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=224407

Introduction

Ever-advancing mobile health (mHealth) technologies hold immense potential to deliver behavior change techniques (BCTs) to diverse audiences, including people with complex medical conditions that involve treatment adherence and other self-management activities. mHealth refers to “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices” [ 1 ]. Common examples include sending smartphone notifications as medication reminders or recording in an app when treatments are completed. Prior mHealth reviews have broadly summarized mHealth interventions as “reminders, education, or behavioral” [ 2 ], which included a wide range of study outcomes beyond adherence or self-management [ 3 ] or limited the outcome to medication taking [ 4 , 5 ]. Therefore, existing reviews have a limited impact on exactly how mHealth can most effectively support adherence and disease self-management or can be adapted and tailored for chronic illnesses with complex regimens beyond simply taking medicine.

The BCT Taxonomy [ 6 ] was created to define discrete, cross-cutting techniques (or approaches) to changing behavior to facilitate the design and evaluation of behavior change interventions, as well as the comparison of BCTs across interventions to identify which BCTs are the most efficacious. The BCT Taxonomy is disease agnostic such that BCTs found to effectively improve treatment adherence and self-management in one complex medical condition should, in theory, generalize to other complex medical conditions with similar adherence and self-management demands. Reviewing mHealth interventions of diseases with complex adherence and self-management demands using BCT Taxonomy could accelerate the design of mHealth solutions by identifying “essential elements” of effective mHealth interventions. Unfortunately, there is no consensus on what essential features should be included in an adherence or self-management mHealth solution for any chronic medical condition.

Our group’s interest in cross-cutting BCTs for adherence and self-management stems from our work with the cystic fibrosis (CF) community. CF is a rare, multisystemic medical condition affecting an estimated 162,428 people worldwide [ 7 ]. CF self-management is complex and typically involves a combination of daily oral medications, inhaled treatment, high calorie diet, chest physiotherapy, airway clearance, and exercise [ 8 ]. Not surprisingly, people with CF have demonstrated high rates of nonadherence across various aspects of the multicomponent treatment regimen, including low medication adherence (48%-68%) [ 9 , 10 ], nonadherence to caloric goals (24%-40%) [ 11 ], and low adherence to airway clearance therapy (28%) [ 12 ]. Effective behavioral interventions are needed to promote CF self-management and, in turn, support health outcomes and quality of life. However, rare diseases with complex regimens are rarely the target population for technology developers, and for almost a decade, people with CF have expressed interest in an app but noted that existing apps do not provide the necessary functionality to address their CF management needs [ 13 - 15 ]. A recent search of the Google Play Store (Android) and Apple App Store (iOS) for health-related apps found that only 29 (1.3%) out of 2272 apps address a rare disease population [ 16 ], including CF, with none having empirical evidence of their efficacy.

Recognizing that there is a dearth of empirical research on mHealth solutions for treatment adherence and self-management of CF and other rare diseases, we aimed to learn from the BCTs used in effective mHealth interventions for other chronic medical conditions with complex treatment adherence and self-management demands. We, therefore, purposefully designed our systematic review to include people with complex diseases and regimens with overlapping characteristics to CF. Our research questions were (1) Which BCTs have been used in mHealth interventions? and (2) Which BCTs have a positive impact on adherence and self-management behaviors? Differences in BCTs in adult-only studies compared to adolescent and young adult studies were examined, as well as interventions explicitly targeting medicine taking compared to studies targeting broader self-management and other areas of treatment adherence. A systematic review was used because heterogeneity in measuring adherence and self-management outcomes across studies precludes a meta-analysis [ 17 - 19 ] (in contrast to a systematic review, a meta-analysis involves statistically summarizing results across reviewed studies using effect sizes [ 20 ]). Our overarching goal was to identify the essential, cross-cutting BCTs delivered via mHealth to effectively facilitate long-term adherence and self-management for people with complex medical regimens, thereby accelerating intervention development, evaluation, and dissemination.

Standardized search strategies, eligibility evaluations, and data extraction procedures were used (detailed below and in Multimedia Appendix 1 ). This review was registered with the International Prospective Register of Systematic Reviews (PROSPERO CRD42021224407), in accordance with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines ( Checklist 1 ).

Ethical Considerations

As this was a systematic review, institutional review board approval was not required.

Search Strategy

A literature search in the PubMed, Scopus, Embase, CENTRAL, Web of Science, and PsycINFO databases identified potentially relevant articles published from 2015 through 2020, to enhance relevance to current technology. Given our group’s focus on CF, 2 categories of search terms were used: “CF-specific” and “other chronic conditions,” which included conditions identified by the study authors as having similar adherence and self-management characteristics to CF (eg, conditions with complex daily medical regimens and diseases often diagnosed in childhood, thus involving caregivers in self-management tasks).

Eligibility Criteria

Peer-reviewed, English language articles published between 2015 and 2020 reporting original empirical findings of mHealth interventions for selected medical conditions and targeting adherence and self-management were included. The mHealth interventions must be accessed on a mobile device (smartphones, cell phones, or tablets, including internet browser programs) and used by a person managing a medical condition or their caregiver.

Post Hoc Exclusions

After executing the search strategy, 3 post hoc exclusion criteria were added. People with chronic obstructive pulmonary disease or engaging in pulmonary rehabilitation were excluded, as it was decided that the former population was too different from people with CF and the latter included medical conditions. Reminder-only text messaging and exclusively synchronous telephone or web videoconferencing interventions were excluded, as our interest was in automated BCTs beyond simple reminders and interventions requiring real-time human interaction. Investigations conducted in low- to middle-income countries were excluded due to potential technology access limitations (unreliable internet or cellular service) that would likely affect the types of interventions tested.

Selection Process

Study records were compiled in a database; duplicates were removed based on DOI number or title. Reviewers (CKE, E McWilliams, DY, TS, and Brandi Blackshear) evaluated each study record (title and citation; blinded double review) for eligibility criteria. The reviewers screened studies for final inclusion and data abstraction using a REDCap (Research Electronic Data Capture; Vanderbilt University) [ 21 , 22 ] form developed for this study.

Data Collection

A reviewer independently abstracted the study data. A second reviewer read the article, reviewed the initial data abstraction, and identified items of disagreement. Discrepancies were discussed and resolved with all team members. Study characteristics were abstracted for each study in the final review (publication year, study location, study design, sample size, medical condition, age group, and theoretically derived intervention). Missing study details were noted.

Key Definitions

The adherence and self-management measurement method was abstracted. Reviewers categorized adherence and self-management measurement as (1) objective behavior (eg, electronic medication monitoring), (2) subjective behavior (eg, patient-reported medication adherence), (3) psychosocial outcome (eg, disease knowledge and adherence self-efficacy), (4) objective health outcome (eg, hemoglobin A 1c and viral load), or (5) subjective health outcome (eg, patient-reported asthma control level). Health outcomes were included if the authors conceptualized them as adherence and self-management indicators.

mHealth tools (eg, app and text messaging) and targets of intervention (eg, taking a specific medicine, airway clearance therapy, diabetes self-management activities, dietary recommendations, exercise and physical activity, managing disease activity and symptoms, etc) were abstracted. mHealth intervention results were categorized based on authors’ conclusion of the results as follows:

• Positive: intervention was associated with improved adherence and self-management.
• Negative: intervention was associated with worse adherence and self-management.
• No impact: intervention had no effect on adherence and self-management.
• Mixed: intervention had different effects (positive effect, negative effect, or no impact) on adherence and self-management due to multiple outcome measures and analytic approaches.

The abstracter used information in the manuscript and the BCT Taxonomy to assign discrete BCTs to each intervention component.

Risk of Bias

The Revised Cochrane Risk-of-Bias tool (RoB 2) [ 23 ] for randomized controlled trials (RCTs) and the Risk of Bias in Non-randomized Studies-of Interventions (ROBINS-I) [ 24 ] tool for nonrandomized studies (excluding qualitative studies) were used to assess risk of bias, certainty, and quality of evidence among the studies reviewed. Blinded double assessments were conducted by 2 independent reviewers (RG, IK, E McWilliams, DY, and AD). The RoB-2 assessed risk of bias due to the randomization process, deviations from intended interventions, missing outcome data, measurement of outcome, and selection of the reported result. The ROBINS-I assessed risk of bias due to confounding, deviations from intended interventions, missing data, bias in measurement of outcomes, and bias in selection of the reported result. Discrepancies in ratings were identified and resolved. If multiple outcomes were assessed, an average risk score was calculated to derive a single rating.

Statistical analyses were conducted in Stata 15 software (StataCorp LLC). Abstracted data were summarized using frequencies and percentages. Subgroup analyses examined differences in study characteristics, including BCTs used, age group (adult only [≥18 years and older] vs adolescent and young adult [11-25 years or sample characterized by authors as “adolescents and young adults”]), study design (RCT vs non-RCT), and whether the intervention was theoretically derived. We also conducted an exploratory subgroup analysis to examine which BCTs appeared the most often in interventions explicitly focusing on medicine taking compared to interventions focusing on self-management and other adherence targets. The results highlight BCTs (1) appearing in ≥5% of studies and (2) with a difference of >10% between positive effects versus no impact on adherence and self-management outcomes. This does not mean that rarely used BCTs are ineffective or that 10% is a verified benchmark of clinically meaningful difference. This pragmatic decision supported the interpretation of a large number of BCTs and comparisons. Statistically significant ( P <.05) differences in the number of BCTs based on the direction of results were tested using 1-way ANOVA. No effect measures, missing summary statistics or data conversions, or meta-regression were used for this systematic review.

Screening Process

Figure 1 presents this review’s PRISMA diagram. The initial search returned 14,889 articles. After removing duplicates and clinical trial registrations, 7400 titles were screened for initial eligibility, 303 articles were potentially eligible, and 122 manuscripts met the criteria for data extraction (see Table S1 in Multimedia Appendix 1 for all included studies and characteristics).

Study Characteristics

The most represented medical conditions were type 1 or 2 diabetes and asthma ( Table 1 ). Only 6 (4.9%) out of 122 studies involved people with CF. Most studies were published in 2020 (32/122, 26.2%), were conducted outside of the United States (64/122, 52.5%), and used an RCT design (75/122, 61.5%). Nonrandomized study designs were primarily observational pre-post (23/122, 18.9%), observational without pre-post measurement (6/122, 4.9%), or mixed methods (6/122, 4.9%) studies. Study sample sizes ranged from 10 to 14,085 (median 92, IQR 44-179) participants. Most studies involved adult-only (81/122, 66.4%) or adolescent and young adult–only samples (22/122, 18%), followed by child, adolescent, young adult (11/122, 9%); child, adolescent, young adult, and adult (6/122, 4.9%); and child-only (2/122, 1.6%) samples.

a Medical conditions are listed from most frequently to least frequently observed in the overall study sample.

Adherence and self-management outcomes were typically evaluated with objective health outcomes or subjective behavior measures (61/122, 50% for each; Table S2 in Multimedia Appendix 1 ). mHealth interventions were delivered via app (75/122, 61.5%), SMS text messaging (34/122, 27.9%), or website (30/122, 24.6%). Nearly all studies presented mHealth tools used by patients (118/122, 96.7%), but many included health care providers (48/122, 39.3%) or caregivers (21/122, 17.2%). Only 22 (18%) interventions were clearly informed by scientific theory. mHealth interventions most often targeted taking medication (68/122, 55.7%), diabetes self-management activities (46/122, 37.7%), dietary recommendations (32/122, 26.2%), exercise and physical activity (27/122, 22.1%), asthma self-management activities (13/122, 10.7%), managing disease activity and symptoms (11/122, 9%), and general “self-care” behaviors (5/122, 4.1%). One (0.8%) study targeted airway clearance therapy.

Study results were characterized as having a positive effect on the outcome or outcomes (59/122, 48.4%), followed by mixed results (40/122, 32.8%) or no impact (23/122, 18.9%). No studies were characterized as having negative effects ( Table 1 ). Comparing studies reporting positive effects to no impact, 34% (20/59) of the positive studies used objective behavior adherence measures compared to 13% (3/23) of no-impact studies (Table S2 in Multimedia Appendix 1 ).

Across all reviewed studies, 32 different BCTs were used (mean 4.30, SD 2.32). Table S3 in Multimedia Appendix 1 provides the frequencies, definitions, and examples of BCTs appearing in ≥5% of reviewed studies.

BCTs by Intervention Effect on Outcomes

Interventions with positive effects contained significantly more BCTs (mean 4.95, SD 2.56) than interventions with mixed effects (mean 3.90, SD 1.93) or no impact (mean 3.57, SD 1.95; P =.02). BCTs used in >10% of studies with positive results versus no impact ( Multimedia Appendix 2 ) were self-monitoring of behavior, self-monitoring of outcomes of behavior, feedback on outcomes of behavior, feedback on behavior, credible source, and goal setting (behavior).

Subgroup Analysis: Age

Table S3 in Multimedia Appendix 1 includes the 15 most common BCTs used in adult-only and adolescent and young adult–only studies. Among adult-only studies (n=81), interventions with positive effects contained significantly more BCTs (mean 5.02, SD 2.13) than studies with mixed effects (mean 4.28, SD 2.27) or no impact (mean 3.28, SD 1.96; P =.02). BCTs used in >10% of studies with positive results compared to no impact included prompts and cues, self-monitoring of outcomes of behavior, feedback on outcomes of behavior, self-monitoring of behavior, feedback on behavior, credible source, and goal setting (behavior; Multimedia Appendix 3 ).

In adolescent and young adult–only studies (n=22), interventions with positive effects contained significantly more BCTs (mean 7.75, SD 5.91) than studies with mixed effects (mean 3.50, SD 1.67) or no impact (mean 5.00, SD 1.41; P =.04). BCTs used in >10% of studies with positive results compared to no impact included self-monitoring of behavior, feedback on behavior, goal setting (behavior), information about health consequences, problem-solving, and material reward (behavior; Multimedia Appendix 4 ).

Subgroup Analysis: Study Design and Theory

Non-RCT studies tended to report positive results (33/59, 56%), whereas RCT designs more commonly reported no impact (21/23, 91%) or mixed results (28/40, 70%). Theory rarely guided intervention design; a small proportion (12/59, 20%) of theory-informed interventions were shown to have a positive effect ( Table 1 ).

Subgroup Analysis: Intervention Target

Table S4 in Multimedia Appendix 1 includes the 16 most common BCTs used in studies targeting medicine taking versus self-management and other adherence targets (appeared in >5% of studies). A total of 62% (42/68) of studies targeting medicine taking included people with diabetes (21/68, 31%) or asthma (21/68, 31%). A total of 80% (43/54) of studies targeting self-management and other adherence targets included people with diabetes (37/54, 69%) or asthma (6/54, 11%). There were no significant differences in the number of BCTs used in interventions targeting medicine taking (mean 4.07, SD 1.94) compared to interventions targeting self-management and other adherence targets (mean 4.69, SD 2.70; P =.16).

Within interventions explicitly targeting medicine taking (n=68), BCTs used in >10% of studies with positive results compared to no impact included prompts and cues, self-monitoring outcomes of behavior, self-monitoring of behavior, feedback on behavior, feedback on outcomes of behavior, and credible source ( Multimedia Appendix 5 ). There were no significant differences in the number of BCTs used based on the direction of results ( P =.06).

Within interventions focused on self-management and other adherence targets (n=54), BCTs used in >10% of studies with positive results compared to no impact included self-monitoring outcomes of behavior, feedback on outcomes of behavior, self-monitoring of behavior, feedback on behavior, instruction on how to perform the behavior, goal setting (behavior), and action planning ( Multimedia Appendix 6 ). There were no significant differences in the number of BCTs used based on the direction of results ( P =.21).

Table S1 in Multimedia Appendix 1 reports each study’s risk of bias rating. No study was excluded due to bias rating. For RCTs, 57% (43/75) received an overall risk of bias rating of “Some concerns,” 39% (29/75) had “High” concerns, and only 4% (3/75) had “Low” concerns. “High” concern ratings were generally due to deviations from the intended interventions (18/29, 62%), the randomization process (11/29, 38%), or missing outcome data (11/29, 38%). For nonrandomized studies, 82% (36/44) received an overall risk of bias rating of “Serious” concerns, 9% (4/44) had “Critical” concerns, and 2% (1/44) had “Moderate” concerns. No nonrandomized study had “Low” risk of bias. “Serious” or “Critical” ratings were generally due to confounding (38/40, 95%) or deviations from intended interventions (13/40, 33%).

Principal Findings

Our literature review of mHealth adherence and self-management interventions returned 122 studies, from which we identified discrete behavioral strategies using the BCT Taxonomy [ 6 ] with promise to promote adherence and self-management for people living with medical conditions requiring complex, daily self-management activities. The BCT Taxonomy provides, to date, the most rigorously tested, standardized method to identify cross-cutting BCTs with potential applicability across chronic medical conditions with overlapping adherence and self-management demands. The BCT Taxonomy also helps compare mHealth interventions and provides a shared language about BCTs for clinicians, researchers, mHealth innovators, and other key stakeholders such as patients and caregivers. As technological advances can quickly outdate mHealth, focusing on BCT principles, rather than the technology to deliver them, enhances the research’s relevance and potential generalizability to a range of complex medical conditions, including rare diseases (an area of focus for our group), which often have significant need for such tools in contrast to the finite resources available to conduct large-scale, multistep mHealth design and evaluation studies.

Consistent with prior research [ 25 , 26 ], using more BCTs was associated with improved adherence and self-management. However, 6 BCTs appear particularly promising: self-monitoring of behavior, self-monitoring of outcomes of behavior, feedback on behavior, feedback on outcomes of behavior, credible source, and goal setting. Self-monitoring of behavior and outcomes of behavior involve tracking health behavior engagement (eg, logging in an app when medicine is taken) or outcomes of behavior (eg, using a Bluetooth-enabled glucometer to monitor blood glucose levels), whereas feedback on behavior and outcomes of behavior involve providing users with a summarized interpretation of the tracked data (eg, providing in-app graphical representations of one’s daily step count over the past month). Consistent with our results, a prior meta-analysis showed that monitoring medication adherence and providing feedback improve medication adherence [ 27 ]. These strategies may build awareness for when the mHealth user engages in a health behavior, provide opportunity to reflect on successes and challenges, and ultimately help the user make informed behavior changes. Credible source involves providing expert-generated information about managing the user’s medical condition (eg, the app contains information about etiology, symptoms, and treatment), which presents users with knowledge to understand the condition and its management. Goal setting (behavior ) involves setting measurable and attainable goals for a target health behavior (eg, set a goal for number of days to exercise in a month), which can help the mHealth user focus on key health behavior and build self-efficacy as goals are met.

Developmental differences emerged between adult samples and adolescents and young adult samples. In adult-only studies, prompts and cues (reminders) were associated with positive outcomes, consistent with reviews showing that reminders are associated with a 2- to 3-fold increase in adherence [ 28 , 29 ], but they were less effective in adolescent and young adult studies. Indeed, a pre-post study of children and adolescents with CF found that adherence did not change after delivering reminders only (therefore excluded from this review) for 6 months [ 30 ]. Adolescents and young adults may benefit from improving knowledge ( information about health consequences ), improving skills ( problem-solving ), and building motivation ( material reward [ behavior ]). Given the small number of adolescent and young adult studies, these results and interpretations should be seen as hypothesis generating.

Differences emerged between interventions explicitly targeting medicine taking versus those focused on disease self-management and other adherence targets. In interventions targeting medicine taking, prompts and cues and credible source were associated with positive outcomes. Reminders and expert information may be the most effective when focused on discrete, clearly defined behaviors rather than complex, multicomponent self-management activities. In interventions focused on self-management and other adherence targets, instruction on how to perform the behavior, goal setting (behavior), and action planning were associated with positive outcomes. Over three-quarters (43/54, 80%) of studies focused on self-management and other adherence targets involved people with diabetes or asthma, which are relatively common yet complex medical conditions involving self-management behaviors that extend beyond simply taking medicine. Skills training, behavioral goals, and assistance with creating a detailed plan for managing a complex medical condition may be the most effective for multicomponent self-management activities that may involve monitoring and intervening upon changes in disease activity (eg, managing fluctuations in blood glucose levels for people with diabetes or managing asthma exacerbations) and self-managing lifestyle and environmental considerations (eg, diet in diabetes and environmental triggers in asthma). Careful consideration of the intervention target will likely help to further guide appropriate BCT selection from the BCTs found to be associated with improved adherence and self-management in our review.

This review has limitations. A meta-analysis was not conducted due to heterogeneous outcomes [ 17 - 19 ], thus we could not conclude which BCTs were statistically the most effective. Our risk-of-bias assessment highlighted methodological concerns across the studies reviewed. No-impact studies were more likely to be RCTs, and positive studies were more likely to be nonrandomized, raising concerns about publication bias toward positive results irrespective of study quality. We excluded reminder-only interventions; thus, most studies incorporated more than 1 BCT. Our reported average number of BCTs is likely higher than that of all adherence-promoting mHealth interventions. Although we identified some BCTs that may be effective, others may be as or more effective in supporting disease self-management but were rarely used in the reviewed studies. Moreover, no BCT was found to do harm. Thus, mHealth innovators should continue to integrate and evaluate how a wide variety of technology-delivered BCTs may support people living with chronic diseases, including rare diseases such as CF. An inherent limitation of conducting literature reviews is that a cutoff date must be selected, yet scientific literature is constantly being published; there may be utility in conducting an updated systematic review of this topic in the future. We only included studies that were published in peer-reviewed journals to focus on interventions with clear evidence of scientific evaluation; however, expanding our review to “gray literature” may have provided more insight into the most current interventions and reduced publication bias. Our review characterized mHealth BCTs generally. Other metrics including digital literacy and socioeconomic barriers to mHealth were not evaluated. Future researchers should evaluate these factors to support sustained mHealth use among diverse audiences. Additionally, BCTs were analyzed across the included chronic medical conditions given the disproportionate number of studies in diabetes and asthma compared to other medical conditions. Although the BCTs are disease agnostic, intervention developers and researchers should carefully consider the applicability of the BCT to the target patient population.

Future Directions

Our review identified discrete BCTs that may have broad cross-cutting applicability across chronic diseases with complex medical regimens, including people with CF, the community with which our group primarily works with. We consider our systematic review approach to be a model for gathering key findings from the extant scientific literature to inform the development of multicomponent behavioral mHealth interventions tailored for a patient population that may be smaller and with less existing research, yet has significant self-management needs warranting further research, such as CF [ 9 , 10 , 31 , 32 ]. Research involving people with chronic medical conditions following complex treatment regimens should prioritize the design and evaluation of mHealth interventions incorporating cross-cutting, evidence-based, and age-appropriate BCTs to promote adherence and self-management. Such an approach could help accelerate mHealth intervention design and evaluation to create effective products that may be efficiently disseminated to communities with significant need for such tools.

Accelerating mHealth design and evaluation by taking a cross-cutting approach to BCT selection would also help answer remaining “unknowns” about mHealth BCTs and strengthen mHealth intervention quality. For example, although interventions including more BCTs appear to have greater benefit, the optimal number, type, and combination of BCTs to include in mHealth interventions have not been determined. For BCTs demonstrating potential to promote adherence or self-management, the ideal delivery method must be determined (eg, should the BCT self-monitoring of behavior be delivered via manual data entry of treatment completion or using an electronic monitoring device to automatically track data?). An overrepresentation of certain BCTs (eg, prompts and cues and self-monitoring of outcomes of behavior ) and underuse of other, potentially more effective techniques (eg, feedback on behavior and goal setting [ behavior ]) highlight mHealth’s focus on simpler technologies at the expense of innovation and efficacy. Collaborations between behavioral scientists, care teams, patients, caregivers, and industry could answer these questions and produce mHealth solutions that are transformative and effective.

When incorporating BCTs that are expected to effectively and appropriately generalize to a range of complex medical conditions and associated regimens, mHealth intervention developers must still consider the unique needs of the target population. In CF, for example, highly effect CF transmembrane conductance regulator modulator therapies have the potential to simplify the regimen and reduce treatment burden [ 33 , 34 ]. The implementation of key BCTs may need to be adapted as new therapies roll out, although the core theory behind the BCT itself is not expected to change. It is critical to build the scientific evidence base for effective adherence and self-management mHealth interventions that maintain pace with rapidly advancing medical management across complex medical conditions.

Acknowledgments

We are thankful for the contributions of Brandi Blackshear, Angela Green, Kirsten Kulik, and Anne Bowen to this important investigation.

Conflicts of Interest

CKE, KR, CS, E McWilliams, MS, KM, E Muther, GRO, TS, and DY receive or received salary and/or grant support from the Cystic Fibrosis Foundation’s Success with Therapies Research Consortium. GSS and KR receive honoraria and/or speaker fees from Vertex Pharmaceuticals. CG is an employee of the Cystic Fibrosis Foundation, which funded this research through the Success with Therapies Research Consortium. GRO receives grants from the Kael Pediatric Research Institute, National Institutes of Health, Health Resources and Services Administration’s Children’s National Research Institute, and the Alabama Department of Public Health, in addition to grant support from the Cystic Fibrosis Foundation. She also receives consulting fees from International Biophysics Corporation. CS serves as an unpaid advisor and consultant to MMNTS, Inc in addition to receiving salary support from the Cystic Fibrosis Foundation’s Success with Therapies Research Consortium. MS receives grant support from Cystic Fibrosis Foundation and Anagram Therapeutics, Inc. AD, RG, IK, MLM, and ML declare no conflicts of interest.

Additional details on search and screening strategies, study characteristics, adherence measure types, behavior change technique definitions, and behavior change techniques by intervention target.

The most common BCTs (>5% of all abstracted studies) by the direction of results in the overall sample. BCTs with >10% difference in how often they appear in studies with positive results compared to no-impact results are highlighted in red boxes. BCT: behavior change technique.

The most common BCTs (>5% of all abstracted studies) by the direction of results in adult-only samples. BCTs with >10% difference in how often they appear in studies with positive results compared to no-impact results are highlighted in red boxes. BCT: behavior change technique.

The most common BCTs (>5% of all abstracted studies) by the direction of results in adolescent and young adult samples. BCTs with >10% difference in how often they appear in studies with positive results compared to no-impact results are highlighted in red boxes. BCT: behavior change technique.

The most common BCTs (>5% of all abstracted studies) by the direction of results in interventions explicitly targeting medicine taking. BCTs with >10% difference in how often they appear in studies with positive results compared to no-impact results are highlighted in red boxes. BCT: behavior change technique.

The most common BCTs (>5% of all abstracted studies) by the direction of results in interventions targeting self-management or other adherence targets. BCTs with >10% difference in how often they appear in studies with positive results compared to no-impact results are highlighted in red boxes. BCT: behavior change technique.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) 2020 checklist.

• WHO Global Observatory for eHealth. mHealth: new horizons for health through mobile technologies: second global survey on eHealth. World Health Organization. 2011. URL: https://iris.who.int/handle/10665/44607 [Accessed 2023-03-23]
• Ahmed I, Ahmad NS, Ali S, et al. Medication adherence apps: review and content analysis. JMIR Mhealth Uhealth. Mar 16, 2018;6(3):e62. [ CrossRef ] [ Medline ]
• Aguiar M, Trujillo M, Chaves D, Álvarez R, Epelde G. mHealth apps using behavior change techniques to self-report data: systematic review. JMIR Mhealth Uhealth. Sep 9, 2022;10(9):e33247. [ CrossRef ] [ Medline ]
• Carmody JK, Denson LA, Hommel KA. Content and usability evaluation of medication adherence mobile applications for use in pediatrics. J Pediatr Psychol. Apr 1, 2019;44(3):333-342. [ CrossRef ] [ Medline ]
• Morrissey EC, Corbett TK, Walsh JC, Molloy GJ. Behavior change techniques in apps for medication adherence: a content analysis. Am J Prev Med. May 2016;50(5):e143-e146. [ CrossRef ] [ Medline ]
• Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. Aug 2013;46(1):81-95. [ CrossRef ] [ Medline ]
• Guo J, Garratt A, Hill A. Worldwide rates of diagnosis and effective treatment for cystic fibrosis. J Cyst Fibros. May 2022;21(3):456-462. [ CrossRef ] [ Medline ]
• Bishay LC, Sawicki GS. Strategies to optimize treatment adherence in adolescent patients with cystic fibrosis. Adolesc Health Med Ther. Oct 21, 2016;7:117-124. [ CrossRef ] [ Medline ]
• Eakin MN, Bilderback A, Boyle MP, Mogayzel PJ, Riekert KA. Longitudinal association between medication adherence and lung health in people with cystic fibrosis. J Cyst Fibros. Jul 2011;10(4):258-264. [ CrossRef ] [ Medline ]
• Quittner AL, Zhang J, Marynchenko M, et al. Pulmonary medication adherence and health-care use in cystic fibrosis. Chest. Jul 2014;146(1):142-151. [ CrossRef ] [ Medline ]
• Simon SL, Duncan CL, Horky SC, Nick TG, Castro MM, Riekert KA. Body satisfaction, nutritional adherence, and quality of life in youth with cystic fibrosis. Pediatr Pulmonol. Nov 2011;46(11):1085-1092. [ CrossRef ] [ Medline ]
• Oates GR, Stepanikova I, Gamble S, Gutierrez HH, Harris WT. Adherence to airway clearance therapy in pediatric cystic fibrosis: socioeconomic factors and respiratory outcomes. Pediatr Pulmonol. Dec 2015;50(12):1244-1252. [ CrossRef ] [ Medline ]
• Floch J, Zettl A, Fricke L, et al. User needs in the development of a health app ecosystem for self-management of cystic fibrosis: user-centered development approach. JMIR Mhealth Uhealth. May 8, 2018;6(5):e113. [ CrossRef ] [ Medline ]
• Rutland SB, Bergquist RP, Hager A, et al. A mobile health platform for self-management of pediatric cystic fibrosis: qualitative study of adaptation to stakeholder needs and integration in clinical settings. JMIR Form Res. Jan 26, 2021;5(1):e19413. [ CrossRef ] [ Medline ]
• Hilliard ME, Hahn A, Ridge AK, Eakin MN, Riekert KA. User preferences and design recommendations for an mHealth app to promote cystic fibrosis self-management. JMIR Mhealth Uhealth. Oct 24, 2014;2(4):e44. [ CrossRef ] [ Medline ]
• Hatem S, Long JC, Best S, Fehlberg Z, Nic Giolla Easpaig B, Braithwaite J. Mobile apps for people with rare diseases: review and quality assessment using Mobile App Rating Scale. J Med Internet Res. Jul 26, 2022;24(7):e36691. [ CrossRef ] [ Medline ]
• Nieuwlaat R, Wilczynski N, Navarro T, et al. Interventions for enhancing medication adherence. Cochrane Database Syst Rev. Nov 20, 2014;2014(11):CD000011. [ CrossRef ] [ Medline ]
• Mellon L, Doyle F, Hickey A, et al. Interventions for increasing immunosuppressant medication adherence in solid organ transplant recipients. Cochrane Database Syst Rev. Sep 12, 2022;9(9):CD012854. [ CrossRef ] [ Medline ]
• Al-Aqeel S, Gershuni O, Al-Sabhan J, Hiligsmann M. Strategies for improving adherence to antiepileptic drug treatment in people with epilepsy. Cochrane Database Syst Rev. Oct 22, 2020;10(10):CD008312. [ CrossRef ] [ Medline ]
• Hedges LV, Tipton E. Meta-analysis. In: Steptoe A, editor. Handbook of Behavioral Medicine. Springer; 2010;909-921. [ CrossRef ]
• Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. Jul 2019;95:103208. [ CrossRef ] [ Medline ]
• Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. Apr 2009;42(2):377-381. [ CrossRef ] [ Medline ]
• Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. Aug 28, 2019.:l4898. [ CrossRef ] [ Medline ]
• Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. Oct 12, 2016.:i4919. [ CrossRef ] [ Medline ]
• van Rhoon L, Byrne M, Morrissey E, Murphy J, McSharry J. A systematic review of the behaviour change techniques and digital features in technology-driven type 2 diabetes prevention interventions. Digit Health. Mar 24, 2020;6:2055207620914427. [ CrossRef ] [ Medline ]
• Bond Z, Scanlon T, Judah G. Systematic review of RCTs assessing the effectiveness of mHealth interventions to improve statin medication adherence: using the behaviour-change technique taxonomy to identify the techniques that improve adherence. Healthcare (Basel). Sep 28, 2021;9(10):1282. [ CrossRef ] [ Medline ]
• Demonceau J, Ruppar T, Kristanto P, et al. Identification and assessment of adherence-enhancing interventions in studies assessing medication adherence through electronically compiled drug dosing histories: a systematic literature review and meta-analysis. Drugs. May 2013;73(6):545-562. [ CrossRef ] [ Medline ]
• Zhao YY, Dang FP, Zhai TT, Li HJ, Wang RJ, Ren JJ. The effect of text message reminders on medication adherence among patients with coronary heart disease: a systematic review and meta-analysis. Medicine (Baltimore). Dec 2019;98(52):e18353. [ CrossRef ] [ Medline ]
• Salisbury KR, Ranpariya VK, Feldman SR. Accountability in reminder-based adherence interventions: a review. Patient Educ Couns. Aug 2022;105(8):2645-2652. [ CrossRef ] [ Medline ]
• Morton RW, Elphick HE, Edwards E, Daw WJ, West NS. Investigating the feasibility of text message reminders to improve adherence to nebulized medication in children and adolescents with cystic fibrosis. Patient Prefer Adherence. May 8, 2017;11:861-869. [ CrossRef ] [ Medline ]
• Sawicki GS, Sellers DE, Robinson WM. High treatment burden in adults with cystic fibrosis: challenges to disease self-management. J Cyst Fibros. Mar 2009;8(2):91-96. [ CrossRef ] [ Medline ]
• Davies G, Rowbotham NJ, Smith S, et al. Characterising burden of treatment in cystic fibrosis to identify priority areas for clinical trials. J Cyst Fibros. May 2020;19(3):499-502. [ CrossRef ] [ Medline ]
• Schechter MS, Sabater-Anaya N, Oster G, et al. Impact of elexacaftor/tezacaftor/ivacaftor on healthcare resource utilization and associated costs among people with cystic fibrosis in the US: a retrospective claims analysis. Pulm Ther. Dec 2023;9(4):479-498. [ CrossRef ] [ Medline ]
• Wajda KE, Roesch EA, Gifford AH. Chronic daily respiratory care needs in people with cystic fibrosis treated with highly effective cystic fibrosis transmembrane conductance regulator modulators. Curr Opin Pulm Med. Nov 1, 2023;29(6):580-586. [ CrossRef ] [ Medline ]

Abbreviations

Edited by Lorraine Buis; submitted 26.05.23; peer-reviewed by Cara Bossley, Uday Kumar Chalwadi; final revised version received 26.01.24; accepted 26.02.24; published 01.05.24.

© Cyd K Eaton, Emma McWilliams, Dana Yablon, Irem Kesim, Renee Ge, Karissa Mirus, Takeera Sconiers, Alfred Donkoh, Melanie Lawrence, Cynthia George, Mary Leigh Morrison, Emily Muther, Gabriela R Oates, Meghana Sathe, Gregory S Sawicki, Carolyn Snell, Kristin Riekert. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 1.5.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/ , as well as this copyright and license information must be included.