journal review methodology

Open access
Published: 08 October 2021

Scoping reviews: reinforcing and advancing the methodology and application

Micah D. J. Peters 1 , 2 , 3 ,
Casey Marnie 1 ,
Heather Colquhoun 4 , 5 ,
Chantelle M. Garritty 6 ,
Susanne Hempel 7 ,
Tanya Horsley 8 ,
Etienne V. Langlois 9 ,
Erin Lillie 10 ,
Kelly K. O’Brien 5 , 11 , 12 ,
Ӧzge Tunçalp 13 ,
Michael G. Wilson 14 , 15 , 16 ,
Wasifa Zarin 17 &
Andrea C. Tricco ORCID: orcid.org/0000-0002-4114-8971 17 , 18 , 19

Systematic Reviews volume 10 , Article number: 263 ( 2021 ) Cite this article

45k Accesses

216 Citations

11 Altmetric

Metrics details

Scoping reviews are an increasingly common approach to evidence synthesis with a growing suite of methodological guidance and resources to assist review authors with their planning, conduct and reporting. The latest guidance for scoping reviews includes the JBI methodology and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses—Extension for Scoping Reviews. This paper provides readers with a brief update regarding ongoing work to enhance and improve the conduct and reporting of scoping reviews as well as information regarding the future steps in scoping review methods development. The purpose of this paper is to provide readers with a concise source of information regarding the difference between scoping reviews and other review types, the reasons for undertaking scoping reviews, and an update on methodological guidance for the conduct and reporting of scoping reviews.

Despite available guidance, some publications use the term ‘scoping review’ without clear consideration of available reporting and methodological tools. Selection of the most appropriate review type for the stated research objectives or questions, standardised use of methodological approaches and terminology in scoping reviews, clarity and consistency of reporting and ensuring that the reporting and presentation of the results clearly addresses the review’s objective(s) and question(s) are critical components for improving the rigour of scoping reviews.

Rigourous, high-quality scoping reviews should clearly follow up to date methodological guidance and reporting criteria. Stakeholder engagement is one area where further work could occur to enhance integration of consultation with the results of evidence syntheses and to support effective knowledge translation. Scoping review methodology is evolving as a policy and decision-making tool. Ensuring the integrity of scoping reviews by adherence to up-to-date reporting standards is integral to supporting well-informed decision-making.

Peer Review reports

Introduction

Given the readily increasing access to evidence and data, methods of identifying, charting and reporting on information must be driven by new, user-friendly approaches. Since 2005, when the first framework for scoping reviews was published, several more detailed approaches (both methodological guidance and a reporting guideline) have been developed. Scoping reviews are an increasingly common approach to evidence synthesis which is very popular amongst end users [ 1 ]. Indeed, one scoping review of scoping reviews found that 53% (262/494) of scoping reviews had government authorities and policymakers as their target end-user audience [ 2 ]. Scoping reviews can provide end users with important insights into the characteristics of a body of evidence, the ways, concepts or terms have been used, and how a topic has been reported upon. Scoping reviews can provide overviews of either broad or specific research and policy fields, underpin research and policy agendas, highlight knowledge gaps and identify areas for subsequent evidence syntheses [ 3 ].

Despite or even potentially because of the range of different approaches to conducting and reporting scoping reviews that have emerged since Arksey and O’Malley’s first framework in 2005, it appears that lack of consistency in use of terminology, conduct and reporting persist [ 2 , 4 ]. There are many examples where manuscripts are titled ‘a scoping review’ without citing or appearing to follow any particular approach [ 5 , 6 , 7 , 8 , 9 ]. This is similar to how many reviews appear to misleadingly include ‘systematic’ in the title or purport to have adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement without doing so. Despite the publication of the PRISMA Extension for Scoping Reviews (PRISMA-ScR) and other recent guidance [ 4 , 10 , 11 , 12 , 13 , 14 ], many scoping reviews continue to be conducted and published without apparent (i.e. cited) consideration of these tools or only cursory reference to Arksey and O’Malley’s original framework. We can only speculate at this stage why many authors appear to be either unaware of or unwilling to adopt more recent methodological guidance and reporting items in their work. It could be that some authors are more familiar and comfortable with the older, less prescriptive framework and see no reason to change. It could be that more recent methodologies such as JBI’s guidance and the PRISMA-ScR appear more complicated and onerous to comply with and so may possibly be unfit for purpose from the perspective of some authors. In their 2005 publication, Arksey and O’Malley themselves called for scoping review (then scoping study) methodology to continue to be advanced and built upon by subsequent authors, so it is interesting to note a persistent resistance or lack of awareness from some authors. Whatever the reason or reasons, we contend that transparency and reproducibility are key markers of high-quality reporting of scoping reviews and that reporting a review’s conduct and results clearly and consistently in line with a recognised methodology or checklist is more likely than not to enhance rigour and utility. Scoping reviews should not be used as a synonym for an exploratory search or general review of the literature. Instead, it is critical that potential authors recognise the purpose and methodology of scoping reviews. In this editorial, we discuss the definition of scoping reviews, introduce contemporary methodological guidance and address the circumstances where scoping reviews may be conducted. Finally, we briefly consider where ongoing advances in the methodology are occurring.

What is a scoping review and how is it different from other evidence syntheses?

A scoping review is a type of evidence synthesis that has the objective of identifying and mapping relevant evidence that meets pre-determined inclusion criteria regarding the topic, field, context, concept or issue under review. The review question guiding a scoping review is typically broader than that of a traditional systematic review. Scoping reviews may include multiple types of evidence (i.e. different research methodologies, primary research, reviews, non-empirical evidence). Because scoping reviews seek to develop a comprehensive overview of the evidence rather than a quantitative or qualitative synthesis of data, it is not usually necessary to undertake methodological appraisal/risk of bias assessment of the sources included in a scoping review. Scoping reviews systematically identify and chart relevant literature that meet predetermined inclusion criteria available on a given topic to address specified objective(s) and review question(s) in relation to key concepts, theories, data and evidence gaps. Scoping reviews are unlike ‘evidence maps’ which can be defined as the figural or graphical presentation of the results of a broad and systematic search to identify gaps in knowledge and/or future research needs often using a searchable database [ 15 ]. Evidence maps can be underpinned by a scoping review or be used to present the results of a scoping review. Scoping reviews are similar to but distinct from other well-known forms of evidence synthesis of which there are many [ 16 ]. Whilst this paper’s purpose is not to go into depth regarding the similarities and differences between scoping reviews and the diverse range of other evidence synthesis approaches, Munn and colleagues recently discussed the key differences between scoping reviews and other common review types [ 3 ]. Like integrative reviews and narrative literature reviews, scoping reviews can include both research (i.e. empirical) and non-research evidence (grey literature) such as policy documents and online media [ 17 , 18 ]. Scoping reviews also address broader questions beyond the effectiveness of a given intervention typical of ‘traditional’ (i.e. Cochrane) systematic reviews or peoples’ experience of a particular phenomenon of interest (i.e. JBI systematic review of qualitative evidence). Scoping reviews typically identify, present and describe relevant characteristics of included sources of evidence rather than seeking to combine statistical or qualitative data from different sources to develop synthesised results.

Similar to systematic reviews, the conduct of scoping reviews should be based on well-defined methodological guidance and reporting standards that include an a priori protocol, eligibility criteria and comprehensive search strategy [ 11 , 12 ]. Unlike systematic reviews, however, scoping reviews may be iterative and flexible and whilst any deviations from the protocol should be transparently reported, adjustments to the questions, inclusion/exclusion criteria and search may be made during the conduct of the review [ 4 , 14 ]. Unlike systematic reviews where implications or recommendations for practice are a key feature, scoping reviews are not designed to underpin clinical practice decisions; hence, assessment of methodological quality or risk of bias of included studies (which is critical when reporting effect size estimates) is not a mandatory step and often does not occur [ 10 , 12 ]. Rapid reviews are another popular review type, but as yet have no consistent, best practice methodology [ 19 ]. Rapid reviews can be understood to be streamlined forms of other review types (i.e. systematic, integrative and scoping reviews) [ 20 ].

Guidance to improve the quality of reporting of scoping reviews

Since the first 2005 framework for scoping reviews (then termed ‘scoping studies’) [ 13 ], the popularity of this approach has grown, with numbers doubling between 2014 and 2017 [ 2 ]. The PRISMA-ScR is the most up-to-date and advanced approach for reporting scoping reviews which is largely based on the popular PRISMA statement and checklist, the JBI methodological guidance and other approaches for undertaking scoping reviews [ 11 ]. Experts in evidence synthesis including authors of earlier guidance for scoping reviews developed the PRISMA-ScR checklist and explanation using a robust and comprehensive approach. Enhancing transparency and uniformity of reporting scoping reviews using the PRISMA-ScR can help to improve the quality and value of a scoping review to readers and end users [ 21 ]. The PRISMA-ScR is not a methodological guideline for review conduct, but rather a complementary checklist to support comprehensive reporting of methods and findings that can be used alongside other methodological guidance [ 10 , 12 , 13 , 14 ]. For this reason, authors who are more familiar with or prefer Arksey and O’Malley’s framework; Levac, Colquhoun and O’Brien’s extension of that framework or JBI’s methodological guidance could each select their preferred methodological approach and report in accordance with the PRISMA-ScR checklist.

Reasons for conducting a scoping review

Whilst systematic reviews sit at the top of the evidence hierarchy, the types of research questions they address are not suitable for every application [ 3 ]. Many indications more appropriately require a scoping review. For example, to explore the extent and nature of a body of literature, the development of evidence maps and summaries; to inform future research and reviews and to identify evidence gaps [ 2 ]. Scoping reviews are particularly useful where evidence is extensive and widely dispersed (i.e. many different types of evidence), or emerging and not yet amenable to questions of effectiveness [ 22 ]. Because scoping reviews are agnostic in terms of the types of evidence they can draw upon, they can be used to bring together and report upon heterogeneous literature—including both empirical and non-empirical evidence—across disciplines within and beyond health [ 23 , 24 , 25 ].

When deciding between whether to conduct a systematic review or a scoping review, authors should have a strong understanding of their differences and be able to clearly identify their review’s precise research objective(s) and/or question(s). Munn and colleagues noted that a systematic review is likely the most suitable approach if reviewers intend to address questions regarding the feasibility, appropriateness, meaningfulness or effectiveness of a specified intervention [ 3 ]. There are also online resources for prospective authors [ 26 ]. A scoping review is probably best when research objectives or review questions involve exploring, identifying, mapping, reporting or discussing characteristics or concepts across a breadth of evidence sources.

Scoping reviews are increasingly used to respond to complex questions where comparing interventions may be neither relevant nor possible [ 27 ]. Often, cost, time, and resources are factors in decisions regarding review type. Whilst many scoping reviews can be quite large with numerous sources to screen and/or include, there is no expectation or possibility of statistical pooling, formal risk of bias rating, and quality of evidence assessment [ 28 , 29 ]. Topics where scoping reviews are necessary abound—for example, government organisations are often interested in the availability and applicability of tools to support health interventions, such as shared decision aids for pregnancy care [ 30 ]. Scoping reviews can also be applied to better understand complex issues related to the health workforce, such as how shift work impacts employee performance across diverse occupational sectors, which involves a diversity of evidence types as well as attention to knowledge gaps [ 31 ]. Another example is where more conceptual knowledge is required, for example, identifying and mapping existing tools [ 32 ]. Here, it is important to understand that scoping reviews are not the same as ‘realist reviews’ which can also be used to examine how interventions or programmes work. Realist reviews are typically designed to ellucide the theories that underpin a programme, examine evidence to reveal if and how those theories are relevant and explain how the given programme works (or not) [ 33 ].

Increased demand for scoping reviews to underpin high-quality knowledge translation across many disciplines within and beyond healthcare in turn fuels the need for consistency, clarity and rigour in reporting; hence, following recognised reporting guidelines is a streamlined and effective way of introducing these elements [ 34 ]. Standardisation and clarity of reporting (such as by using a published methodology and a reporting checklist—the PRISMA-ScR) can facilitate better understanding and uptake of the results of scoping reviews by end users who are able to more clearly understand the differences between systematic reviews, scoping reviews and literature reviews and how their findings can be applied to research, practice and policy.

Future directions in scoping reviews

The field of evidence synthesis is dynamic. Scoping review methodology continues to evolve to account for the changing needs and priorities of end users and the requirements of review authors for additional guidance regarding terminology, elements and steps of scoping reviews. Areas where ongoing research and development of scoping review guidance are occurring include inclusion of consultation with stakeholder groups such as end users and consumer representatives [ 35 ], clarity on when scoping reviews are the appropriate method over other synthesis approaches [ 3 ], approaches for mapping and presenting results in ways that clearly address the review’s research objective(s) and question(s) [ 29 ] and the assessment of the methodological quality of scoping reviews themselves [ 21 , 36 ]. The JBI Scoping Review Methodology group is currently working on this research agenda.

Consulting with end users, experts, or stakeholders has been a suggested but optional component of scoping reviews since 2005. Many of the subsequent approaches contained some reference to this useful activity. Stakeholder engagement is however often lost to the term ‘review’ in scoping reviews. Stakeholder engagement is important across all knowledge synthesis approaches to ensure relevance, contextualisation and uptake of research findings. In fact, it underlines the concept of integrated knowledge translation [ 37 , 38 ]. By including stakeholder consultation in the scoping review process, the utility and uptake of results may be enhanced making reviews more meaningful to end users. Stakeholder consultation can also support integrating knowledge translation efforts, facilitate identifying emerging priorities in the field not otherwise captured in the literature and may help build partnerships amongst stakeholder groups including consumers, researchers, funders and end users. Development in the field of evidence synthesis overall could be inspired by the incorporation of stakeholder consultation in scoping reviews and lead to better integration of consultation and engagement within projects utilising other synthesis methodologies. This highlights how further work could be conducted into establishing how and the extent to which scoping reviews have contributed to synthesising evidence and advancing scientific knowledge and understandings in a more general sense.

Currently, many methodological papers for scoping reviews are published in healthcare focussed journals and associated disciplines [ 6 , 39 , 40 , 41 , 42 , 43 ]. Another area where further work could also occur is to gain greater understanding on how scoping reviews and scoping review methodology is being used across disciplines beyond healthcare including how authors, reviewers and editors understand, recommend or utilise existing guidance for undertaking and reporting scoping reviews.

Whilst available guidance for the conduct and reporting of scoping review has evolved over recent years, opportunities remain to further enhance and progress the methodology, uptake and application. Despite existing guidance, some publications using the term ‘scoping review’ continue to be conducted without apparent consideration of available reporting and methodological tools. Because consistent and transparent reporting is widely recongised as important for supporting rigour, reproducibility and quality in research, we advocate for authors to use a stated scoping review methodology and to transparently report their conduct by using the PRISMA-ScR. Selection of the most appropriate review type for the stated research objectives or questions, standardising the use of methodological approaches and terminology in scoping reviews, clarity and consistency of reporting and ensuring that the reporting and presentation of the results clearly addresses the authors’ objective(s) and question(s) are also critical components for improving the rigour of scoping reviews. We contend that whilst the field of evidence synthesis and scoping reviews continues to evolve, use of the PRISMA-ScR is a valuable and practical tool for enhancing the quality of scoping reviews, particularly in combination with other methodological guidance [ 10 , 12 , 44 ]. Scoping review methodology is developing as a policy and decision-making tool, and so ensuring the integrity of these reviews by adhering to the most up-to-date reporting standards is integral to supporting well informed decision-making. As scoping review methodology continues to evolve alongside understandings regarding why authors do or do not use particular methodologies, we hope that future incarnations of scoping review methodology continues to provide useful, high-quality evidence to end users.

Availability of data and materials

All data and materials are available upon request.

Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5(4):371–85.

Article Google Scholar

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16:15.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Peters M, Marnie C, Tricco A, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26.

Paiva L, Dalmolin GL, Andolhe R, dos Santos W. Absenteeism of hospital health workers: scoping review. Av enferm. 2020;38(2):234–48.

Visonà MW, Plonsky L. Arabic as a heritage language: a scoping review. Int J Biling. 2019;24(4):599–615.

McKerricher L, Petrucka P. Maternal nutritional supplement delivery in developing countries: a scoping review. BMC Nutr. 2019;5(1):8.

Article CAS Google Scholar

Fusar-Poli P, Salazar de Pablo G, De Micheli A, et al. What is good mental health? A scoping review. Eur Neuropsychopharmacol. 2020;31:33–46.

Jowsey T, Foster G, Cooper-Ioelu P, Jacobs S. Blended learning via distance in pre-registration nursing education: a scoping review. Nurse Educ Pract. 2020;44:102775.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid-based Healthc. 2015;13(3):141–6.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). In: Aromataris E, Munn Z, editors. JBI manual for evidence synthesis: JBI; 2020.

Google Scholar

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.

Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5(1):28.

Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements. Health Inf Libr J. 2019;36(3):202–22.

Brady BR, De La Rosa JS, Nair US, Leischow SJ. Electronic cigarette policy recommendations: a scoping review. Am J Health Behav. 2019;43(1):88–104.

Truman E, Elliott C. Identifying food marketing to teenagers: a scoping review. Int J Behav Nutr Phys Act. 2019;16(1):67.

Tricco AC, Antony J, Zarin W, et al. A scoping review of rapid review methods. BMC Med. 2015;13(1):224.

Moher D, Stewart L, Shekelle P. All in the family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more. Syst Rev. 2015;4(1):183.

Tricco AC, Zarin W, Ghassemi M, et al. Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis. J Clin Epidemiol. 2018;96:133–42.

Barker M, Adelson P, Peters MDJ, Steen M. Probiotics and human lactational mastitis: a scoping review. Women Birth. 2020;33(6):e483–e491.

O’Donnell N, Kappen DL, Fitz-Walter Z, Deterding S, Nacke LE, Johnson D. How multidisciplinary is gamification research? Results from a scoping review. Extended abstracts publication of the annual symposium on computer-human interaction in play. Amsterdam: Association for Computing Machinery; 2017. p. 445–52.

O’Flaherty J, Phillips C. The use of flipped classrooms in higher education: a scoping review. Internet High Educ. 2015;25:85–95.

Di Pasquale V, Miranda S, Neumann WP. Ageing and human-system errors in manufacturing: a scoping review. Int J Prod Res. 2020;58(15):4716–40.

Knowledge Synthesis Team. What review is right for you? 2019. https://whatreviewisrightforyou.knowledgetranslation.net/

Lv M, Luo X, Estill J, et al. Coronavirus disease (COVID-19): a scoping review. Euro Surveill. 2020;25(15):2000125.

Shemilt I, Simon A, Hollands GJ, et al. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods. 2014;5(1):31–49.

Khalil H, Bennett M, Godfrey C, McInerney P, Munn Z, Peters M. Evaluation of the JBI scoping reviews methodology by current users. Int J Evid-based Healthc. 2020;18(1):95–100.

Kennedy K, Adelson P, Fleet J, et al. Shared decision aids in pregnancy care: a scoping review. Midwifery. 2020;81:102589.

Dall’Ora C, Ball J, Recio-Saucedo A, Griffiths P. Characteristics of shift work and their impact on employee performance and wellbeing: a literature review. Int J Nurs Stud. 2016;57:12–27.

Feo R, Conroy T, Wiechula R, Rasmussen P, Kitson A. Instruments measuring behavioural aspects of the nurse–patient relationship: a scoping review. J Clin Nurs. 2020;29(11-12):1808–21.

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci. 2012;7(1):33.

Colquhoun HL, Levac D, O’Brien KK, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67(12):1291–4.

Tricco AC, Zarin W, Rios P, et al. Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review. Implement Sci. 2018;13(1):31.

Cooper S, Cant R, Kelly M, et al. An evidence-based checklist for improving scoping review quality. Clin Nurs Res. 2021;30(3):230–240.

Pollock A, Campbell P, Struthers C, et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Rev. 2018;7(1):208.

Tricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open. 2016;6(12):e013929.

Denton M, Borrego M. Funds of knowledge in STEM education: a scoping review. Stud Eng Educ. 2021;1(2):71–92.

Masta S, Secules S. When critical ethnography leaves the field and enters the engineering classroom: a scoping review. Stud Eng Educ. 2021;2(1):35–52.

Li Y, Marier-Bienvenue T, Perron-Brault A, Wang X, Pare G. Blockchain technology in business organizations: a scoping review. In: Proceedings of the 51st Hawaii international conference on system sciences ; 2018. https://core.ac.uk/download/143481400.pdf

Houlihan M, Click A, Wiley C. Twenty years of business information literacy research: a scoping review. Evid. Based Libr. Inf. Pract. 2020;15(4):124–163.

Plug I, Stommel W, Lucassen P, Hartman T, Van Dulmen S, Das E. Do women and men use language differently in spoken face-to-face interaction? A scoping review. Rev Commun Res. 2021;9:43–79.

McGowan J, Straus S, Moher D, et al. Reporting scoping reviews - PRISMA ScR extension. J Clin Epidemiol. 2020;123:177–9.

Download references

Acknowledgements

The authors would like to acknowledge the other members of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) working group as well as Shazia Siddiqui, a research assistant in the Knowledge Synthesis Team in the Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto.

The authors declare that no specific funding was received for this work. Author ACT declares that she is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. KKO is supported by a Canada Research Chair in Episodic Disability and Rehabilitation with the Canada Research Chairs Program.

Author information

Authors and affiliations.

University of South Australia, UniSA Clinical and Health Sciences, Rosemary Bryant AO Research Centre, Playford Building P4-27, City East Campus, North Terrace, Adelaide, 5000, South Australia

Micah D. J. Peters & Casey Marnie

Adelaide Nursing School, Faculty of Health and Medical Sciences, The University of Adelaide, 101 Currie St, Adelaide, 5001, South Australia

Micah D. J. Peters

The Centre for Evidence-based Practice South Australia (CEPSA): a Joanna Briggs Institute Centre of Excellence, Faculty of Health and Medical Sciences, The University of Adelaide, 5006, Adelaide, South Australia

Department of Occupational Science and Occupational Therapy, University of Toronto, Terrence Donnelly Health Sciences Complex, 3359 Mississauga Rd, Toronto, Ontario, L5L 1C6, Canada

Heather Colquhoun

Rehabilitation Sciences Institute (RSI), University of Toronto, St. George Campus, 160-500 University Avenue, Toronto, Ontario, M5G 1V7, Canada

Heather Colquhoun & Kelly K. O’Brien

Knowledge Synthesis Group, Ottawa Hospital Research Institute, 1053 Carling Avenue, Ottawa, Ontario, K1Y 4E9, Canada

Chantelle M. Garritty

Southern California Evidence Review Center, University of Southern California, Los Angeles, CA, 90007, USA

Susanne Hempel

Royal College of Physicians and Surgeons of Canada, 774 Echo Drive, Ottawa, Ontario, K1S 5N8, Canada

Tanya Horsley

Partnership for Maternal, Newborn and Child Health (PMNCH), World Health Organisation, Avenue Appia 20, 1211, Geneva, Switzerland

Etienne V. Langlois

Sunnybrook Research Institute, 2075 Bayview Ave, Toronto, Ontario, M4N 3M5, Canada

Erin Lillie

Department of Physical Therapy, University of Toronto, St. George Campus, 160-500 University Avenue, Toronto, Ontario, M5G 1V7, Canada

Kelly K. O’Brien

Institute of Health Policy, Management and Evaluation (IHPME), University of Toronto, St. George Campus, 155 College Street 4th Floor, Toronto, Ontario, M5T 3M6, Canada

UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), Department of Sexual and Reproductive Health and Research, World Health Organisation, Avenue Appia 20, 1211, Geneva, Switzerland

Ӧzge Tunçalp

McMaster Health Forum, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S 4L8, Canada

Michael G. Wilson

Department of Health Evidence and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S 4L8, Canada

Centre for Health Economics and Policy Analysis, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S 4L8, Canada

Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, Ontario, M5B 1T8, Canada

Wasifa Zarin & Andrea C. Tricco

Epidemiology Division and Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, 155 College St, Room 500, Toronto, Ontario, M5T 3M7, Canada

Andrea C. Tricco

Queen’s Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, School of Nursing, Queen’s University, 99 University Ave, Kingston, Ontario, K7L 3N6, Canada

You can also search for this author in PubMed Google Scholar

Contributions

MDJP, CM, HC, CMG, SH, TH, EVL, EL, KKO, OT, MGW, WZ and AT all made substantial contributions to the conception, design and drafting of the work. MDJP and CM prepared the final version of the manuscript. All authors reviewed and approved the final version of the manuscript.

Corresponding author

Correspondence to Andrea C. Tricco .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

Author ACT is an Associate Editor for the journal. All other authors declare no conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Peters, M.D.J., Marnie, C., Colquhoun, H. et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Rev 10 , 263 (2021). https://doi.org/10.1186/s13643-021-01821-3

Download citation

Received : 29 January 2021

Accepted : 27 September 2021

Published : 08 October 2021

DOI : https://doi.org/10.1186/s13643-021-01821-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Scoping reviews
Evidence synthesis
Research methodology
Reporting guidelines
Methodological guidance

Systematic Reviews

ISSN: 2046-4053

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
PMID: 30089228
DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

LinkOut - more resources

Full text sources.

Ingenta plc
Ovid Technologies, Inc.

Other Literature Sources

scite Smart Citations

Miscellaneous

NCI CPTAC Assay Portal
Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Open access
Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

Lawrence Mbuagbaw ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
Daeria O. Lawson 1 ,
Livia Puljak 4 ,
David B. Allison 5 &
Lehana Thabane 1 , 2 , 6 , 7 , 8

BMC Medical Research Methodology volume 20 , Article number: 226 ( 2020 ) Cite this article

41k Accesses

53 Citations

60 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig. 1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig. 2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed Google Scholar

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed PubMed Central Google Scholar

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS PubMed Google Scholar

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS Google Scholar

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS PubMed PubMed Central Google Scholar

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Methodological study
Meta-epidemiology
Research methods
Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

General enquiries: [email protected]

How to Review a Journal Article

For many kinds of assignments, like a literature review , you may be asked to offer a critique or review of a journal article. This is an opportunity for you as a scholar to offer your qualified opinion and evaluation of how another scholar has composed their article, argument, and research. That means you will be expected to go beyond a simple summary of the article and evaluate it on a deeper level. As a college student, this might sound intimidating. However, as you engage with the research process, you are becoming immersed in a particular topic, and your insights about the way that topic is presented are valuable and can contribute to the overall conversation surrounding your topic.

IMPORTANT NOTE!!

Some disciplines, like Criminal Justice, may only want you to summarize the article without including your opinion or evaluation. If your assignment is to summarize the article only, please see our literature review handout.

Before getting started on the critique, it is important to review the article thoroughly and critically. To do this, we recommend take notes, annotating , and reading the article several times before critiquing. As you read, be sure to note important items like the thesis, purpose, research questions, hypotheses, methods, evidence, key findings, major conclusions, tone, and publication information. Depending on your writing context, some of these items may not be applicable.

Questions to Consider

To evaluate a source, consider some of the following questions. They are broken down into different categories, but answering these questions will help you consider what areas to examine. With each category, we recommend identifying the strengths and weaknesses in each since that is a critical part of evaluation.

Evaluating Purpose and Argument

How well is the purpose made clear in the introduction through background/context and thesis?
How well does the abstract represent and summarize the article’s major points and argument?
How well does the objective of the experiment or of the observation fill a need for the field?
How well is the argument/purpose articulated and discussed throughout the body of the text?
How well does the discussion maintain cohesion?

Evaluating the Presentation/Organization of Information

How appropriate and clear is the title of the article?
Where could the author have benefited from expanding, condensing, or omitting ideas?
How clear are the author’s statements? Challenge ambiguous statements.
What underlying assumptions does the author have, and how does this affect the credibility or clarity of their article?
How objective is the author in his or her discussion of the topic?
How well does the organization fit the article’s purpose and articulate key goals?

Evaluating Methods

How appropriate are the study design and methods for the purposes of the study?
How detailed are the methods being described? Is the author leaving out important steps or considerations?
Have the procedures been presented in enough detail to enable the reader to duplicate them?

Evaluating Data

Scan and spot-check calculations. Are the statistical methods appropriate?
Do you find any content repeated or duplicated?
How many errors of fact and interpretation does the author include? (You can check on this by looking up the references the author cites).
What pertinent literature has the author cited, and have they used this literature appropriately?

Following, we have an example of a summary and an evaluation of a research article. Note that in most literature review contexts, the summary and evaluation would be much shorter. This extended example shows the different ways a student can critique and write about an article.

Chik, A. (2012). Digital gameplay for autonomous foreign language learning: Gamers’ and language teachers’ perspectives. In H. Reinders (ed.), Digital games in language learning and teaching (pp. 95-114). Eastbourne, UK: Palgrave Macmillan.

Be sure to include the full citation either in a reference page or near your evaluation if writing an annotated bibliography .

In Chik’s article “Digital Gameplay for Autonomous Foreign Language Learning: Gamers’ and Teachers’ Perspectives”, she explores the ways in which “digital gamers manage gaming and gaming-related activities to assume autonomy in their foreign language learning,” (96) which is presented in contrast to how teachers view the “pedagogical potential” of gaming. The research was described as an “umbrella project” consisting of two parts. The first part examined 34 language teachers’ perspectives who had limited experience with gaming (only five stated they played games regularly) (99). Their data was recorded through a survey, class discussion, and a seven-day gaming trial done by six teachers who recorded their reflections through personal blog posts. The second part explored undergraduate gaming habits of ten Hong Kong students who were regular gamers. Their habits were recorded through language learning histories, videotaped gaming sessions, blog entries of gaming practices, group discussion sessions, stimulated recall sessions on gaming videos, interviews with other gamers, and posts from online discussion forums. The research shows that while students recognize the educational potential of games and have seen benefits of it in their lives, the instructors overall do not see the positive impacts of gaming on foreign language learning.

The summary includes the article’s purpose, methods, results, discussion, and citations when necessary.

This article did a good job representing the undergraduate gamers’ voices through extended quotes and stories. Particularly for the data collection of the undergraduate gamers, there were many opportunities for an in-depth examination of their gaming practices and histories. However, the representation of the teachers in this study was very uneven when compared to the students. Not only were teachers labeled as numbers while the students picked out their own pseudonyms, but also when viewing the data collection, the undergraduate students were more closely examined in comparison to the teachers in the study. While the students have fifteen extended quotes describing their experiences in their research section, the teachers only have two of these instances in their section, which shows just how imbalanced the study is when presenting instructor voices.

Some research methods, like the recorded gaming sessions, were only used with students whereas teachers were only asked to blog about their gaming experiences. This creates a richer narrative for the students while also failing to give instructors the chance to have more nuanced perspectives. This lack of nuance also stems from the emphasis of the non-gamer teachers over the gamer teachers. The non-gamer teachers’ perspectives provide a stark contrast to the undergraduate gamer experiences and fits neatly with the narrative of teachers not valuing gaming as an educational tool. However, the study mentioned five teachers that were regular gamers whose perspectives are left to a short section at the end of the presentation of the teachers’ results. This was an opportunity to give the teacher group a more complex story, and the opportunity was entirely missed.

Additionally, the context of this study was not entirely clear. The instructors were recruited through a master’s level course, but the content of the course and the institution’s background is not discussed. Understanding this context helps us understand the course’s purpose(s) and how those purposes may have influenced the ways in which these teachers interpreted and saw games. It was also unclear how Chik was connected to this masters’ class and to the students. Why these particular teachers and students were recruited was not explicitly defined and also has the potential to skew results in a particular direction.

Overall, I was inclined to agree with the idea that students can benefit from language acquisition through gaming while instructors may not see the instructional value, but I believe the way the research was conducted and portrayed in this article made it very difficult to support Chik’s specific findings.

Some professors like you to begin an evaluation with something positive but isn’t always necessary.

The evaluation is clearly organized and uses transitional phrases when moving to a new topic.

This evaluation includes a summative statement that gives the overall impression of the article at the end, but this can also be placed at the beginning of the evaluation.

This evaluation mainly discusses the representation of data and methods. However, other areas, like organization, are open to critique.

Research Methods

Getting Started
Literature Review Research
Research Design
Research Design By Discipline
SAGE Research Methods
Teaching with SAGE Research Methods

Literature Review

What is a Literature Review?
What is NOT a Literature Review?
Purposes of a Literature Review
Types of Literature Reviews
Literature Reviews vs. Systematic Reviews
Systematic vs. Meta-Analysis

Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

Summarizes and analyzes previous research relevant to a topic
Includes scholarly books and articles published in academic journals
Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

Help gather ideas or information
Keep up to date in current trends and findings
Help develop new questions

A literature review is important because it:

Explains the background of research on a topic.
Demonstrates why a topic is significant to a subject area.
Helps focus your own research questions or problems
Discovers relationships between research studies/ideas.
Suggests unexplored ideas or populations
Identifies major themes, concepts, and researchers on a topic.
Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University

Keep in mind the following, a literature review is NOT:

Not an essay

Not an annotated bibliography in which you summarize each article that you have reviewed. A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper where you select resources to support one side of an issue versus another. A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

provides thorough knowledge of previous studies; introduces seminal works.
helps focus one’s own research topic.
identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
helps the researcher avoid repetition of earlier research.
suggests unexplored populations.
determines whether past studies agree or disagree; identifies controversy in the literature.
tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015), Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Diagram for "What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters"

Systematic review or meta-analysis?

A systematic review answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A meta-analysis is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

clearly stated objectives with pre-defined eligibility criteria for studies
explicit, reproducible methodology
a systematic search that attempts to identify all studies
assessment of the validity of the findings of the included studies (e.g. risk of bias)
systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis.

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. More information on meta-analyses can be found in Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies. It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings. Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted. In that case, an integrative review is an appropriate strategy.

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

<< Previous: Getting Started
Next: Research Design >>
Last Updated: Jul 15, 2024 10:34 AM
URL: https://guides.lib.udel.edu/researchmethods

Steps for Conducting a Scoping Review

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

What is a Scoping Review?

A scoping review is a type of research synthesis that maps the existing literature on a broad topic to identify key concepts, research gaps, and types of evidence.

This mapping exercise involves systematically searching for, identifying, and charting relevant literature to understand its characteristics, such as the volume of research, types of studies conducted, key concepts addressed, and prevalent research gaps.

Unlike systematic reviews, which aim to answer specific questions, scoping reviews are exploratory and often used to assess the extent of available evidence and inform future research directions. They involve comprehensive searches and data extraction but do not typically include a detailed synthesis of findings or a critical appraisal of study quality.

When a scoping review methodology would be appropriate:

Scoping reviews can be used as a preliminary step to a systematic review , helping to identify the types of evidence available, potential research questions, and relevant inclusion criteria.

They can save time and resources by identifying potential challenges or limitations before embarking on a full systematic review.

Scoping reviews can help clarify key concepts/definitions in the literature. If a research area has inconsistent terminology or definitions, a scoping review can map out how different concepts are used and potentially propose a unified understanding. This can help refine the focus and scope of a subsequent systematic review.

To determine if a systematic review is feasible and worthwhile . By identifying the breadth of evidence, researchers can gauge whether there is sufficient literature to warrant a full systematic review.
To identify gaps in the existing research . Scoping reviews can highlight areas where little or no research has been conducted, helping inform future research priorities.
To clarify key concepts and definitions in the field . This can help refine the focus and scope of a subsequent systematic review.
To examine how research is conducted on a certain topic . This can inform the methodology of a future systematic review
To refine and narrow down research questions . The broad overview provided by a scoping review can help researchers develop more specific, focused questions for a systematic review.

When not to choose a scoping review methodology:

If a systematic review already exists on the topic: A systematic review will offer a more rigorous and comprehensive analysis of the evidence if one is already available.
Examining the range of interventions for a health condition
Identifying types of studies conducted
Noting populations studied
Summarizing outcomes measured

Scoping reviews help identify areas needing further research, whereas systematic reviews aim to draw conclusions about intervention effectiveness.

Methodological Guidelines

Methodological guidelines aim to improve the consistency and transparency of scoping reviews, enabling researchers to synthesize evidence effectively.

Methodological guidelines for scoping reviews have evolved over time:

Arksey and O’Malley (2005) proposed the initial framework.
Levac et al. (2010) refined and extended this framework, offering more detailed guidance.
The Joanna Briggs Institute ( JBI ) further developed the methodology, introducing a more structured and transparent process.

Arksey and O’Malley (2005)	Levac et al. (2010)	Joanna Briggs Institute
6 stages, including optional consultation; most flexible approach	6 stages with more detailed guidance; moderate flexibility	More prescriptive approach with additional elements; most structured
Broad research question	Clearly articulated research question	Clearly defined research question with concept, population, and context
Study selection process not specified	Recommends two reviewers for study selection	Provides detailed guidance on study selection process
Basic data charting	More comprehensive data extraction	Detailed guidance on data extraction with specific tools
Basic summary of findings	Numeric summary and qualitative thematic analysis	Introduces evidence mapping for analysis
Quality assessment not included	Quality assessment not emphasized	Introduces potential for quality appraisal
Optional stakeholder consultation	Recommended stakeholder consultation	Stakeholder consultation as an integral part of the process
Provides basic framework	Offers enhanced detail on methodology	Provides most detailed guidance on conducting scoping reviews

1. Developing review objective(s) & question(s)

A well-defined objective and a set of aligned research questions are crucial for a scoping review’s coherence and direction.

They guide the subsequent steps of the review process, including determining the inclusion and exclusion criteria, developing a search strategy, and guiding data extraction and analysis.

This stage involves a thoughtful and iterative process to ensure that the review’s aims and questions are explicitly stated and closely intertwined.

Defining Objectives:

This step outlines the overarching goals of the scoping review. It explains the rationale behind conducting the review and what the reviewers aim to achieve.

The objective statement should succinctly capture the essence of the review and provide a clear understanding of its purpose.

For instance, a scoping review’s objective might be to map the existing literature on a particular topic and identify knowledge gaps.

“Parents, in particular, greatly influence participation at school, at home and in the community. They undertake many actions to improve their children’s participation in daily life. Understanding the actions of parents and also their challenges and needs will contribute to how society can support these parents and thereby enable the participation of children with physical disabilities. Pediatric rehabilitation, aiming for optimal participation, could benefit from this understanding to improve Family-centered services (FCS)… However, it is unclear what kind of information is available in literature about what parents live through, do, and what kind of problems and needs they have in supporting their child’s participation? For these reasons, a scoping review was conducted in order to systematically map the research done in this area, as well as to identify any existing gaps in knowledge”

Piškur, B., Beurskens, A. J., Jongmans, M. J., Ketelaar, M., Norton, M., Frings, C. A., … & Smeets, R. J. (2012). Parents’ actions, challenges, and needs while enabling participation of children with a physical disability: a scoping review. BMC pediatrics , 12 , 1-13.

Developing Research Questions:

The research question(s) stem from the objectives and provide a focused roadmap for the review. These questions should be answerable through the scoping review process. The research question(s) should be clear, concise, and directly relevant to the overall objectives.

Using Frameworks: While not mandatory, frameworks can be helpful tools to guide the development of objectives and research questions. Frameworks like PCC (Population, Concept, Context).

Population: Clearly define the specific group of individuals or entities that the scoping review will focus on. This could be patients, healthcare professionals, or even organizations.
Concept: Articulate the central idea, topic, or phenomenon that the review aims to investigate. This might include interventions, diagnostic tests, or theoretical models.
Context: Specify the setting, environment, or circumstances relevant to the research question. This could involve geographical locations, healthcare systems, or cultural contexts.

How do cultural beliefs and practices ( C -context) influence the ways in which parents ( P -parents of children with physical disabilities) perceive and address ( C -concept) their children’s physical disabilities?

What are the barriers and facilitators ( C -concept) to mental health service utilization ( C -concept) among veterans ( P -population) experiencing homelessness ( C -context)?

This scoping review aims to summarize what is known in the African scientific literature ( C -context) among cisgender persons ( P ) about a) individual experiences of GBS within health care settings ( C -concept) and b) associations between GBS experiences and health care-related outcomes ( C -concept).

What are the main theoretical and methodological characteristics ( C -concept) of the current literature ( C -context) in the area of stigma and hearing loss and stigma and hearing aids in the elderly population ( P -older adults with acquired hearing impairment), and how should future research proceed in expanding this important field of enquiry?

2. Write A Research Protocol

A research protocol is a detailed plan that outlines the methodology to be employed throughout the review process, detailing steps like documenting results, outlining search strategy, and stating the review’s objective

The protocol should be created a priori (before starting the review) to ensure transparency and reproducibility.

While not mandatory, registering your protocol is highly recommended, e.g. FigShare and Open Science Framework (OSF).

Some journals, such as the Journal of Advanced Nursing , Systematic Reviews , BMC Medical Research Methodology , BMJ Open , and JBI Evidence Synthesis , accept scoping review protocols for publication.

It’s important to note that PROSPERO, the international prospective register of systematic reviews, does not currently accept scoping review protocols for registration.

Registering a scoping review protocol is highly recommended, even if not mandatory, as it promotes transparency, reduces duplication of effort, and helps to prevent publication bias

Example Protocols:

The nutritional care of people living with dementia at home: a protocol for a scoping study
End-of-life care in long-term care homes: A scoping review protocol
Delaying knee flexion following knee arthroplasty surgery: A Scoping Review Protocol

Report in the Methods Section

“Our protocol was drafted using the Preferred Reporting Items for Systematic Reviews and Meta-analysis Protocols (PRISMAP…), which was revised by the research team and members of Health Canada, and was disseminated through our programme’s Twitter account (@KT-Canada) and newsletter to solicit additional feedback. The final protocol was registered prospectively with the Open Science Framework on 6 September 2016 ( https://osf.io/kv9hu/ ).”

Tricco, A. C., Zarin, W., Lillie, E., Pham, B., & Straus, S. E. (2017). Utility of social media and crowd-sourced data for pharmacovigilance: a scoping review protocol. BMJ open , 7 (1), e013474.

“ Our protocol was developed using the scoping review methodological framework proposed by Arksey and O’Malley (2005) [1] and further refined by the Joanna Briggs Institute [3]. The draft protocol was revised upon receiving feedback from the research team, including methodologists and healthcare providers, as well as the peer-review panel of the Canadian Institutes of Health Research. The final version of the protocol is available upon request from the corresponding author. ”

Tricco, A. C., Lillie, E., Zarin, W., O’brien, K., Colquhoun, H., Kastner, M., … & Straus, S. E. (2016). A scoping review on the conduct and reporting of scoping reviews. BMC medical research methodology , 16 , 1-10.

3. Developing eligibility criteria

This step involves developing and aligning the inclusion criteria with the objective(s) and question(s).

By providing transparent and well-justified eligibility criteria, researchers can ensure the replicability of their scoping review and allow readers to assess the relevance and appropriateness of the included sources.

When reporting eligibility criteria, emphasize the importance of clarity, justification, and a clear link to the review’s objectives.

Describe the eligibility criteria with a rationale for why they were selected : It’s crucial to clearly articulate the specific characteristics of sources that make them eligible for inclusion in the review. Each criterion should be accompanied by a rationale explaining why it was chosen. This rationale should be grounded in scientific arguments and clearly demonstrate how the criterion aligns with the review’s objectives.
Identify specific restrictions and provide a rationale : Restrictions, such as date range, language, or publication status, also need clear justification. For instance, limiting the review to articles published within the past ten years might be necessary to capture the most current evidence. Similarly, restricting the review to sources in a specific language, like English, should be justified, acknowledging the potential exclusion of relevant research in other languages.

When specifying the inclusion and exclusion criteria, consider the following aspects:

By using the PCC framework, researchers can systematically establish boundaries for their scoping review, ensuring that the included sources are relevant to the research question. The framework helps to ensure that the eligibility criteria are comprehensive and well-defined, enabling a more focused and meaningful synthesis of the literature

Population : The specific characteristics of the individuals or groups being studied. For instance, a scoping review about interventions for heart failure should specify the intended patient population (e.g., adults with heart failure, elderly patients with heart failure).
Concept : This refers to the central idea, topic, or phenomenon under investigation. In the heart failure example, the concept could be “interventions for heart failure” itself, or it could be narrowed down to a specific type of intervention, such as “exercise interventions for heart failure.”
Context : This element considers the setting or environment in which the concept is being explored. For instance, the context of the heart failure review could be “hospital settings,” “community-based care,” or “telehealth interventions.”

It is important to note that the absence of an explicitly stated framework (e.g. PCC) does not necessarily mean that the authors did not utilize a systematic approach when developing their eligibility criteria. It is possible that they employed a framework implicitly or that their criteria development was guided by other factors.

Iterative Process

The initial set of eligibility criteria outlined in the protocol may be subject to adjustments based on the type and volume of studies identified in the initial searches.

Initial Development : Establish preliminary inclusion and exclusion criteria at the onset of the review based on their existing knowledge of the subject area. This can be adjusted as you become more familiar with the literature and data retrieved during the search process.
Iterative Refinement : Inclusion criteria are refined iteratively based on pilot searches and the evolving understanding of the data. This initial search is crucial as it exposes researchers to a broader range of literature, revealing additional keywords, relevant concepts, and potentially useful search terms that might not have been initially considered.

“ Studies that identified the key terms in the title, abstract, article, or MeSH heading were retained for further examination. Studies published as abstracts, conference proceedings or pilot results published in non-peer-reviewed journals were excluded. In addition, books, book chapters, comments on publications, and dissertations were also excluded. No exclusion criteria were established regarding the type of research design. Inclusion criteria were (a) older adults with progressive hearing loss being the population of interest and (b) the outcome measure was clearly focused on (or at least on some aspects of) stigma regarding hearing loss and/or hearing aids. Although given the descriptive aim of the review, no definitions of stigma and/or hearing aids were set a priori, and all articles including these terms were retrieved, the analysis of the data relied on the most common dimensions of the concept of stigma cited in the literarture: the cognitive dimension (i.e., stereotypes), the emotional dimension (i.e., prejudice) and the behavioral dime. ”

David, D., & Werner, P. (2016). Stigma regarding hearing loss and hearing aids: A scoping review. Stigma and Health , 1 (2), 59.

“ An extensive search was conducted to locate peer-reviewed articles that addressed questions related to parent involvement in organized youth sport. To guide article retrieval, two inclusion criteria were used. First, articles were required to highlight some form of parent involvement in organized youth sport. In the present study, organized youth sport was operationalized as “adultorganized and controlled athletic programs for young people,” wherein “participants are formally organized [and] attend practices and scheduled competitions under the supervision of an adult leader” (Smoll & Smith, 2002, p. xi). In line with this criterion, we did not include physical activity, exercise, physical education, and free play settings, which comprise a substantial volume of research in sport and exercise psychology. We also excluded research that simply collected data on parents or from parents but did not explicitly assess their involvement in their children’s sport participation. Second, articles were required to have been published in peer-reviewed, Englishlanguage, academic journals. As such, we did not include books, chapters, reviews, conceptual papers, conference proceedings, theses and (Jones, 2004) dissertations, or organizational “white papers” in this scoping review. ”

Dorsch, T. E., Wright, E., Eckardt, V. C., Elliott, S., Thrower, S. N., & Knight, C. J. (2021). A history of parent involvement in organized youth sport: A scoping review. Sport, Exercise, and performance psychology , 10 (4), 536.

“…to be included in the review, papers needed to measure or focus on specific dimensions of treatment burden, developed in the conceptual framework (e.g. financial, medication, administrative, lifestyle, healthcare and time/travel). Peer-reviewed journal papers were included if they were: published between the period of 2000–2016, written in English, involved human participants and described a measure for burden of treatment, e.g. including single measurements, measuring and/or incorporating one or two dimensions of burden of treatment. Quantitative, qualitative and mixed-method studies were included in order to consider different aspects of measuring treatment burden. Papers were excluded if they did not fit into the conceptual framework of the study, focused on a communicable chronic condition, for example human immunodeficiency virus infection and acquired immune deficiency syndrome (HIV/AIDS) or substance abuse. Papers talking about carer burden, in addition to patient burden of treatment, were also included.”

Sav, A., Salehi, A., Mair, F. S., & McMillan, S. S. (2017). Measuring the burden of treatment for chronic disease: implications of a scoping review of the literature. BMC medical research methodology , 17 , 1-14.

4. Information Sources

Scoping reviews aim to identify a broad range of relevant studies, including both published and unpublished literature, to provide a comprehensive overview of the topic.

The goal is to be inclusive rather than exhaustive, which differentiates scoping reviews from systematic reviews that seek to collate all empirical evidence fitting pre-specified criteria to answer specific research questions.

Information sources for scoping reviews can include a wide range of resources like scholarly databases, unpublished literature, conference papers, books, and even expert consultations.

Report who developed and executed the search strategy, such as an information specialist or librarian. Mention if the search strategy was peer-reviewed using the Peer Review of Electronic Search Strategies (PRESS) checklist.

Electronic Databases : Make a comprehensive list of all electronic databases you used. Common databases for health-related scoping reviews include: CINAHL, Medline, Embase, PsycINFO, SocINDEX with Full Text, and Web of Science: Core Collections.
Specify date ranges : For each database, note the date range of your search. For example: “MEDLINE was searched from inception to July 30, 2024.”
Grey Literature : In addition to databases, forensic or ‘expansive’ searches can be conducted. This includes: grey literature database searches (e.g. OpenGrey , WorldCat , Ethos ),  conference proceedings, unpublished reports, theses  , clinical trial databases , searches by names of authors of relevant publications.
Citation chasing : If you manually searched specific journals or reference lists, document this. For example: “We hand-searched the reference lists of all included studies and relevant systematic reviews.”
Contacting Experts : If you contacted experts in the field for additional sources, mention this: “We contacted five experts in the field of [topic] to identify any additional relevant studies.”

“To identify potentially relevant documents, the following bibliographic databases were searched from 2004 to June 2015: MEDLINE, EMBASE, LexisNexis Academic, the Legal Scholarship Network, Justis, LegalTrac, QuickLaw, and HeinOnline. The search strategies were drafted by an experienced librarian [name] and further refined through team discussion. The final search strategy for MEDLINE can be found in Additional file 3. The final search results were exported into EndNote, and duplicates were removed by a library technician. The electronic database search was supplemented by searching the Canadian Medical Protective Association website (https://www.cmpa-acpm.ca/en) and scanning relevant reviews.”

Cardoso, R., Zarin, W., Nincic, V., Barber, S. L., Gulmezoglu, A. M., Wilson, C., … & Tricco, A. C. (2017). Evaluative reports on medical malpractice policies in obstetrics: a rapid scoping review. Systematic reviews , 6 , 1-11.

5. Searching for the evidence

Scoping reviews typically start with a broader, more inclusive search strategy. The initial search is intentionally wide-ranging to capture the breadth of available literature on the topic

To balance breadth and depth in your initial search strategy for a scoping review, consider the following tips based on the gathered search results:

Start with a broad initial search : Begin with a broad search across at least two relevant databases (e.g., MEDLINE and Scopus) to capture a wide range of literature. This helps identify the scope of available studies and key themes in the field .
Test and refine your search strategy : After initial searches, review the titles and abstracts of retrieved articles to assess relevance. Analyze the text words and index terms used in these articles to refine your understanding of the topic and identify additional keywords, synonyms, and subject headings to include in subsequent searches .
Multiple Databases : Search across a variety of databases to ensure a comprehensive literature capture. Each database may index different journals and articles, which can help broaden your search results .
Boolean operators: The use of Boolean operators (AND/OR/NEAR/NOT) helps to combine these terms effectively, ensuring that the search strategy is both sensitive and specific. For instance, using “AND” narrows the search to include only results containing both terms, while “OR” expands it to include results containing either term.
Truncation symbols : These broaden the search by capturing variations of a keyword. They function by locating every word that begins with a specific root. For example, if a user was researching interventions for smoking, they might use a truncation symbol to search for “smok*” to retrieve records with the words “smoke,” “smoker,” “smoking,” or “smokes.” This can save time and effort by eliminating the need to input every variation of a word into a database.
Citation chasing : Document the specific studies whose reference lists were examined. Include the titles, authors, and publication years of these studies. Note how you identified articles that cite the studies. This could be through citation databases like Google Scholar, Scopus, or Web of Science.
Detailed documentation : Keep thorough records of your search strategies, including the databases searched, keywords used, and any filters applied. This documentation is crucial for transparency and reproducibility .

” The planned literature search was developed on June 23, 2022. The inclusion and exclusion criteria were further refined, along with electronic databases to identify psychological and education literature (e.g., ProQuest), programs for storing data (i.e., Covidence, n.d. accessed via https://www.covidence.org/) and key search terms (e.g., resistance and transgender). The key search terms were “transgender/trans/LGBT/gender diverse/gender expansive/nonbinary,” “resistance,” and “faith/economic status/ethnicity/gender.” Daniel Abela used terms such as nonbinary, gender diverse, LGBT, and gender expansive to capture the broad spectrum of language employed in the literature when relating to individuals whose gender identification extends beyond conventional norms associated with their assigned sex at birth. Moreover, the authors wanted a diverse sample through an intersectionality lens; therefore, terms such as faith, economic status, and ethnicity were used. These terms were selected as they were deemed by all authors to be most appropriate to evaluate this study’s research question. A complete list of the final search terms and the entire electronic search strategy for the Ovid database are presented in Table 1. ”

Abela, D., Patlamazoglou, L., & Lea, S. (2024). The resistance of transgender and gender expansive people: A scoping review. Psychology of Sexual Orientation and Gender Diversity .

Ovid Search Strategy (Table 1)

transgender.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
trans.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
LGBT.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
gender diverse.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
gender expansive.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
non-binary.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
1 or 2 or 3 or 4 or 5 or 6
resistance.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
faith.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
economic status.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
ethnicity.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
gender identification.mp. [mp = title, abstract, heading word, table of contents, key concepts, original title, tests & measures, mesh word]
10 or 11 or 12 or 13
limit 15 to (peer-reviewed journal and English language and “0110 peer-reviewed journal” and English and yr = “2012-Current”)

Search strategy can also be reported in the appendix. For example: Supplementary A: Search strategy for scoping review .

Citation Chasing Process

Citation chasing involves reviewing the reference lists of included studies and examining articles that cite those studies to identify additional relevant literature. This process helps ensure that you capture a comprehensive view of the research landscape.

If citation chasing leads to the identification of new keywords or concepts, document these adjustments and how they were incorporated into the overall search strategy.

Document the rationale : Clearly state why citation chasing is being conducted. This could include the goal of identifying additional studies that may not have been captured through database searches or to explore the context and impact of key studies.
Reference list review : Document the specific studies whose reference lists were examined. Include the titles, authors, and publication years of these studies.
Citing articles : Note how you identified articles that cite the studies. This could be through citation databases like Google Scholar, Scopus, or Web of Science.
Record number of additional studies identified : Keep a count of how many additional studies were found through citation chasing.
A flowchart : Adapt the PRISMA flow diagram to illustrate the stages of citation chasing, the number of sources identified at each stage, and reasons for exclusion.
Tables : Summarize key information about the sources identified through citation chasing, such as author, year, title, and reasons for inclusion or exclusion.

6. Selecting the evidence

While articles included in a scoping review are selected systematically, it is important to acknowledge that there is no assumption that the evidence reviewed is exhaustive. This is often due to limitations in the search strategy or difficulty locating specific types of sources.

The search results are screened against pre-defined eligibility criteria to determine inclusion in the review.

The goal is to identify relevant studies, with less emphasis on methodological quality. Scoping reviews generally do not appraise the quality of included studies.

Instead, scoping reviews prioritize mapping the existing literature and identifying gaps in research, regardless of the quality of the individual studies.

Two reviewers should independently screen titles and abstracts, removing duplicates and irrelevant studies based on predefined inclusion and exclusion criteria.

Initial screening of titles and abstracts: After applying a strategy to search the literature, the next step involves screening the titles and abstracts of the identified articles against the predefined inclusion and exclusion criteria. During this initial screening, reviewers aim to identify potentially relevant studies while excluding those clearly outside the scope of the review. It is crucial to prioritize over-inclusion at this stage, meaning that reviewers should err on the side of keeping studies even if there is uncertainty about their relevance. This cautious approach helps minimize the risk of inadvertently excluding potentially valuable studies.
Retrieving and assessing full texts: For studies which a definitive decision cannot be made based on the title and abstract alone, reviewers need to obtain the full text of the articles for a comprehensive assessment against the predefined inclusion and exclusion criteria. This stage involves meticulously reviewing the full text of each potentially relevant study to determine its eligibility definitively.
Resolution of disagreements : In cases of disagreement between reviewers regarding a study’s eligibility, a predefined strategy involving consensus-building discussions or arbitration by a third reviewer should be in place to reach a final decision. This collaborative approach ensures a fair and impartial selection process, further strengthening the review’s reliability.

“To increase consistency among reviewers, all reviewers screened the same 50 publications, discussed the results and amended the screening and data extraction manual before beginning screening for this review. Nine reviewers working in pairs sequentially evaluated the titles, abstracts and then full text of all publications identified by our searches for potentially relevant publications. . . . We resolved disagreements on study selection and data extraction by consensus and discussion with other reviewers if needed.”

Duffett, M., Choong, K., Hartling, L., Menon, K., Thabane, L., & Cook, D. J. (2013). Randomized controlled trials in pediatric critical care: a scoping review. Critical care , 17 , 1-9.

7. Extracting the evidence

Charting, also known as data extraction, is a crucial stage in conducting a scoping review.

This process involves systematically collecting relevant information from the sources included in the review using a structured form. It is considered best practice to have at least two reviewers independently extract data from each source

Data charting in scoping reviews differs from data extraction in systematic reviews. While systematic reviews aim to synthesize the results and assess the quality of individual studies, scoping reviews focus on mapping the existing literature and identifying key concepts, themes, and gaps in the research.

Therefore, the data charting process in scoping reviews is typically broader in scope and may involve collecting a wider range of data items compared to the more focused data extraction process used in systematic reviews.

This process goes beyond simply extracting data; it involves characterizing and summarizing research evidence, which ultimately helps identify research gaps.

Develop a Standardized Form: Creating a structured form helps to standardize the selection of sources. The form should incorporate clear questions that align with the eligibility criteria defined in the review protocol. The specific software used to create and manage the form should be specified in the review, with options such as Covidence , EndNote , or JBI SUMARI .
Year of publication
Origin/country of origin (where the study was published or conducted)
Aims/purpose
Study population and sample size (if applicable)
Methodology/methods
Outcomes and details of these (e.g. how measures) (if applicable)
Key findings that relate to the scoping review question/s.
Testing the Form: All reviewers involved in the selection process should participate in testing the standardized form. Screen the titles and abstracts of the identified articles against the predefined inclusion and exclusion criteria.
Sample Size: A random sample of 5–10 citations can be used for the initial calibration of title and abstract screening.
Resolving Inconsistencies: After independent screening, discrepancies between reviewers are identified and discussed. A roundtable discussion involving the review team is an effective method to address these inconsistencies and clarify any ambiguities in the form or eligibility criteria.
Form Refinement: Based on the calibration exercise, the standardized form and its accompanying explanation should be revised and refined as needed to enhance clarity and consistency. A second calibration exercise might be necessary if the desired agreement level, typically 70%–80%, is not achieved or if reviewers require further training.
Number of Reviewers: A minimum of two independent reviewers should be engaged in the screening process.
Duplicate Screening: The review process should clearly state how duplicates were managed, ideally removing them before proceeding to the screening stage.
Verification: The sources describe different approaches to verification, including independent screening by two reviewers followed by comparison of their results or a single reviewer screening followed by verification from another reviewer. The chosen approach and its rationale should be explicitly stated in the scoping review.
Resolving Disagreements: Any disagreements arising during the screening process should be documented and resolved, ideally through discussion and consensus among the reviewers. If consensus cannot be reached, involving a third reviewer to provide an independent assessment can help in making the final decision.
The number of reviewers involved at each stage
How duplicates were addressed
The software used to manage the screening process
How disagreements were resolved
The number of sources excluded at each stage, along with a clear rationale for their exclusion

“Search results for all databases were merged. Duplicates and nonrelated papers were excluded. Titles and abstracts of the remaining papers were assessed against the inclusion and exclusion criteria independently by both authors. The resulting papers were pooled and disagreements were resolved through discussion based on the full text article. Following this stage, a standardized form was used to summarize the information in each article. The variables extracted were: reference/ country, aim of the study, study design, year of publication, and main finding/results.”

“A data-charting form was jointly developed by two reviewers to determine which variables to extract. The two reviewers independently charted the data, discussed the results and continuously updated the data-charting form in an iterative process.”

Lenzen, S. A., Daniëls, R., van Bokhoven, M. A., van der Weijden, T., & Beurskens, A. (2017). Disentangling self-management goal setting and action planning: A scoping review. PloS one , 12 (11), e0188822.

If an article was eligible for inclusion in this study, data related to the patient-centered care framework or model presented in the article was extracted by the lead author and reviewed by a second author (JCM). Data extracted from the reviewed patient-centered care frameworks and models was entered into data extraction records and synthesized in summary format. Data were systematically charted using the data charting form developed in Microsoft Excel. Information on authorship, article type, population, and patientcentered care approach were recorded on this form. A second data charting form was developed to chart data on the communication systematic reviews identified. Information on clinical context, patient-centered care focus, number of studies reviewed and key findings were recorded on this form.

Constand, M. K., MacDermid, J. C., Dal Bello-Haas, V., & Law, M. (2014). Scoping review of patient-centered care approaches in healthcare. BMC health services research , 14 , 1-9.

The final charting form, which clearly defines each item, should be included in the scoping review as an appendix or supplementary file, if possible.

Author: This information is essential for referencing and should be consistent throughout the scoping review document.
Year of Publication: Noting the publication year of each source helps analyze trends and changes in research over time. This variable can highlight areas where research has progressed or where further investigation is necessary.
Country: This variable involves noting the country of the study and the bibliographic details of each source. The country of origin provides context and helps assess the generalizability of findings to other settings.
Objective(s): The objectives of each included source of evidence should be clearly stated. This variable helps understand the aim of each study and how it contributes to the overall scoping review question.
Participants (characteristics/total number): This variable involves describing the defining characteristics of the participants in the included sources of evidence. Details like diagnostic criteria, age, ethnicity, and the total number of participants are crucial elements of this variable. This information provides context to the scoping review findings.
Concept: This variable pertains to extracting and mapping data related to the core concept being investigated in the scoping review. The specific data extracted will depend on the nature of the concept, which should be clearly defined in the scoping review.
Intervention Type: If applicable to the scoping review question, the type of intervention used in each source should be recorded. This might include details like the specific intervention method, the comparator used, and the duration of the intervention. This information helps compare and contrast different interventions explored in the included studies.
Methodology: Describing the methodology employed by each source is essential to understand how the research was conducted. This variable provides insights into the study design, data collection methods, and analysis techniques used. Categorizing study designs is essential to compare and contrast different research approaches and their potential implications for the scoping review’s conclusions.
Outcome Measures: This variable focuses on the tools or methods used to assess the effects of an intervention or phenomenon. It’s essential to describe the specific outcome measures used in each study, including details on how they were measured. This information helps compare findings across studies using similar outcome assessment tools.
Main Finding: This variable focuses on extracting the primary findings or results of each study that are relevant to the scoping review’s research question. These findings form the core evidence base and are crucial for addressing the scoping review objectives.

“We abstracted data on article characteristics (e.g., country of origin, funder), engagement characteristics and contextual factors (e.g., type of knowledge user, country income level, type of engagement activity, frequency and intensity of engagement, use of a framework to inform the intervention), barriers and facilitators to engagement, and results of any formal assessment of engagement (e.g., attitudes, beliefs, knowledge, benefits, unintended consequences).”

Tricco, A. C., Zarin, W., Rios, P., Nincic, V., Khan, P. A., Ghassemi, M., … & Langlois, E. V. (2018). Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review. Implementation Science , 13 , 1-19.

8. Analyzing the evidence

The key element of a scoping review is the synthesis: that is the process that brings together the findings from the set of included studies in order to draw conclusions based on the body of evidence.

Data synthesis in a scoping review involves collating, combining, and summarizing findings from the included studies.

This process aims to provide a reliable and comprehensive answer to the review question by considering the strength of the evidence, examining the consistency of observed effects, and investigating any inconsistencies.

The data synthesis will be presented in the results section of the scoping review.

Develop a clear text narrative that explains the key findings
Use a logical heading structure to guide readers through your results synthesis
Use tables to summarise findings (can be same table as data extraction)

Scoping reviews often use a more descriptive approach to synthesis, summarizing the types of evidence available, key findings, and research gaps.

Research design (e.g., experimental, observational, qualitative)
Population characteristics
Intervention types
Outcome measures
Theoretical frameworks
Geographic regions
Time periods
The predominant study designs used in the field
The range of methodologies employed
The diversity (or lack thereof) in research approaches
Primary outcomes
Major conclusions drawn by the authors
Any notable or unexpected findings
Recurring themes in the literature
Evolving research focuses over time
Commonly used methodologies or theoretical frameworks
Consistency (or inconsistency) in findings across different studies
Identifying areas that have been extensively studied
Noting topics that have received less attention
Highlighting any shifts in research focus over time
Populations that have been understudied
Methodologies that haven’t been widely applied
Questions that remain unanswered or inadequately addressed
Contradictions in the literature that need further investigation
Summarizing key concepts: Identify and describe the central ideas, theories, or constructs that emerge from the literature. This helps to provide a conceptual overview of the field.
Tables summarizing study characteristics
Charts showing the distribution of studies across categories
Concept maps illustrating relationships between key ideas

Remember, the goal in a scoping review is not to critically appraise the quality of individual studies or to provide a definitive answer to a narrow research question.

Instead, the synthesis aims to provide a broad overview of the field, mapping out the existing literature and identifying areas for further research.

This descriptive approach allows for a comprehensive understanding of the landscape of a particular research area.

“We grouped the studies by the types of behavior they analyzed, and summarized the type of settings, populations and study designs for each group, along with the measures used and broad findings. Where we identified a systematic review, we counted the number of studies included in the review that potentially met our inclusion criteria and noted how many studies had been missed by our search.”

Hutchinson, J., Prady, S. L., Smith, M. A., White, P. C., & Graham, H. M. (2015). A scoping review of observational studies examining relationships between environmental behaviors and health behaviors. International journal of environmental research and public health , 12 (5), 4833-4858.

9. Presenting the results

The findings should be presented in a clear and logical way that answers the research question(s). This section might include tables, figures, or narrative summaries to illustrate the data.

Narrative Summaries

Write a clear, concise narrative that brings together all of these elements. This should provide readers with a comprehensive overview of the current state of knowledge in the field, highlighting both what is known and what remains to be explored.

The primary goal of a narrative summary is to weave together the information extracted from multiple sources into a cohesive and understandable narrative. This story should focus on why a specific action is necessary, should be discontinued, or lacks sufficient evidence to determine its efficacy

A well-crafted narrative summary often utilizes headings and subheadings to organize the synthesized information logically.

This approach makes it easier for readers to follow the thought process and understand the relationships between different pieces of evidence.

Strategies on how to be sensitive to patient needs were primarily discussed in the qualitative research articles included in this review. Such strategies included acknowledging and adapting to unique patient identifiers [19,24,25]. For example, clinicians are urged to observe and reflect on fluctuating levels of patient alertness, patient comfort levels in the presence or absence of family members, and different communication barriers such as hearing loss, in order to facilitate clinical interactions [15,19,22]. Of the articles reviewed, 58% identified that careful observation of unique patient characteristics is necessary to providing care that will lead to optimal patient receptiveness and positive health outcomes.

While narrative summaries primarily use text, incorporating tables, charts, or diagrams can enhance clarity, particularly when presenting complex data patterns.

However, always accompany these visual aids with a clear textual explanation to ensure comprehensive understanding.

PRISMA Flowchart

Using a PRISMA flowchart in a scoping review is considered good practice. It promotes transparency and allows for a clear understanding of how sources were selected.

The flowchart illustrates the step-by-step process of screening, filtering, and selecting studies based on predefined inclusion and exclusion criteria.

The flowchart visually depicts the following stages:

Identification: The initial number of titles and abstracts identified through database searches.
Screening: The screening process, based on titles and abstracts.
Eligibility: Full-text copies of the remaining records are retrieved and assessed for eligibility.
Inclusion: Applying the predefined inclusion criteria resulted in the inclusion of publications that met all the criteria for the review.
Exclusion: The flowchart details the reasons for excluding the remaining records.

Petersen, B., Koshy-Chenthittayil, S., DeArmond, M., & Caromile, L. A. (2023). Assessment of diversity-based approaches used by American Universities to increase recruitment and retention of biomedical sciences research faculty members: A scoping review protocol. Plos one , 18 (6), e0276089.

10. Discussion Section And Conclusion

Summarizing the evidence in relation to the purpose of the review, making conclusions and noting any implications of the findings.

It is also essential to remember that scoping reviews, unlike systematic reviews, do not aim to provide concrete recommendations for practice or policy.

Their primary function is to map the existing evidence, identify knowledge gaps, and clarify concepts, rather than synthesize results for direct application in clinical or policy settings

Summarizing the Evidence

Summarize key findings in relation to your research questions
Highlight main themes or patterns across studies
Explain the nuances and complexities in the evidence
Tailor overall findings of the scoping review to the relevant knowledge users such as policymakers, health care providers and patients or consumers
Discuss the consistency of the evidence
This provides a clear takeaway message for readers

“In this scoping review we identified 88 primary studies addressing dissemination and implementation research across various settings of dementia care published between 1998 and 2015. Our findings indicate a paucity of research focusing specifically on dissemination of knowledge within dementia care and a limited number of studies on implementation in this area. We also found that training and educating professionals, developing stakeholder interrelationships, and using evaluative and iterative strategies are frequently employed to introduce and promote change in practice. However, although important and feasible, these strategies only partly address what is repeatedly highlighted in the evidence base: that organisational factors are reported as the main barrier to implementation of knowledge within dementia care. Moreover, included studies clearly support an increased effort to improve the quality of dementia care provided in residential settings in the last decade.”

Lourida, I., Abbott, R. A., Rogers, M., Lang, I. A., Stein, K., Kent, B., & Thompson Coon, J. (2017). Dissemination and implementation research in dementia care: a systematic scoping review and evidence map. BMC geriatrics , 17 , 1-12.

Limitations

When considering the limitations of a review process, particularly scoping reviews, it’s essential to acknowledge that the goal is breadth, not depth, of information.

This means that unlike systematic reviews, scoping reviews generally don’t involve a formal appraisal of the methodological quality of included studies, unless specifically required by the review’s aim.

One significant limitation frequently encountered in reviews is the restriction to English-language sources. This decision, often made for feasibility, can inadvertently introduce bias by excluding valuable research from non-English speaking communities and potentially limiting the generalizability of the findings.
For instance, if a scoping review protocol initially excludes gray literature but later incorporates it due to the emergence of relevant findings during the review process, this change needs to be explicitly stated and justified in the final report.

“Our scoping review has some limitations. To make our review more feasible, we were only able to include a random sample of rapid reviews from websites of rapid review producers. Further adding to this issue is that many rapid reviews contain proprietary information and are not publicly available. As such, our results are only likely generalizable to rapid reviews that are publicly available. Furthermore, this scoping review was an enormous undertaking and our results are only up to date as of May 2013.”

Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., … & Straus, S. E. (2015). A scoping review of rapid review methods. BMC medicine , 13 , 1-15.

Conclusions

Discuss implications:.

Note that recommendations for practice and policy will not be relevant for most scoping reviews as the goal is to provide a preliminary map of the evidence without appraising the quality and validity of the results.
Consider both positive and negative implications.
This helps translate your findings into real-world applications.

Identify gaps and future research:

Point out areas where evidence is lacking or inconsistent.
Suggest specific research questions or study designs to address these gaps.
Recommendations for future research are often a key element, particularly suggestions for more focused systematic reviews based on the scoping review’s findings.
For instance, a scoping review might reveal a need for research linking specific features of expertise to mental and physical health outcomes. Similarly, there might be methodological gaps regarding the validation of certain measures or understanding experiences across diverse contexts and populations.

“The lack of evidence to support physiotherapy interventions for this population appears to pose a challenge to physiotherapists. The aim of this scoping review was to identify gaps in the literature which may guide a future systematic review. However, the lack of evidence found means that undertaking a systematic review is not appropriate or necessary […]. This advocates high quality research being needed to determine what physiotherapy techniques may be of benefit for this population and to help guide physiotherapists as how to deliver this.”

Hall, A. J., Lang, I. A., Endacott, R., Hall, A., & Goodwin, V. A. (2017). Physiotherapy interventions for people with dementia and a hip fracture—a scoping review of the literature. Physiotherapy , 103 (4), 361-368.

Potential Challenges

Balancing breadth and depth: Scoping reviews necessitate a careful balance between covering a wide range of literature (breadth) and providing sufficient depth of analysis. A scope that is too broad can become unmanageable and result in superficial treatment of the topic. Conversely, excessive focus on depth might compromise the comprehensiveness of the review. This balance requires careful consideration during the planning stages, particularly when defining the review question and inclusion criteria.
Lack of standardized terminology and methods: While frameworks for scoping reviews exist, there is still a lack of consensus on terminology and methods, potentially leading to inconsistencies in how they are conducted and reported. This variability can make it challenging to assess the quality and reliability of scoping review findings.
Difficulty in analyzing and presenting findings: Scoping reviews often involve synthesizing information from a large and diverse body of literature. Analyzing and presenting this information in a meaningful and concise way can be demanding, requiring a high level of analytical skill and clarity of presentation. The absence of standardized analysis methods further exacerbates this challenge, leading to potential inconsistencies in how data is extracted, analyzed, and presented.
Limited resources and time constraints: Scoping reviews, although sometimes perceived as a quicker alternative to systematic reviews, can still be resource-intensive. They require meticulous planning, comprehensive searching, and rigorous analysis.

Reading List

Arksey, H., & O’Malley, L. (2005). Scoping studies: towards a methodological framework . International journal of social research methodology , 8 (1), 19-32.
Levac, D., Colquhoun, H., & O’brien, K. K. (2010). Scoping studies: advancing the methodology . Implementation science , 5 , 1-9.
Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology , 18 , 1-7.
Pearson, A., Wiechula, R., & Lockwood, C. (2005). The JBI model of evidence-based healthcare. JBI Evidence Implementation , 3 (8), 207-215.
Peters, M. D., Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C., & Khalil, H. (2020). Scoping reviews . JBI manual for evidence synthesis , 10 .
Peters, M., Godfrey, C., McInerney, P., Soares, C. B., Khalil, H., & Parker, D. (2015). Methodology for JBI scoping reviews. In The Joanna Briggs institute reviewers manual 2015 (pp. 3-24). Joanna Briggs Institute.
Peters, M., Godfrey, C., Khalil, H., McInerney, P., Soares, C., & Parker, D. (2017). 2017 guidance for the conduct of JBI scoping reviews . Joana Briggs Inst Rev Man , 13 , 141-6.
Pollock, D., Davies, E. L., Peters, M. D., Tricco, A. C., Alexander, L., McInerney, P., … & Munn, Z. (2021). Undertaking a scoping review: A practical guide for nursing and midwifery students, clinicians, researchers, and academics . Journal of advanced nursing , 77 (4), 2102-2113.
Pollock, D., Peters, M. D., Khalil, H., McInerney, P., Alexander, L., Tricco, A. C., … & Munn, Z. (2023). Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI evidence synthesis , 21 (3), 520-532.
Scott, H., Sweet, L., Strauch, L., & Muller, A. (2019). Expressed breastmilk handling and storage guidelines available to mothers in the community: A scoping review. Women and Birth, 33 (5), 426–432.
Tricco, AC, Lillie, E, Zarin, W, O’Brien, KK, Colquhoun, H, Levac, D, Moher, D, Peters, MD, Horsley, T, Weeks, L, Hempel, S et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018,169(7):467-473.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

What Is Peer Review? | Types & Examples

What Is Peer Review? | Types & Examples

Published on December 17, 2021 by Tegan George . Revised on June 22, 2023.

Peer review, sometimes referred to as refereeing , is the process of evaluating submissions to an academic journal. Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.

Peer-reviewed articles are considered a highly credible source due to the stringent process they go through before publication.

There are various types of peer review. The main difference between them is to what extent the authors, reviewers, and editors know each other’s identities. The most common types are:

Single-blind review
Double-blind review
Triple-blind review

Collaborative review

Open review.

Relatedly, peer assessment is a process where your peers provide you with feedback on something you’ve written, based on a set of criteria or benchmarks from an instructor. They then give constructive feedback, compliments, or guidance to help you improve your draft.

What is the purpose of peer review, types of peer review, the peer review process, providing feedback to your peers, peer review example, advantages of peer review, criticisms of peer review, other interesting articles, frequently asked questions about peer reviews.

Many academic fields use peer review, largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the manuscript. For this reason, academic journals are among the most credible sources you can refer to.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure.

Peer assessment is often used in the classroom as a pedagogical tool. Both receiving feedback and providing it are thought to enhance the learning process, helping students think critically and collaboratively.

Prevent plagiarism. Run a free check.

Depending on the journal, there are several types of peer review.

Single-blind peer review

The most common type of peer review is single-blind (or single anonymized) review . Here, the names of the reviewers are not known by the author.

While this gives the reviewers the ability to give feedback without the possibility of interference from the author, there has been substantial criticism of this method in the last few years. Many argue that single-blind reviewing can lead to poaching or intellectual theft or that anonymized comments cause reviewers to be too harsh.

Double-blind peer review

In double-blind (or double anonymized) review , both the author and the reviewers are anonymous.

Arguments for double-blind review highlight that this mitigates any risk of prejudice on the side of the reviewer, while protecting the nature of the process. In theory, it also leads to manuscripts being published on merit rather than on the reputation of the author.

Triple-blind peer review

While triple-blind (or triple anonymized) review —where the identities of the author, reviewers, and editors are all anonymized—does exist, it is difficult to carry out in practice.

Proponents of adopting triple-blind review for journal submissions argue that it minimizes potential conflicts of interest and biases. However, ensuring anonymity is logistically challenging, and current editing software is not always able to fully anonymize everyone involved in the process.

In collaborative review , authors and reviewers interact with each other directly throughout the process. However, the identity of the reviewer is not known to the author. This gives all parties the opportunity to resolve any inconsistencies or contradictions in real time, and provides them a rich forum for discussion. It can mitigate the need for multiple rounds of editing and minimize back-and-forth.

Collaborative review can be time- and resource-intensive for the journal, however. For these collaborations to occur, there has to be a set system in place, often a technological platform, with staff monitoring and fixing any bugs or glitches.

Lastly, in open review , all parties know each other’s identities throughout the process. Often, open review can also include feedback from a larger audience, such as an online forum, or reviewer feedback included as part of the final published product.

While many argue that greater transparency prevents plagiarism or unnecessary harshness, there is also concern about the quality of future scholarship if reviewers feel they have to censor their comments.

In general, the peer review process includes the following steps:

First, the author submits the manuscript to the editor.
Reject the manuscript and send it back to the author, or
Send it onward to the selected peer reviewer(s)
Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made.
Lastly, the edited manuscript is sent back to the author. They input the edits and resubmit it to the editor for publication.

In an effort to be transparent, many journals are now disclosing who reviewed each article in the published product. There are also increasing opportunities for collaboration and feedback, with some journals allowing open communication between reviewers and authors.

It can seem daunting at first to conduct a peer review or peer assessment. If you’re not sure where to start, there are several best practices you can use.

Summarize the argument in your own words

Summarizing the main argument helps the author see how their argument is interpreted by readers, and gives you a jumping-off point for providing feedback. If you’re having trouble doing this, it’s a sign that the argument needs to be clearer, more concise, or worded differently.

If the author sees that you’ve interpreted their argument differently than they intended, they have an opportunity to address any misunderstandings when they get the manuscript back.

Separate your feedback into major and minor issues

It can be challenging to keep feedback organized. One strategy is to start out with any major issues and then flow into the more minor points. It’s often helpful to keep your feedback in a numbered list, so the author has concrete points to refer back to.

Major issues typically consist of any problems with the style, flow, or key points of the manuscript. Minor issues include spelling errors, citation errors, or other smaller, easy-to-apply feedback.

Tip: Try not to focus too much on the minor issues. If the manuscript has a lot of typos, consider making a note that the author should address spelling and grammar issues, rather than going through and fixing each one.

The best feedback you can provide is anything that helps them strengthen their argument or resolve major stylistic issues.

Give the type of feedback that you would like to receive

No one likes being criticized, and it can be difficult to give honest feedback without sounding overly harsh or critical. One strategy you can use here is the “compliment sandwich,” where you “sandwich” your constructive criticism between two compliments.

Be sure you are giving concrete, actionable feedback that will help the author submit a successful final draft. While you shouldn’t tell them exactly what they should do, your feedback should help them resolve any issues they may have overlooked.

As a rule of thumb, your feedback should be:

Easy to understand
Constructive

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Below is a brief annotated research example. You can view examples of peer feedback by hovering over the highlighted sections.

Influence of phone use on sleep

Studies show that teens from the US are getting less sleep than they were a decade ago (Johnson, 2019) . On average, teens only slept for 6 hours a night in 2021, compared to 8 hours a night in 2011. Johnson mentions several potential causes, such as increased anxiety, changed diets, and increased phone use.

The current study focuses on the effect phone use before bedtime has on the number of hours of sleep teens are getting.

For this study, a sample of 300 teens was recruited using social media, such as Facebook, Instagram, and Snapchat. The first week, all teens were allowed to use their phone the way they normally would, in order to obtain a baseline.

The sample was then divided into 3 groups:

Group 1 was not allowed to use their phone before bedtime.
Group 2 used their phone for 1 hour before bedtime.
Group 3 used their phone for 3 hours before bedtime.

All participants were asked to go to sleep around 10 p.m. to control for variation in bedtime . In the morning, their Fitbit showed the number of hours they’d slept. They kept track of these numbers themselves for 1 week.

Two independent t tests were used in order to compare Group 1 and Group 2, and Group 1 and Group 3. The first t test showed no significant difference ( p > .05) between the number of hours for Group 1 ( M = 7.8, SD = 0.6) and Group 2 ( M = 7.0, SD = 0.8). The second t test showed a significant difference ( p < .01) between the average difference for Group 1 ( M = 7.8, SD = 0.6) and Group 3 ( M = 6.1, SD = 1.5).

This shows that teens sleep fewer hours a night if they use their phone for over an hour before bedtime, compared to teens who use their phone for 0 to 1 hours.

Peer review is an established and hallowed process in academia, dating back hundreds of years. It provides various fields of study with metrics, expectations, and guidance to ensure published work is consistent with predetermined standards.

Protects the quality of published research

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. Any content that raises red flags for reviewers can be closely examined in the review stage, preventing plagiarized or duplicated research from being published.

Gives you access to feedback from experts in your field

Peer review represents an excellent opportunity to get feedback from renowned experts in your field and to improve your writing through their feedback and guidance. Experts with knowledge about your subject matter can give you feedback on both style and content, and they may also suggest avenues for further research that you hadn’t yet considered.

Helps you identify any weaknesses in your argument

Peer review acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process. This way, you’ll end up with a more robust, more cohesive article.

While peer review is a widely accepted metric for credibility, it’s not without its drawbacks.

Reviewer bias

The more transparent double-blind system is not yet very common, which can lead to bias in reviewing. A common criticism is that an excellent paper by a new researcher may be declined, while an objectively lower-quality submission by an established researcher would be accepted.

Delays in publication

The thoroughness of the peer review process can lead to significant delays in publishing time. Research that was current at the time of submission may not be as current by the time it’s published. There is also high risk of publication bias , where journals are more likely to publish studies with positive findings than studies with negative findings.

Risk of human error

By its very nature, peer review carries a risk of human error. In particular, falsification often cannot be detected, given that reviewers would have to replicate entire experiments to ensure the validity of results.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Normal distribution
Measures of central tendency
Chi square tests
Confidence interval
Quartiles & Quantiles
Cluster sampling
Stratified sampling
Thematic analysis
Discourse analysis
Cohort study
Ethnography

Research bias

Implicit bias
Cognitive bias
Conformity bias
Hawthorne effect
Availability heuristic
Attrition bias
Social desirability bias

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

In general, the peer review process follows the following steps:

Reject the manuscript and send it back to author, or
Send it onward to the selected peer reviewer(s)
Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made.
Lastly, the edited manuscript is sent back to the author. They input the edits, and resubmit it to the editor for publication.

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. It also represents an excellent opportunity to get feedback from renowned experts in your field. It acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process.

Peer-reviewed articles are considered a highly credible source due to this stringent process they go through before publication.

Many academic fields use peer review , largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the published manuscript.

A credible source should pass the CRAAP test and follow these guidelines:

The information should be up to date and current.
The author and publication should be a trusted authority on the subject you are researching.
The sources the author cited should be easy to find, clear, and unbiased.
For a web source, the URL and layout should signify that it is trustworthy.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). What Is Peer Review? | Types & Examples. Scribbr. Retrieved July 30, 2024, from https://www.scribbr.com/methodology/peer-review/

Is this article helpful?

Tegan George

Other students also liked, what are credible sources & how to spot them | examples, ethical considerations in research | types & examples, applying the craap test & evaluating sources, what is your plagiarism score.

Harvard Library
Research Guides
Faculty of Arts & Sciences Libraries

Finding and Reading Journal Articles

Journal Articles: Why You Use Them

Why are articles so important to research?

Subject Databases: Organizing Research Conversations
Databases We Recommend For You
Sources in the Disciplines
Reading in the Disciplines

Journal articles are the academic's stock in trade, t he basic means of communicating research findings to an audience of one’s peers. That holds true across the disciplinary spectrum, so no matter where you land as a concentrator, you can expect to rely on them heavily.

Regardless of the discipline, moreover, journal articles perform an important knowledge-updating function .

image of 4 journals repesenting the life and physical science, the social sciences (examples from education and sociology) and the humanities (example from literary studies)

Textbooks and handbooks and manuals will have a secondary function for chemists and physicists and biologists, of course. But in the sciences, articles are the standard and preferred publication form.

In the social sciences and humanities , where knowledge develops a little less rapidly or is driven less by issues of time-sensitivity , journal articles and books are more often used together.

Not all important and influential ideas warrant book-length studies, and some inquiry is just better suited to the size and scope and concentrated discussion that the article format offers.

Journal articles sometimes just present the most appropriate solution for communicating findings or making a convincing argument. A 20-page article may perfectly fit a researcher's needs. Sustaining that argument for 200 pages might be unnecessary -- or impossible.

The quality of a research article and the legitimacy of its findings are verified by other scholars, prior to publication, through a rigorous evaluation method called peer-review . This seal of approval by other scholars doesn't mean that an article is the best, or truest, or last word on a topic. If that were the case, research on lots of things would cease. Peer review simply means other experts believe the methods, the evidence, the conclusions of an article have met important standards of legitimacy, reliability, and intellectual honesty.

Searching the journal literature is part of being a responsible researcher at any level: professor, grad student, concentrator, first-year. Knowing why academic articles matter will help you make good decisions about what you find -- and what you choose to rely on in your work.

Think of journal articles as the way you tap into the ongoing scholarly conversation , as a way of testing the currency of a finding, analysis, or argumentative position, and a way of bolstering the authority (or plausibility) of explanations you'll offer in the papers and projects you'll complete at Harvard.

Next: Subject Databases: Organizing Research Conversations >>

Except where otherwise noted, this work is subject to a Creative Commons Attribution 4.0 International License , which allows anyone to share and adapt our material as long as proper attribution is given. For details and exceptions, see the Harvard Library Copyright Policy ©2021 Presidents and Fellows of Harvard College.

Deep learning based network intrusion detection system: a systematic literature review and future scopes

Regular Contribution
Published: 02 August 2024

Cite this article

Yogesh ORCID: orcid.org/0000-0003-4881-1622 1 &
Lalit Mohan Goyal ORCID: orcid.org/0000-0003-4618-0281 1

15 Accesses

Explore all metrics

With the immense growth of the internet, sensitive, confidential, important corporate and individual data passing through the internet has grown rapidly. Due to the limitation of security systems, potential hackers and attackers have possessed vulnerabilities and attacks for intruding into the network to gain confidential and sensitive information to affect the performance of networks by breaching network confidentiality. Thereby, to counterfeit these attacks and abnormal behaviors, a network intrusion detection system (NIDS), acts as a crucial branch of cybersecurity for analysis and monitoring the network traffic regularly to report and detect abnormal and malicious activities in a network. Currently, various reviews and survey papers have covered various techniques for NIDS, out of which, mostly followed a non-systematic way of approach without an in-depth analysis of techniques and evaluation metrics used by deep learning(DL) based NIDS models. In addition, various reviews focused on machine learning (ML) and DL-based methodology, but with less emphasis on DL techniques (i.e. AE, CNN, DNN, DBN, RNN, and Hybrid DL) based classification. Thereby, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology was used to accomplish this work by providing a comprehensive and detailed overview of DL-based NIDS. Research papers for this work were collected from five well-known databases (ScienceDirect, IEEE, Hindawi, SpringerNature, and MDPI) which were cut among several reputable conference proceedings and reputable journals. Across the 750 articles identified in the literature, 72 research papers were finally marked and selected for synthesis and analysis to find the answers to research questions. In addition, we identified various potential research challenges in the current domain based on research findings. Lastly, to design an efficient NIDS, we concluded our study by identifying high-impact and promising future research areas in the NIDS domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

Get 10 units per month
Download Article/Chapter or eBook
1 Unit = 1 Article or 1 Chapter
Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data availability

No datasets were generated or analysed during the current study.

Abbasi, J.S., Bashir, F., Qureshi, K.N., Islam, M.N., Jeon, G.: Deep learning-based feature extraction and optimizing pattern matching for intrusion detection using finite state machine. Comput. Electr. Eng. 92 , 1–12 (2021). https://doi.org/10.1016/j.compeleceng.2021.107094

Article Google Scholar

Abdulganiyu, O.H., Tchakoucht, T.A., Saheed, Y.K.: A systematic literature review for network intrusion detection system (IDS). Int. J. Inf. Secur. 1–38 (2023). https://doi.org/10.1007/s10207-023-00682-2

Ahmad, Z., Khan, A.S., Shiang, C.W., Abdullah, J., Ahmad, F.: Network Intrusion Detection System: A systematic study of machine learning and deep learning approaches. Trans. Emerg. Telecommunications Technol. 1–29 (2020). https://doi.org/10.1002/ett.4150

Aldarwbi, M.Y., Lashkari, A.H., Ghorbani, A.A.: The sound of intrusion: A novel intrusion detection system. Comput. Electr. Eng. 104 , 1–12 (2022). https://doi.org/10.1016/j.compeleceng.2022.108455

Al-Hawawreh, M., Sitnikova, E., Aboutorabb, N.: X-IIoTID: A connectivity-agnostic and device-agnostic Intrusion Data Set for Industrial Internet of things. IEEE Internet Things J., 3962–3977. (2022)

Almomani, I., Kasabeh, B.A., Al-Akhras, M.: WSN-DS: A dataset for intrusion detection systems in Wireless Sensor Networks. J. Sens. 1–17 (2016). https://doi.org/10.1155/2016/4731953

Almutairi, L., Daniel, R., Khasimbee, S., Lydia, E.L., Acharya, S., Kim, H.-I.: Quantam Dwarf Mongoose Optimization with Ensemble Deep Learning based intrusion detection in Cyber-physical systems. IEEE Access. 11 (3), 66828–66837 (2023). https://doi.org/10.1109/ACCESS/2023.3287896

Alotaibi, S.D., Yadav, K., Aledaily, A.N., Alkwai, L.M., Dafhalla, A.K.Y., Almansour, S., Lingamuthu, V.: Deep neural network-based intrusion detection system through PCA. Math. Probl. Eng. 1–9 (2022). https://doi.org/10.1155/2022/6488571

Al-Qatf, M., Lasheng, Y., Al-Habib, M., Al-Sabahi, K.: Deep Learning Approach combining sparse autoencoder with SVM for Network Intrusion Detection. IEEE Access. 6 , 52843–52856 (2018)

Alsaedi, A., Moustafa, N., Tari, Z., Mahmood, A., Anwar, A.: TON_IoT Telemetry Dataset: A New Generation Dataset of IoT and IIoT for Data-Driven Intrusion Detect. Syst. IEEE Access., 165130–165150. (2020)

Altunay, H.C., Albayrak, Z.: A hybrid CNN + LSTM -based intrusion detection system for industrial IoT networks. Eng. Sci. Technol. 38 , 1–13 (2023). https://doi.org/10.1016/j.jestch2022.101322

Alyousuf, M.Y., Abdelmajeeb, N.T.: Dynamically Detecting Security Threats and Updating a Signature-Based Intrusion Detection System’s Database. In 23rd International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, Procedia Computer Science, 159, 1507–1516. (2019)

Ashiku, L., Dagli, C.: Network Intrusion Detection System using Deep Learning. In Complex Adaptive System Conference Theme: Big Data, IoT, and AI for a Smarter Future, 239–247. (Elsevier) (2021)

Bakhsh, S.A., Khan, M.A., Ahmed, F., Alshehri, M.S., Ali, H., Ahmad, J.: Enhancing IoT network security through deep learning-powered intrusion detection system. Internet Things. 24 , 1–36 (2023). https://doi.org/10.1016/j.iot.2023.100936

Balamurgan, E., Mehbodniya, A., Kariri, E., Yadav, K., Kumar, A., Haq, M.A.: Network optimization using defender system in cloud computing security-based intrusion detection system with game theory deep neural network (IDSGT-DNN). Pattern Recognit. Lett. 56 , 142–151 (2022). https://doi.org/10.1016/j.pattrec.2022.02.013

Basati, A., Faghih, M.M.: APAE: An IoT intrusion detection system using asymmetric parallel auto-encoder. Neural Comput. Appl. 35 , 4813–4833 (2021). https://doi.org/10.1007/s00521-021-06011-9

Bridges, R.A., Tarrah, G.-V.R., Iannacone, M.D., Vincent, M.S., Chen, Q.: ACM Comput. Surveys. 52 (6), 1–35 (2019). https://doi.org/10.1145/3344382 A Survey of Intrusion Detection Systems Leveraging Host Data

Cao, B., Li, C., Song, Y., Fan, X.: Network Intrusion Detection Technology Based on Convolutional Neural Network and BiGRU. Comput. Intell. Neurosci. 1–20 (2022). https://doi.org/10.1155/2022/1942847

Chen, Y., Lin, Q., Wei, W., Ji, J., Wong, Coello, K.-C., C.A.C: Intrusion detection using multi-objective evolutionary convolutional neural network for internet of things in Fog Computing. Knowl. Based Syst. 244 , 1–14 (2022). https://doi.org/10.1016/j.knosys.2022.108505

CIC-Darknet: (2020). https://www.unb.ca/cic/datasets/darknet2020.html , last accessed 10th April 2024

CIC-DDoS: (2019). https://www.unb.ca/cic/datasets/ddos-2019.html , last accessed 12th April 2024

CIC-IDS: (2017). https://www.unb.ca/cic/datasets/ids-2017.html , last accessed 12th April 2024

CIC-IoT22: (2024). https://www.unb.ca/cic/datasets/iotdataset-2022.html , last accessed 15th

CSE-CIC-IDS: https://www.unb.ca/cic/datasets/ids-2018.html , (2018). last accessed 18th April 2024.

CSIC-: (2010). https://www/isi.csic.es/dataset , last accessed 30th March 2024

Cui, J., Sun, H., Zhong, H., Zhang, J., Wei, L., Bolodurina, I., He, D.: Collaborative intrusion detection system for SDVN: A Fairness Federated Deep Learning Approach. IEEE Trans. Parallel Distrib. Syst. 34 (9), 2512–2528 (2023)

Dao, T.-N., Lee, H.: Stacked autoencoder-based Probabilistic feature extraction for On-Device Network Intrusion Detection. IEEE Internet Things J. 9 (16), 14438–14451 (2021)

Deng, H., Yang, H.: Network Intrusion Detection based on sparse autoencoder and IGA-BP network. Wirel. Communication Mob. Comput. 1–11 (2021). https://doi.org/10.1155/2021/9510858

Deng, X., Zhu, J., Pei, X., Zhang, L., Ling, Z.: Flow Topology-based graph Convolutional Network for Intrusion Detection in label-limited IoT networks. IEEE Trans. Netw. Serv. Manage. 20 (1), 684–696 (2023)

Dixit, P., Silakari, S.: Deep learning algorithms for Cybersecurity Applications: A Technological and Status Review. Comput. Sci. Rev. 39 , 1–15 (2021). https://doi.org/10.1016/j.cosrev.2020.100317

Article MathSciNet Google Scholar

Dong, B., Wang, X.: Comparison Deep Learning Method to Traditional Methods Using for Network Intrusion Detection. In 2016 8th IEEE International Conference on Communication Software and Networks, 581–585. (IEEE) (2016)

DS2OS: (2024). https://www/kaggle.com/datasets/francoisxa/ds2ostraffictraces , last accessed 30th

Elaveini, M.A., Simon, J., Kapileswar, N., Polasi, P.K.: Hybrid intrusion detection system for wireless IoT networks using deep learning algorithm. Comput. Electr. Eng. 102 , 1–11 (2022). https://doi.org/10.1016/j.compeleceng.2022.108190

Elmasry, W., Akbulut, A., Zaim, A.H.: Evolving deep learning architectures for network intrusion detection using a double PSO metaheuristic. Comput. Netw. 168 , 1–21 (2020). https://doi.org/10.1016/j.comnet.2019.107042

Elsayed, M.S., Le-Khac, N.A., Jurcut, A.D.: InSDN: A Novel SDN Intrusion Dataset, pp. 165263–165284. IEEE Access (2020)

Ferrag, M.A., Maglaras, L., Moschoyiannis, S., Janicke, H.: Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. J. Inform. Secur. Appl. 50 , 1–19 (2020). https://doi.org/10.1016/j.jisa.2019.102419

Fu, Z.: Computer Network Intrusion Anomaly Detection with recurrent neural network. Mob. Inform. Syst. 1–11 (2022). https://doi.org/10.1155/2022/6576023

Gamage, S., Samarabandu, J.: Deep learning methods in network intrusion detection: A survey and an objective comparison. J. Netw. Comput. Appl. 169 , 1–21 (2020). https://doi.org/10.1016/j.jnca.2020.102767

Ghurab, M., Gaphari, G., Alshami, F., Alshamy, R., Othman, S.: A detailed analysis of Benchmark datasets for Network Intrusion Detection System. Asian J. Res. Comput. Sci. 7 (4), 14–33 (2021)

Halbouni, A., Gunawan, T.S., Habaebi, M.H., Halbouni, M., Kartiwi, M., Ahmad, R.: CNN-LSTM: Hybrid Deep learning neural network for Network Intrusion Detection System. IEEE Access. 10 , 99837–99849 (2022)

Hamada, R.A., Elsayed, R.A., Abdalla, M.I., Elsaid, A.S.: Securing IoT and SDN systems using deep-learning based automatic intrusion detection. Ain Shams Eng. J. 14 , 1–13 (2023). https://doi.org/10.1016/j.asej/2023/102211

Hamid, Y., Balasaraswathi, V.R., Journaux, L., Sugumaran, M.: Benchmark Datasets for Network Intrusion Detection: A review. Int. J. Netw. Secur. 20 (4), 645–654 (2018)

Google Scholar

Hindy, H., Brosset, D., Bayne, E., Seeam, A., Tachtatzis, C., Atkinson, R., Bellekens, X.: A taxonomy of Network threats and the Effect of current datasets on Intrusion Detection systems. IEEE Access. 8 , 104650–104675 (2020)

Hnamte, V., Hussain, J.: Dependable intrusion detection system using deep convolutional neural network: A novel framework and performance evaluation approach. Telematics Inf. Rep. 11 , 1–13 (2023). https://doi.org/10.1016/j.teler.2023.100077

Hu, B., Bi, Y., Zhi, M., Zhang, K., Yan, F., Zhang, Q., Liu, Z.: A Deep one-class intrusion detection Scheme in Software-defined Industrial Networks. IEEE Trans. Industr. Inf. 18 (6), 4286–4296 (2022)

IoT23: (2024). https://www.unb.ca/cic/datasets/iotdataset-2023.html , last accessed 05th

ISCX-IDS https://www.unb.ca/cic/datasets/ids/html , (2012). last accessed 25th March 2024.

ISCXTor https://www.unb.ca/cic/datasets/tor.html , (2016). last accessed 02nd April 2024.

Jahangir, A.H., Soltani, M., Ousat, B., Siavoshani, M.J.: An adaptable deep learning-based intrusion detection system to zero-day attacks. J. Inform. Secur. Appl. 76 , 1–13 (2023). https://doi.org/10.1016/j.jisa.2023.103516

Kasongo, S.M.: A deep learning technique for intrusion detection system using a recurrent neural networks based framework. Comput. Commun. 199 , 113–125 (2023). https://doi.org/10.1016/j.comcom.2022.12.010

Khan, F.A., Gumaei, A., Derhab, A., Hussain, A.: A Novel two-stage deep learning model for efficient network intrusion detection. IEEE Access. 7 , 30373–30385 (2019)

Khraisat, A., Gondal, I., Vamplew, P., Joarder, K.: Survey of intrusion detection systems: Techniques, datasets, and challenges. Cybersecurity. 20 (2), 1–22 (2019). https://doi.org/10.1186/s42400-019-0038-7

Kitchenham, B., Brereton, P.: A systematic review of systematic review process research in software engineering. ‎Inf. Softw. Technol. 55 , 2049–2075 (2013)

Kitchenham, B.A., Stuart, C.: Guidelines for performing systematic literature reviews in software engineering. In: EBSE Technical Report, Keele University and Durham University Joint 94, Report, Report EBSE 2007-001. (2007). https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf

Lansky, J., Ali, S., Mohammadi, M., Majeed, M.K., Karim, S.H.T., Rashidi, S., Hosseinzadeh, M., Rahmani, M.: Deep learning-based intrusion detection systems: A systematic review. IEEE Access. 9 , 101574–101599 (2021)

Li, J., Tong, X., Liu, J., Cheng, L.: An efficient Federated Learning System for Network Intrusion Detection. IEEE Syst. J. 17 (2), 2455–2464 (2023)

Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzche, P., Loannidis, J.P.A., Clarke, M., Devereaux, P., Kleijnen, J., Moher, D.: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration. BMJ. 339 , b2700 (2009). https://doi.org/10.1136/bmj.b2700

Liu, G., Zhang, J.: CNID: Research on Network Intrusion Detection based on convolutional neural network. Discrete Dynamics Nat. Soc. 1–11 (2020). https://doi.org/10.1155/2020/4705982

Liu, H., Lang, B.: Machine learning and deep learning methods for intrusion detection systems: A Survey. Appl. Sci. 9 , 1–28 (2019)

Louati, F., Ktata, F.B.: A deep learning-based multi-agent system for intrusion detection. Appl. Sci. 675 (2), 1–13 (2020). https://doi.org/10.1007/s42452-020-2414-z

Kalpana, R.: Recurrent nonsymmetric deep auto encoder approach for network intrusion detection system. Measurement: Sens. 24 , 1–7 (2022). https://doi.org/10.1016/j.measen.2022.100527

Maheswari, K.G., Siva, C., Nalinipriya, G.: Optimal cluster based feature selection for intrusion detection system in web and cloud computing environment using hybrid teacher learning optimization enables deep recurrent network. Comput. Commun. 202 , 145–153 (2023). https://doi.org/10.1016/j.comcom.2023.02.003

Man, J., Sun, G.: A residual learning-based Network Intrusion Detection System. Secur. Communication Networks. 1–9 (2021). https://doi.org/10.1155/2021/5593435

Marir, N., Wang, H., Feng, G., Li, B., Jia, M.: Distributed abnormal Behaviour Detection Approach based on Deep Belief Network and Ensemble SVM using spark. IEEE Access., 59657–59671. (2018)

Masdari, M., Jalali, M.: A survey and taxonomy of DoS attacks in cloud computing. Secur. Communication Networks. (2016). https://doi.org/10.1002/sec.1539

Mohammadian, H., Ghorbani, A., A., Lashkari, A.H.: A gradient-based approach for adversarial attack on deep learning-based network intrusion detection systems. Appl. Soft Comput. 137 , 1–15 (2023). https://doi.org/10.1016/j.asoc.2023.110173

Mohammadpour, L., Ling, T.C., Liew, C.S., Aryanfar, A.: Survey of CNN-Based Network Intrusion Detection. Appl. Sci. 1–34 (2022). https://doi.org/10.3390/app12168162

Moore, S.J., Cruciani, F., Nugent, C.D., Zhang, S., Cleland, I., Sani, S.: Deep learning for network intrusion: A hierarchical approach to reduce false alarm. Intell. Syst. Appl. 18 , 1–13 (2023). https://doi.org/10.1016/j.jswa.2023.200215

NF-UQ-NIDS: (2024). https://www.kaggle.com/datasets/dhoogla/nfuqnids , last accessed 08th

Oseni, A., Moustafa, N., Creech, G., Sohrabi, N., Strelzoff, A., Tari, Z., Linkov, I.: An Explainable Deep Learning Framework for resilient intrusion detection in IoT-Enabled transportation networks. IEEE Trans. Intell. Transp. Syst. 24 (1), 1000–1014 (2023)

Otoum, S., Kantarci, B., Mouftah: On the feasibility of Deep Learning in Sensor Network Intrusion Detection. IEEE Netw. Lett. 1 (2), 68–71 (2019)

Pang, G., Shen, C., Cao, L., Hengel, D.V.A.: Deep learning for Anomaly detection: A review. ACM Comput. Surveys. 54 (2), 1–38 (2021). https://doi.org/10.1145/3439950

Panigrahi, R., Borah, S.: Syst. Int. J. Eng. Technol. 7 , 479–482 (2018). A detailed analysis of CICIDS2017 dataset for designing Intrusion Detection

Park, C., Lee, J., Kim, Y., Park, J.-G., Kim, H., Hong, D.: An enhanced AI-Based Network Intrusion Detection System using Generative Adversarial Networks. IEEE Internet Things J. 10 (3), 2330–2345 (2023)

Peterson, J.M., Leevy, J.L., Khoshgoftaar, T.M.: A review and Analysis of the BoT-IoT Dataset. In 2021 IEEE International Conference on Service-Oriented System Engineering (SOSE), 20–27. (2021)

Pingale, S.V., Sutar, S.R.: Remora based Deep Maxout model for network intrusion detection using convolutional neural network features. Comput. Electr. Eng. 110 , 1–12 (2023). https://doi.org/10.1016.j.compeleceng.2023.108831

Qazi, E., Haider, N., Shoaib, M., Razzak, I.: An intelligent and efficient network intrusion detection system using deep learning. Comput. Electr. Eng. 99 , 1–10 (2022). https://doi.org/10.1016/j.compeleceng.2022.107764

Ravi, V., Chaganti, R., Alazab, M.: Recurrent deep learning-based feature future ensemble meta-classifier approach for intelligent network intrusion detection system. Comput. Electr. Eng. 102 , 1–17 (2022). https://doi.org/10.1016/j.compeleceng.2022.108156

Ring, M., Wunderlich, S., Scheuring, D., Landes, D., Hatha, A.: Computers Secur. 1–41 (2019). https://doi.org/10.1016/j.cose.2019.06.005 A Survey of Network-based Intrusion Detection Data Sets

Saba, T., Rehman, A., Sadad, T., Kolivand, H., Bahaj, A.S.: Anamoly-based intrusion detection system for IoT networks through deep learning model. Comput. Electr. Eng. 99 , 1–10 (2022). https://doi.org/10.1016/j.compeleceng.2022.107810

Sanju, P.: Enhancing intrusion detection in IoT systems: A hybrid metaheuristics-deep learning approach with ensemble of recurrent neural networks. J. Eng. Res. 1–6 (2023). https://doi.org/10.1016/j.jer.2023/100122

Sharma, B., Sharma, L., Lal, C., Roy, S.: Anomaly based network intrusion detection for IoT attacks using deep learning technique. Comput. Electr. Eng. 107 , 1–14 (2023). https://doi.org/10.1016/j.compeleceng.2023.108626

Shone, N., Ngoc, T.N., Phai, V.D., Shi, Q.: A Deep Learning Approach to Network Intrusion Detection. IEEE Trans. Emerg. Top. Comput. Intell. 2 (1), 41–50 (2018)

Siddiqui, M.A., Pak, W.: Tier-based optimization for Synthesized Network Intrusion Detection System. IEEE Access. 10 , 108530–108544 (2022)

Sohi, S.M., Seifert, J.-P., Ganji, F.: RNNIDS: Enhancing network intrusion detection system through deep learning. Computers Secur. 102 , 1–18 (2021). https://doi.org/10.1016/j.cose.2020.102151

Susilo, B., Sari, R.F.: Intrusion Detection in IoT Networks Using Deep Learning Algorithm, pp. 1–11. Information (2020). https://doi.org/10.3390/info11050279

Thakkar, A., Lohiya, R.: Attack classification of Imbalanced Intrusion Data for IoT Network using ensemble-learning-based deep neural network. IEEE Internet Things J. 10 (13), 11888–11895 (2023)

Thakur, S., Chakraborty, A., De, R., Kumar, N., Sarkar, R.: Intrusion detection in cyber-physical systems using a generic and domain specific deep autoencoder model. Comput. Electr. Eng. 91 , 1–11 (2021). https://doi.org/10.1016/j.compeleceng.2021.107044

Tian, Q., Han, D., Li, K.-C., Liu, X., Duan, L., Castiglione, A.: An intrusion detection approach based on improved deep belief network. Appl. Intell. 3162–3178 (2020). https://doi.org/10.1007/s10489-020-01694-4

Tseng, C.H., Chang, Y.-T.: EBDM: Ensemble binary detection models for multi-class wireless intrusion detection based on deep neural network. Computers Secur. 133 , 1–8 (2023). https://doi.org/10.1016/j.cose.2023.103419

Udas, P.B., Karim, M.E., Roy, K.S.: SPIDER: A shallow PCA based network intrusion detection system with enhanced recurrent neural networks. J. King Saud Univ. – Comput. Inform. Sci. 34 , 10246–10272 (2022). https://doi.org/10.1016/j.jksuci.2022.10.019

Viegas, E.K., Santin, A.O., Oliveira, L.S.: Towards a reliable anamoly-based intrusion detection in real-world environments. Comput. Netw. 127 , 200–216 (2017). https://doi.org/10.1016/j.comnet.2017.08.013

Vinaykumar, R., Sonam, K.P., Pooranchandran, P.: A Comparative Analysis of Deep Learning Approaches for Network Intrusion Detection Systems (N-IDSs): Deep learning for N-IDSs. Int. J. Digit. Crime. Forensics. 11 (3), 65–89 (2019)

Vishwakarma, M., Kesswani, N.: DIDS: A deep neural network based real-time intrusion detection system for IoT. Decis. Analytics J. 5 , 1–9 (2022). https://doi.org/10.1016/j.dajour.2022.100142

Wahab, O.A.: Intrusion detection in the IoT Under Data and Concept drfits: Online Deep Learning Approach. IEEE Internet Things J. 9 (20), 19706–19716 (2022)

Wang, K., Zhang, A., Sun, H., Wang, B.: Analysis of recent deep-learning-based intrusion detection methods for In-Vehicle Network. IEEE Trans. Intell. Transp. Syst. 24 (2), 1843–1854 (2023)

Wang, Z., Xie, X., Chen, L., Song, S., Wang, Z.: Intrusion Detection and Network Information Security Based on Deep Learning Algorithm in Urban Rail Transit Management System. IEEE Trans. Intell. Transp. Syst. 24 (2), 2135–2143 (2023)

Wei, P., Li, Y., Zhang, Z., Hu, T., Li, Z., Liu, D.: An Optimization Method for Intrusion Detection Classification Model Based on Deep Belief Network, pp. 87593–87605. IEEE Access (2019)

Wu, Y., Lee, W.W., Xu, Z., Ni, M.: Large-Scale and Robust Intrusion Detection Model Combining Improved Deep Belief Network with Feature-Weighted SVM, pp. 98600–98611. IEEE Access (2020)

Wu, Y., Nie, L., Wang, S., Ning, Z., Li, S.: Intelligent Intrusion detection for internet of things Security: A Deep Convolution Generative Adversarial Network-Enabled Approach. IEEE Internet Things J. 10 (4), 3094–3106 (2023)

Wu, Z., Wang, J., Hu, L., Zhang, Z., Wu, H.: A network intrusion detection method based on semantic re-encoding and deep learning. J. Netw. Comput. Appl. 164 , 1–12 (2020). https://doi.org/10.1016/j.jnca.2020.102688

Xu, H., Sun, L., Fan, G., Li, W., Kuang, G.: A hierarchical intrusion detection model combining multiple deep learning models with attention mechanism. IEEE Access., 66212–66225. (2023)

Yadav, N., Pande, S., Khamparia, A., Gupta, D.: Intrusion detection system on IoT with 5G network using deep learning. Wirel. Commun. Mob. Comput. 1–13 (2022). https://doi.org/10.1155/2022/9304689

Yang, H., Qin, G., Ye, L.: Combining Wireless Network Intrusion Detection Model based on deep learning. IEEE Access. 7 , 82624–82632 (2019)

Yang, L., Li, J., Yin, L., Sun, Z., Zhao, Y., Li, Z.: Real-time intrusion detection in Wireless Network: A deep learning-based Intelligent mechanism. IEEE Access., 170128–170139. (2020)

Yang, Y., Zheng, K., Wu, C.,Y: Improving the Classification Effectiveness of Intrusion Detection by Using Conditional Variational Autoencoder and Deep Neural Network, pp. 1–20. Sensors (2019). https://doi.org/10.3390/s19112528

Chintapalli, N.S.S., Singh, P.S., Frnda, J., Divakarachari, B.P., Sarraju, L.V., Falkowski-Gilski, P.: OOA-modified Bi-LSTM network: An effective intrusion detection framework for IoT systems. Heliyon. 10 , 1–15 (2024)

Yogesh, Goyal, L.M.: A Systematic Literature Review of Network Intrusion Detection System Models. In: Yadav, A., Nanda, S.J., Lim, MH. (eds) Proceedings of International Conference on Paradigms of Communication, Computing and Data Analytics. PCCDA 2023. Algorithms for Intelligent Systems. Springer, Singapore. (2023). https://doi.org/10.1007/978-981-99-4626-6_38

Zhang, C., Costa-Perez, X., Patras, P.: Adversarial Attacks against Deep Learning-Based Network Intrusion Detection Systems and Defense mechanisms. IEEE/ACM Trans. Networking. 30 (3), 1294–1311 (2022)

Zhang, C., Ruan, F., Yin, L., Chen, X., Zhai, L., Liu, F.: A Deep Learning Approach for Network Intrusion Detection Based on NSL-KDD Dataset. In 2019 IEEE 13th International Conference on Anti-counterfeiting, Security, and Identification (ASID), 41–45. (IEEE) (2019)

Zhang, H., Li, Y., Lv, Z., Sangaiah, A.K., Huang, T.: A real-time and Ubiquitous Network Attack Detection Based on Deep Belief Network and Support Vector Machine. IEEE/CAA J. Automatica Sinica. 7 (3), 790–799 (2020). https://doi.org/10.1109/JAS.2020.1003099

Zhao, R., Yin, J., Xue, Z., Adebisi, B., Ohtsuki, T., Gacanin, H., Sari, H.: An efficient intrusion detection method based on dynamic autoencoder. IEEE Wirel. Communication Lett. 10 (8), 1707–1711 (2021)

Nojood, O., Aljehane, H.A., Mengash, M.M., Eltahir, F.A., Alotaibi, S.S., Aljameel, A., Yafoz, R., Alsini, M.A.: Golden jackal optimization algorithm with deep learning assisted intrusion detection system for network security. Alexandria Eng. J. 86 , 415–424 (2024). https://doi.org/10.1016/j.aej.2023.11.078

Devendiran, R., Turukmane, V.A.: Dugat-LSTM: Deep learning based network intrusion detection system using chaotic optimization strategy. Expert Syst. Appl. 245 (2024). https://doi.org/10.1016/j.eswa.2023.123027

Osa, E., Orukpe, E.P., Irunasi, U.: Eng. Electron. Energy. 7 , 1–6 (2024). https://doi.org/10.1016/j.prime.2024.100434 Design and implementation of a deep neural network approach for intrusion detection systems

Wu, Y., Hu, Y., Wang, J., Feng, M., Dong, A., Yang, Y.: An active learning using deep Q-network for zero-day attack detection. Computers Secur. 139 (2024). https://doi.org/10.1016/j.cose.2024.103713

117. Maddireddy, B. R., & Maddireddy, B. R. (2024). A Comprehensive Analysis of Machine Learning Algorithms in Intrusion Detection Systems. Journal Environmental Sciences And Technology, 3(1), 877-891.

Bhati, B. S., Shankar, A., Saxena, S., Saxena, T., Anbarasi, M., & Kumar, M. (2022). Analysis of Support Vector Machine-based Intrusion Detection Techniques. Arabian Journal for Science and Engineering Springer, 1-13. DOI: https://doi.org/10.1007/s13369-019-03970-z

Bhati, B.S., Shankar, A., Saxena, S., Saxena, T., Anbarasi, M., Kumar, M.: An ensemble-based approach for image classification using voting classifier. Int. J. Model. Ident. Control. 41 (1–2), 87–97 (2022)

Bhati, B.S., Rai, C.S.: A survey on intrusion detection tools. In 2019 6th International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 806–810). IEEE. (2019), March

Bhati, N.S., Khari, M., Comparative Analysis of Classification Based Intrusion Detection Techniques,: 5th International Conference on Information Systems and Computer Networks (ISCON) , Mathura, India, 2021, pp. 1–6, (2021). https://doi.org/10.1109/ISCON52037.2021.9702411

Bhati, N.S., Khari, M.: An Ensemble Model for Network Intrusion Detection Using AdaBoost, Random Forest and Logistic Regression. In: Unhelker, B., Pandey, H.M., Raj, G. (eds.) Applications of Artificial Intelligence and Machine Learning. Lecture Notes in Electrical Engineering, vol. 925. Springer, Singapore (2022). https://doi.org/10.1007/978-981-19-4831-2_64

Chapter Google Scholar

Bhati, B.S., Dikshita, Bhati, N.S., Chugh, G.: A comprehensive study of intrusion detection and prevention systems. Wirel. Communication Secur., 115–142. (2022). (Bhoopesh Singh Bhati, Dikshita, Nitesh Singh Bhati, Garvit Chugh Book Editor(s):Manju Khari, Manisha Bharti, M. Niranjanamurthy)

Bhati, N.S., Khari, M., García-Díaz, V., Verdú, E.: A review on intrusion detection systems and techniques. Int. J. Uncertain. Fuzziness Knowledge-Based Syst. 28 (Supp02), 65–91 (2020)

Vanin, P., Newe, T., Dhirani, L.L., O’Connell, E., O’Shea, D., Lee, B., Rao, M.: A study of network intrusion detection systems using artificial intelligence/machine learning. Appl. Sci. 12 (22), 11752 (2022)

Zhang, C., Jia, D., Wang, L., Wang, W., Liu, F., Yang, A.: Comparative research on network intrusion detection methods based on machine learning. Computers Secur. 121 , 102861 (2022)

Download references

The author did not receive any support from any organization for the submitted work.

Author information

Authors and affiliations.

Department of Computer Engg., J.C. Bose University of Science & Technology, YMCA, Faridabad, Haryana, 121006, India

Yogesh & Lalit Mohan Goyal

You can also search for this author in PubMed Google Scholar

Contributions

Yogesh: Conceptualization, methodology, Investigation, Validation, Formal analysis, Writing — original Draft, Writing-Review & editing, VisualizationLalit Mohan Goyal: Validation, Writing — review & editing, Project administration, Resources, Supervision.

Corresponding author

Correspondence to Yogesh .

Ethics declarations

Ethical approval.

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of interest

The authors declare that there is no conflict of interest in this paper.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Yogesh, Goyal, L.M. Deep learning based network intrusion detection system: a systematic literature review and future scopes. Int. J. Inf. Secur. (2024). https://doi.org/10.1007/s10207-024-00896-y

Download citation

Published : 02 August 2024

DOI : https://doi.org/10.1007/s10207-024-00896-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Convolutional neural network
Deep learning
Network security
Autoencoder
Network intrusion detection system

Find a journal
Publish with us
Track your research

Loading metrics

Open Access

Peer-reviewed

Research Article

Uncertainty in tuberculosis clinical decision-making: An umbrella review with systematic methods and thematic analysis

Roles Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Oxford Vaccine Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom, Department of Global Health and Development, London School of Hygiene & Tropical Medicine, London, United Kingdom

Roles Conceptualization, Methodology, Project administration, Writing – review & editing

Affiliation Department of Global Health and Development, London School of Hygiene & Tropical Medicine, London, United Kingdom

Roles Methodology, Writing – review & editing

Roles Supervision, Writing – review & editing

Affiliations Oxford Vaccine Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom, Department of Paediatrics, Maastricht University Medical Centre, MosaKids Children’s Hospital, Maastricht, the Netherlands

Roles Conceptualization, Writing – review & editing

Affiliation Department of Epidemiology of Microbial Diseases, Yale School of Public Health, New Haven, Connecticut, United States of America

Affiliations Department of Global Health and Population, Harvard TH Chan School of Public Health, Boston, Massachusetts, United States of America, Center for Health Decision Science, Harvard TH Chan School of Public Health, Boston, Massachusetts, United States of America

Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

Roles Conceptualization, Methodology, Project administration, Supervision, Writing – review & editing

Francesca Wanda Basile,
Sedona Sweeney,
Maninder Pal Singh,
Else Margreet Bijker,
Ted Cohen,
Nicolas A. Menzies,
Anna Vassall,
Pitchaya Indravudh

Published: July 23, 2024
https://doi.org/10.1371/journal.pgph.0003429
Peer Review
Reader Comments

Tuberculosis is a major infectious disease worldwide, but currently available diagnostics have suboptimal accuracy, particularly in patients unable to expectorate, and are often unavailable at the point-of-care in resource-limited settings. Test/treatment decision are, therefore, often made on clinical grounds. We hypothesized that contextual factors beyond disease probability may influence clinical decisions about when to test and when to treat for tuberculosis. This umbrella review aimed to identify such factors, and to develop a framework for uncertainty in tuberculosis clinical decision-making. Systematic reviews were searched in seven databases (MEDLINE, CINAHL Complete, Embase, Scopus, Cochrane, PROSPERO, Epistemonikos) using predetermined search criteria. Findings were classified as barriers and facilitators for testing or treatment decisions, and thematically analysed based on a multi-level model of uncertainty in health care. We included 27 reviews. Study designs and primary aims were heterogeneous, with seven meta-analyses and three qualitative evidence syntheses. Facilitators for decisions to test included providers’ advanced professional qualification and confidence in tests results, availability of automated diagnostics with quick turnaround times. Common barriers for requesting a diagnostic test included: poor provider tuberculosis knowledge, fear of acquiring tuberculosis through respiratory sampling, scarcity of healthcare resources, and complexity of specimen collection. Facilitators for empiric treatment included patients’ young age, severe sickness, and test inaccessibility. Main barriers to treatment included communication obstacles, providers’ high confidence in negative test results (irrespective of negative predictive value). Multiple sources of uncertainty were identified at the patient, provider, diagnostic test, and healthcare system levels. Complex determinants of uncertainty influenced decision-making. This could result in delayed or missed diagnosis and treatment opportunities. It is important to understand the variability associated with patient-provider clinical encounters and healthcare settings, clinicians’ attitudes, and experiences, as well as diagnostic test characteristics, to improve clinical practices, and allow an impactful introduction of novel diagnostics.

Citation: Basile FW, Sweeney S, Singh MP, Bijker EM, Cohen T, Menzies NA, et al. (2024) Uncertainty in tuberculosis clinical decision-making: An umbrella review with systematic methods and thematic analysis. PLOS Glob Public Health 4(7): e0003429. https://doi.org/10.1371/journal.pgph.0003429

Editor: Sok King Ong, The Chinese University of Hong Kong Faculty of Medicine, HONG KONG

Received: December 19, 2023; Accepted: June 12, 2024; Published: July 23, 2024

Copyright: © 2024 Basile et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All data relevant to replicate the study have been provided in the article or uploaded as supplementary information .

Funding: Research reported in this publication was supported by the National Institute Of Allergy And Infectious Diseases of the National Institutes of Health (U01AI152084 to FWB, SS, MPS, TC, NAM, AV, PI). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Tuberculosis (TB) is a major infectious cause of morbidity and mortality globally. In 2022, 7.5 million people were diagnosed with TB, and 1.3 million people died because of the disease [ 1 ]. Missed or delayed TB diagnosis and treatment and low quality of care remain critical obstacles to disease control and improving health outcomes [ 2 , 3 ].

To minimize diagnostic and treatment delays, high quality TB services should include access to rapid, affordable and accurate tests, such as the molecular WHO-recommended rapid diagnostics (mWRD) [ 4 ]. However, mWRD are seldom available at the point-of-care in resource-limited settings. Despite massive efforts to coordinate the global roll-out of GeneXpert (Cepheid, USA), recent data still show that this test is unavailable in many peripheral settings and more generally the underutilization of modern TB diagnostic technologies [ 5 , 6 ].

The underutilization of diagnostics may arise due to a variety of factors, including as a consequence of providers’ know-do gap [ 7 ]. This may become particularly evident in situations where care is tailored around the patient’s perceived needs (e.g., clinicians offering a more affordable but less accurate diagnostic test), and best practices are not implemented (e.g., clinicians choosing quick symptom relief with low-cost pharmaceuticals over diagnostic certainty) [ 7 ]. Moreover, in resource-limited settings, when a patient presents with signs and symptoms suggestive of TB, clinicians may decide to start treatment based solely on clinical grounds, regardless of test availability [ 8 , 9 ].

To standardize decision-making, pre- and post-test disease probabilities have been used to determine the thresholds for testing and treatment decisions [ 10 , 11 ]. The provider determines a pre-test probability of disease, which varies depending on clinical signs and symptoms as well as the provider’s experience, knowledge, and health care setting. The provider then decides whether to move forward with testing or initiating treatment. Following testing, the provider determines the post-test probability of disease and decides whether to start or withhold TB therapy [ 11 ].

There have also been multiple attempts to develop scoring systems and clinical prediction models for TB screening and diagnosis [ 12 – 16 ]. Scoring systems can help to calculate the probability of TB disease in a reproducible way and might be particularly helpful in paediatric TB, where currently available diagnostic tests lack high sensitivity. Additionally, clinical algorithms might help determine when testing is helpful and when a negative test is insufficient to withhold treatment [ 17 ].

However, in reality, the decision to test or treat presumptive TB cases can be affected by contextual variables beyond accessibility to diagnostics, or a mere computation of disease probability [ 18 ]. Provider characteristics, including their ability to cope with complexity, risk, and uncertainty, contribute to process variability [ 19 ]. Uncertainty is an inevitable component of clinical practice and can occur throughout the decision-making process: when formulating clinical hypotheses, identifying a diagnosis, choosing a test and interpreting its result, and interpreting patient preferences [ 20 ]. Multilevel models of uncertainty emphasize the dynamic interplay between different sources and types of uncertainty at each level, and may be useful to classify the challenges of clinical decision-making [ 20 ].

Understanding uncertainty in the TB decision-making process and the reasons why a provider would initiate empiric treatment or would not utilize a microbiological test even when available, is important to develop diagnostic tools that improve TB diagnosis and care behaviours and practices, and to project the impact of the introduction of novel diagnostic aids [ 21 ]. This umbrella review of systematic reviews (SR) aimed to identify factors influencing providers’ decisions to test for TB, and initiate TB treatment in adult and paediatric patients with presumptive TB in high-TB and TB/HIV burden countries [ 22 ].

Study design rationale and methodology

An initial scoping search was conducted on MEDLINE (via OVID) for terms related to “tuberculosis” and “decision-making”, and identified several reviews relevant to our research question [ 23 – 25 ]. Since most records evaluated either qualitative or quantitative primary studies, and often reported complementary findings, we chose an umbrella review design to allow for the inclusion of these reviews with a broad scope of inquiry and to achieve a higher level of synthesis [ 26 – 28 ].

The study was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [ 29 ]. The Joanna Briggs Institute (JBI) guidelines for umbrella reviews [ 28 , 30 ], and the Cochrane guidance for overviews of reviews [ 31 ] were also followed to address the specific issues arising when conducting umbrella reviews. The methodology of this review was prespecified in a protocol [ 32 ].

Search strategy

Using a combination of key terms to maximize sensitivity, seven electronic databases were searched: MEDLINE (via OVID), CINAHL Complete, Embase, Scopus, Cochrane Central, the PROSPERO register, and Epistemonikos database. The search was limited from January 2007 (considering that the development Xpert MTB/RIF was completed in 2009) to the date of search, which was the 4th of July 2022. The search was rerun on the 21st of July 2023. We developed a comprehensive list of keywords and synonyms for each broad domain: 1) TB, 2) clinical decision-making. Terms were searched individually first and then combined using Boolean operators. The search was piloted in MEDLINE and repeated in all databases. Where applicable, MeSH and free text terms were combined to identify relevant studies. The search strategy was developed with the support of a librarian at the LSHTM.

Details on the search strategy are presented in S1 Appendix . Articles in English, French, Spanish, Portuguese, or Italian were considered. A search of the grey literature was not conducted.

Selection and appraisal of records

Records were selected on predefined inclusion and exclusion criteria guided by the Population, Intervention, Comparison, Outcome and Study design/setting (PICOS) framework ( S1 Table ) [ 33 ]. Inclusion criteria consisted of population (individuals with presumptive pulmonary TB and health care providers involved in TB diagnosis and treatment), findings/outcomes (any relevant to clinical decision-making), and setting (high TB burden countries). We considered relevant to the decision-making any intervention, action, or event that influenced the diagnosis of TB. SRs, meta-analyses, and SRs of qualitative studies (hereinafter referred to as qualitative evidence syntheses) were included. Articles exclusively on drug-resistant TB, non-review articles, and reviews that did not use systematic methods were excluded ( S1 Table ).

Following removal of duplicates, the title/abstract screening was carried out by a single reviewer (FWB). The full text of selected records was then examined for inclusion in the study, based on the predefined criteria ( S1 Table ).

Quality appraisal

Methodological quality, risk of bias and reporting quality of reviews were assessed using the JBI checklist for SRs [ 28 , 30 ]. No records were excluded on grounds of quality due to lack of consensus on the most appropriate tools and approaches for managing low-quality reviews in umbrella reviews [ 34 ] ( S2 Table ). Where available, GRADE assessments [ 35 , 36 ] were extracted and reported.

Overlap assessment

Several approaches have been proposed for overlap management in umbrella reviews [ 37 ]. We included all eligible reviews and documented the extent of overlap in primary studies using the Corrected Covered Area (CCA) index [ 37 ]. After obtaining the overall CCA, pairwise indexes were calculated ( S1 Fig ). For reviews with moderate to high pairwise CCA, research aims and reported outcomes were examined. If two reviews had the same aims, findings from the highest quality review were described [ 37 ].

Data extraction

Study characteristics and data of interest for included records were extracted by a single reviewer (FWB) [ 30 , 31 , 38 ]. These included: type of review, title, authors, publication year, number of studies and participants included in the review, aims/objectives/PICO question (or equivalent), search strategy, methodological quality/risk of bias and certainty of evidence assessment. For reviews examining global data, only findings pertinent to high-burden TB settings were extracted. Data extraction also indicated where pooled analyses included non-high TB burden countries.

Primary studies from reviews were not retrieved.

Data synthesis

Data synthesis used a systematic narrative approach for umbrella reviews [ 38 ], which involved thematic content analysis and coding of findings from each review to identify recurring themes associated with factors influencing TB clinical decision-making. Nvivo (version 1.5, 2021, QSR International Pvt. LTD, Australia) was used to iteratively code extracted key data. Themes were developed separately for quantitative and qualitative studies, then combined and presented complementarily [ 39 , 40 ].

Barriers and facilitators for TB testing or treatment decisions from each review were coded first, and then grouped under common themes associated with decision-making uncertainty, based on the taxonomy developed by Eachempati et al . [ 20 ]. The taxonomy develops around macro (society and community), meso (group relationships), and micro (individual) levels of uncertainty to emphasize the dynamic interplay between different sources and types of uncertainty at each level, and may be useful to classify challenges in health care decision-making [ 20 ].

Recurring themes were further classified based on an adapted version of the WHO conceptual framework representing the TB diagnosis and care continuum [ 41 ]. The framework helped to identify four levels (patient, provider, health system, diagnostic test) of factors influencing TB clinical decision-making, including three time-points (patient-provider encounter, diagnosis, treatment initiation) for decision-making. The framework captures both the determinants (i.e., what causes decisional uncertainty) and broader sources (i.e., what contributes to the variability of decisional outcomes) of uncertainty in the decision-making process.

Definitions

Presumptive pulmonary TB was defined as clinical/pre-test suspicion or post-test suspicion despite a negative test. Diagnostic delay was defined as the time lag from first access to health system/consultation with provider to diagnosis; treatment delay was defined as the time lag from diagnosis to treatment initiation. Provider/health system delay was used to refer to any diagnostic or treatment delay attributable to provider or health systems factors (to differentiate from causes of delay attributable to patient factors).

Review characteristics

Database searches yielded 8542 records. After duplicate removal, 7345 unique records were screened by title/abstract. After full text screening of 110 records, a total of 27 reviews were included ( Table 1 ). The PRISMA flow chart detailing the phases of study selection is presented in Fig 1 .

PPT PowerPoint slide
PNG larger image
TIFF original image

https://doi.org/10.1371/journal.pgph.0003429.g001

https://doi.org/10.1371/journal.pgph.0003429.t001

Articles were published between 2008 and 2023. Records included nine meta-analyses, three qualitative evidence syntheses, and 15 mixed-methods narrative synthesis ( Table 1 ). Primary studies included in the reviews spanned from 1970 to 2021 and were mostly observational ( Table 1 ).

Reviews varied in inclusion criteria, outcomes, settings, and population. Based on their primary aim, reviews were classified into four main categories: diagnostic and treatment delays (n = 8); knowledge, attitudes, and practices of TB healthcare providers and end users (n = 5); barriers and facilitators to utilization of TB diagnostic services (n = 10); diagnostic test impact on diagnosis and treatment (n = 4). Most reviews included primary studies with adult populations or did not include sub-group analysis by age. One review focused specifically on children and adolescents [ 25 ]. Key population and outcome definitions were generally consistent. Prior to the review, standardized definitions were developed allowing for direct comparison and a narrative synthesis of findings ( Table 1 ).

Most reviews were of fair or good methodological quality. The main areas compromising methodological quality and confidence in findings were publication bias, not using consistent methods to minimize errors in data extraction, and not grading the quality of evidence ( S2 Table ). Based on the global CCA index, most reviews had very low to no overlap. Seven pairs had high or very high primary source overlap. The citations list from one review [ 42 ] was not available, hence not included in CCA calculations ( S1 Fig ).

Main findings from thematic content analysis

Through iterative thematic analysis, 15 recurring themes were identified. Applying an integrated multilevel model of uncertainty in health care [ 20 ], the themes were classified by type of uncertainty ( Table 2 ). Findings were then classified as barriers or facilitators for testing and treatment decisions ( Fig 2 ).

https://doi.org/10.1371/journal.pgph.0003429.g002

https://doi.org/10.1371/journal.pgph.0003429.t002

Synthesis enabled the development of a framework for uncertainty in TB decision-making, presented in Fig 3 . Types of uncertainty were grouped in four macro-levels, corresponding to sources of uncertainty in TB clinical decision-making: patient-, provider-, diagnostics-, and health system-related uncertainty. The framework represents the relationship between the four sources of uncertainty and three key moments in clinical decision-making: the clinical encounter, the formulation of a diagnostic hypothesis, and the treatment initiation. Different types of uncertainty may act synergistically at given time-points ( Fig 3 ).

https://doi.org/10.1371/journal.pgph.0003429.g003

Clinical uncertainty

Clinical presentation..

Half of the reviews mentioned the relationship between clinical presentation and clinicians’ suspicion of TB. Meta-analysis data from GeneXpert and urine lipoarabinomannan (LAM) diagnostic impact studies suggested a higher likelihood of being treated empirically for sicker patients requiring hospitalization [ 43 , 44 ]. In the review by Getnet et al ., findings from observational studies indicated that the absence of cough and the presence of atypical symptoms, fever, or good clinical conditions were associated with provider diagnostic delay [ 45 ]. Similarly, patients presenting with chronic cough and other concomitant lung disease including COVID-19 were reported to experience delays [ 24 , 46 ].

Socio-demographic characteristics.

Three reviews identified several indicators of patient socio-economic status, including poor literacy, low income or unemployment, lack of health insurance, and rural residence, as factors associated with diagnostic delay [ 45 , 47 , 48 ]. The reviews by Yang et al . and Krishnan et al ., which focused on gender-related differences in access to TB services and had moderate overlap, reported inconsistent evidence of a positive relationship between female sex and provider delay in TB diagnosis [ 47 , 49 ]. Yang et al . additionally reported differences by setting. For example, providers from Thailand and Vietnam were more likely to adhere to diagnostic guidelines with male patients, whereas providers from India offered testing with similar frequency to both women and men [ 49 ]. The meta-analysis by Getnet et al . showed no evidence of a difference in the proportion of male versus female patients diagnosed with TB at the 30-day mark (pooled odds ratio (OR) = 1.08, 95% CI 0.95–1.23) [ 45 ]. Similarly, Li et al . reported no evidence of an association between female sex for patients and diagnostic delay in China (pooled OR = 1, 95%CI 0.83–1.22) [ 48 ].

TB-related risk factors.

Treatment was often delayed in patients with a previous diagnosis of TB [ 50 ] and in patients who reported antibiotic usage prior to the clinical encounter [ 51 ]. Among paediatric patients, providers were less likely to start empiric treatment in cases with unknown TB exposure [ 25 ].

Collection of diagnostic specimens.

Reviews reported that the inability of patients to produce sputum influenced decisions to initiate testing and treatment [ 52 ]. In Nathavitharana et al ., the proportion of adults able to provide a sputum sample ranged between 57% and 97% in people living with HIV (PLHIV), depending on setting and severity/type of symptoms. In contrast, urine collection (for the LAM assay) was achieved in 99% of PLHIV aged 15 and above across three RCTs [ 43 ]. Challenges with specimen collection influenced decisions to withhold microbiological testing and either initiate empiric treatment or to exclude TB solely based on the clinical interview or radiological examination findings [ 52 , 53 ]. Engel et al ., found with high confidence in evidence that providers highly valued the possibility of using alternative samples for testing such as urine or stool, particularly for paucibacillary cases and paediatric TB [ 52 ].

Side effects.

Reviews also reported provider decisions to withhold or delay treatment initiation because of fear of TB drug side effects in children [ 25 , 52 ].

Personal uncertainty

Provider attitudes, beliefs, and stigma..

Multiple reviews found that provider behaviour and discriminatory attitudes can impact TB diagnosis and treatment initiation [ 25 , 39 , 49 , 52 – 57 ]. In a qualitative evidence synthesis, Barnabishvili et al . reported that providers were less rigorous when interviewing older patients or foreigners during the clinical encounter [ 39 ]. Provider discrimination towards female patients, resulting in tests underutilization and delays, emerged from narrative syntheses [ 39 , 45 , 48 , 51 ]. Provider TB/HIV coinfection-related stigma was reported in three reviews as a factor delaying diagnosis or treatment initiation [ 52 , 53 , 55 ], including one review with high confidence in evidence [ 52 ].

Fear of infection.

Two reviews based on qualitative data, including one review with high confidence in evidence [ 52 ], reported that providers were generally aware of the aerosol biohazard and hesitant to test for TB because of fear of acquiring the disease [ 51 , 52 ]. Fear of infection from respiratory specimen collection, particularly gastric aspiration, resulted in underutilization of diagnostic tools [ 52 ] or collection of poor quality respiratory specimens [ 51 , 52 ]. In the context of the SARS-CoV-2 pandemic, some providers refused to collect respiratory specimens among presumptive TB patients presenting with COVID-19 symptoms [ 46 ].

Test characteristics and provider preference.

Diagnostic accuracy, automation, and computer-based tests were highly valued by providers based on moderate confidence in evidence [ 52 ]. Among paediatric patients, difficulties in collecting respiratory specimens (e.g., induced sputum or gastric aspirate), invasiveness of the procedure, and the lack of adequately trained staff were reported as barriers to test utilization [ 52 , 58 ].

Relational and knowledge exchange uncertainty

Patient-provider communication dynamics..

Some reviews reported that provider miscommunication with patients was a potential cause of missed diagnoses [ 39 , 42 , 51 – 54 ]. The difficulty in communicating with the patient was often reported as the consequence of TB-related stigma, but it also arose from the use of metaphors for clinical explanations, resulting in patients not understanding diagnostic and therapeutic plans, and losses to follow up [ 53 , 54 ]. One review reported that male providers disclosed difficulties communicating with, and understanding health concerns from, female patients during consultation [ 39 ].

Epistemic uncertainty

Provider knowledge and qualification..

Qualitative findings from twelve reviews suggested that suboptimal TB knowledge impacted providers’ ability to prescribe diagnostic tests or caused providers to delay TB diagnosis and miss treatment opportunities [ 24 , 25 , 48 , 50 , 52 , 54 , 56 , 57 , 59 – 62 ]. In a review on practices and knowledge of Indian providers, Satyanarayana et al. reported that the proportion of providers that suspected TB in the presence of a persistent cough of more than 2–3 weeks duration ranged from 21% to 81%, and less than 60% of patients with persistent cough were advised to undergo sputum examination [ 59 ].

A review by Teo et al . reported that poor clinical standards and low levels of knowledge of TB among providers led to delays in TB diagnosis in 12 qualitative studies, with high confidence in evidence [ 50 ]. Braham et al. reported one primary study where less than 50% of providers were aware of the principal diagnostic tools needed for TB diagnosis [ 61 ]. Poor TB knowledge and clinical skills resulted in deferral of bacteriological testing and preference for smear microscopy over mWRD, according to the narrative review by Shah et al .[ 57 ]. Additionally, the same review reported that providers’ unawareness and non-adherence to diagnostic algorithms was a reason for missed diagnoses [ 57 ].

Health care workers with particularly low levels of knowledge included informal providers [ 62 ], public providers working at the primary level, private practitioners with limited awareness of TB, and traditional healers [ 23 , 24 ]. One review found that recognition of TB symptoms was associated with providers’ level of qualification and public sector employment [ 42 ]. In contrast, age, sex, years of practice, experience, and level of qualification were not associated with identification of TB symptoms [ 42 , 54 ].

The meta-analysis by Amare et al . of nine intervention trials demonstrated that training interventions improve the ability of providers to diagnose TB, significantly increasing the number of bacteriologically confirmed cases [ 60 ].

Availability of policies and guidelines.

Lack of clear and updated guidelines and poor dissemination at primary healthcare levels and among private providers led to poor referral to GeneXpert testing, or inconsistency in the types of samples used [ 52 , 57 , 62 ]. The review by Shah et al. reported guidelines and policies variability in the private sector as one cause of missed diagnoses [ 57 ].

Test uncertainty

Utilization and impact of diagnostic tools..

Engel et al . found that in settings where low-complexity mWRDs were easily accessible, providers reported a high level of trust in the test result [ 52 ]. The meta-analysis by Lee et al reported that availability of mWRDs reduced diagnostic and treatment delays [ 63 ]. Three meta-analyses examined GeneXpert diagnostic impact [ 44 , 58 , 64 ], with outcomes reported only from the most recent review [ 44 ]. The use of GeneXpert (versus smear microscopy) had no effect on the proportion of participants treated for TB (risk ratio 1.10, 95% CI 0.98–1.23; GRADE: moderate confidence) [ 44 ]. This could reflect decisions to treat some patients empirically regardless of test results. The lower sensitivity of GeneXpert in paucibacillary forms of the disease, such as paediatric TB, was recognized as a limitation that would justify empiric treatment initiation [ 58 ].

Health-system uncertainty

Operational setting deficiencies..

Twelve reviews reported on challenges at the health systems level. Inadequate staff trainings, lack of diagnostic resources, lack of personal protective equipment and infection prevention control measures, and absence of private rooms for clinical assessment were mentioned as potential contributors to missed diagnosis and treatment opportunities [ 25 , 42 , 46 , 51 , 55 – 57 ]. Private and rural clinics not offering TB services were associated with diagnostic delays compared with public, urban facilities, where providers had better access to tests and infrastructure [ 39 , 50 , 53 , 61 ].

Availability and timing of test results.

Sullivan et al . reported missed treatment opportunities in children due to long waiting times for culture results [ 25 ]. Reviews found that rapid test turnaround time was important to accelerate therapeutic decisions [ 25 , 52 , 56 ], and that offering same-day test and treat would reduce gaps in missed treatment according to providers [ 52 ].

Diagnostic test availability, accessibility, and affordability.

Reviews reported that the limited availability of resources for microbiological diagnosis (e.g., due to stock-outs, power cuts, and unreliable supply chains) was associated with GeneXpert underutilization and diagnostic delays [ 48 , 51 , 52 , 57 ]. Engel et al . reported on providers’ perspectives regarding the impact of diagnostic accessibility and affordability on test and treatment decisions. Frequent stock-outs were reported to potentially hinder providers’ faith in the adoption of new diagnostics and hamper their reliance on prescribing diagnostic tests in the future [ 52 ]. Further, some providers disclosed a preference for initiating treatment if patients incurred excessive costs for testing, regardless of test availability [ 52 ].

This umbrella review showed the complexity of multi-level factors that contribute to uncertainty in TB clinical decision-making, often resulting in under-utilization of diagnostic resources, misdiagnoses, empirical treatment or missed treatment opportunities, and diagnostic and treatment delays. The results of this study reinforce the concept that clinical decision-making is highly dependent on individual and interpersonal factors (provider, patient), but also closely linked to the operational context and the usability of diagnostic resources. These findings are important to inform the development of successful diagnostic aids and programs implementation strategies, and to improve TB practices in high-burden, resource-limited settings.

An important output from this study was the consolidation of a framework to present multilevel factors associated with uncertainty in TB decision-making. We found that several factors related to the local context and often beyond providers’ control were responsible for the discrepancy between TB testing and treatment decisions and scientific guidelines’ recommendations. Most of the existing literature on TB diagnostics includes diagnostic accuracy studies or randomized controlled trials that do not examine the challenges of clinical decision-making and the impact of health systems factors on diagnostic interventions. Rapid molecular diagnostics such as GeneXpert have had a great influence on TB care but there are ongoing concerns about underutilization and sustainability that need to be addressed [ 6 ]. Unfortunately, diagnostic tests, despite being cheap, fast, and accurate, are not always used as recommended–or not used at all–in high-burden settings, and it is crucial to increase our understanding of the underlying reasons [ 8 , 65 ].

Reviews reported consistent evidence for patient characteristics, symptom variability and severity as primary sources of clinical uncertainty in TB decision-making [ 24 , 47 , 54 , 59 , 61 ]. When confronted with hospitalized patients, patients with advanced HIV disease, or paediatric patients, providers seemed more inclined to treat empirically, regardless of the availability of diagnostic aids, possibly also because of the complexity of obtaining clinical specimens from people in these categories [ 25 , 43 ]. Additionally, history of previous TB diagnosis was associated with retreatment delays [ 50 ], potentially due to lack of confidence in diagnosis, or fear of drug side effects with injectables [ 66 ]. Further research is needed to uncover provider-related factors associated with retreatment decision-making, as rapid tests for second-line drug resistance testing and all-oral regimens become available [ 67 , 68 ].

Providers’ limited knowledge of TB symptoms and approaches for clinical and diagnostic management, and insufficient familiarity with guidelines, were reported consistently as key contributors to delay in test and treatment decisions [ 24 , 25 , 48 , 50 , 52 , 54 , 56 , 57 , 59 – 62 ]. Epistemic uncertainty affected several aspects of the decision-making, including estimating pre-test disease probabilities, deciding to use a diagnostic test, selecting appropriate specimens based on age and disease localization, collecting good quality samples, and interpreting test results [ 50 , 59 , 61 ]. Conversely, the availability of highly qualified physicians, public sector facilities, and ease of access to mWRD had a positive influence on testing decisions [ 42 , 52 ]. Notably, training interventions significantly improved case detection and test uptake by providers [ 60 ].

The central role of the provider in the decision-making process was also supported by extensive evidence on how interpersonal attitudes, beliefs, stigma, fear of infection, and test preferences affected test utilization and treatment decisions [ 25 , 39 , 49 , 52 – 57 ]. Personal sources of uncertainty, including fear of acquiring TB through respiratory sampling, were commonly reported barriers for underutilization of diagnostics [ 51 , 52 ]. As seen with other respiratory infectious diseases, fear of infection was mostly associated with poor knowledge of biohazard mitigation strategies, ambiguous guidelines, and lack of resources [ 69 ]. These findings support the importance of enhancing comprehensive national training and educational programs for providers at all levels of care, and engaging the private sector [ 61 , 70 ]. Similarly, the fear of acquiring TB could be, at least partially, addressed through continuous training, and implementation of infection prevention control measures [ 71 ].

The high variability of provider-patient interactions during the clinical encounter was often reported as a source of relational uncertainty affecting the outcomes of the clinical decision-making process [ 39 , 42 , 53 ]. Provider personal biases could result in the inability or unwillingness to collect all necessary clinical information, diagnostic test under-utilization, misdiagnosis and diagnostic delays, especially when confronted with female patients [ 48 , 49 , 51 , 61 ]. Although findings from meta-analyses did not confirm the association between female sex and diagnostic delays, moderate-quality qualitative sources reported the impact of gender on clinical decision-making [ 48 , 49 , 51 ]. Gender-related disparities in TB are well-known, especially with regards to health seeking behaviours and retention in care [ 51 ]. While TB incidence is greater in men [ 72 ], women generally face additional barriers related to care access, stigma and psychosocial consequences of the diagnosis [ 51 ]. The findings from this study confirm the importance of a gender-based approach to TB as advocated by WHO [ 73 ]. At the same time, quantitative and qualitative studies across settings and countries with different gender norms are needed to gain further insight on gaps in the TB diagnostic cascade, gender inequalities and discrimination, to inform TB interventions that have the capacity to overcome gender barriers [ 74 ].

Providers had high confidence in rapid diagnostic tests, but the confidence in mWRDs, namely GeneXpert, appeared to be generated by trust in a computer-based test, rather than from understanding of the technology and knowledge about its diagnostic accuracy [ 43 , 52 ]. It should be noted that, paradoxically, a blind use of diagnostics could represent a double-edged sword, if overconfidence in results became a substitute for clinical reasoning [ 75 ]. The burden of misdiagnosis was also supported by findings from a large autopsy study, demonstrating a high prevalence of TB among children and PLHIV that were missed at clinical diagnosis [ 76 ]. Evaluating the impact of testing on clinical decisions and empiric treatment [ 77 , 78 ] will be important as missing false negative patients contributes to TB morbidity and mortality, particularly among people who cannot expectorate or who have paucibacillary disease such as young children, where currently available assays have lower sensitivities [ 79 – 81 ]

Health system uncertainty emerged as an important driver of variability in TB decision-making. The unavailability or inaccessibility of diagnostic resources contributed to uncertainty in the decisional process and outcomes [ 25 , 52 , 55 , 56 ].

When diagnostic tests were available, several contextual factors, such as poor infrastructure and lack of administrative resources (infection prevention control policies, insufficient trainings), represented barriers to test adoption, shifting the decisional bar towards empiric treatment initiation, particularly in children or very sick patients, or leading to missed treatment opportunities [ 25 , 52 ]. The absence of locally tailored guidelines was reported to contribute to epistemic uncertainty and variability in clinical management [ 52 , 62 ]. These findings confirm that resource allocation strategies, as well as trainings and guidelines, need to be more inclusive of the lower tiers of the health system [ 82 ].

This study also found that providers highly valued the possibility to use non-sputum samples for testing, such as urine or stool [ 52 ], highlighting the need for a rapid addition of sputum-free diagnostics, particularly for paucibacillary cases and paediatric TB [ 83 ].

In recent years, there has been unprecedented development of novel TB diagnostic technologies. As new products come to market, policy makers must decide which available tools to implement. Findings from this review support the idea that such decisions should not exclusively account for diagnostic assay characteristics (e.g., accuracy), but also consider acceptability and feasibility of tests within the health care infrastructure. As suggested by meta-analyses reporting inconclusive findings regarding the impact of GeneXpert on treatment initiation decisions [ 44 , 64 ], it is key to understand the real-world impact of diagnostics through robust operational research at the point-of-care.

Additionally, the increasing utilization of multiple tests or different specimens in parallel, may exacerbate the challenges of results interpretation, particularly in children [ 84 ]. Understanding how clinicians manage conflicting results will be important to inform clinical algorithms.

Recently, significant progress has been made in the development and validation of clinical prediction models and algorithms to help standardize the decision-making process, particularly in contexts not yet reached by new diagnostic tools [ 85 ]. However, such tools rely on the assumption that a clinical consultation is a standardized event where relevant clinical variables or risk factors would always be disclosed and inform disease probability. Nonetheless, as suggested by the findings of this review, a clinical encounters is an event influenced by multiple uncertainties [ 39 , 53 , 55 , 56 ]. Hence, it will be important to collect data on real-life performance of such prediction models and algorithms, and to consider setting-specific adjustments and the integration of variables beyond patient clinical and risk factors. At the same time, the complex roots of uncertainty call for integrated efforts by policy makers, researchers, and programs to combine diagnostics research and implementation with staff trainings, guidelines implementation and uptake, infrastructure development, transversal health education to combat stigma and discrimination, and investments at the most peripheral levels of health care systems globally.

Strengths and limitations

To the best of our knowledge, this is the first study to conceptualize and summarize sources and types of uncertainty in TB decision-making. The umbrella review approach allowed us to triangulate findings from varied study designs and outcomes while preserving high methodological standards. The review was conducted in a systematic manner in accordance with standardized guidance. Nonetheless, some limitations must be mentioned. First, a limitation of the umbrella review approach is our inability to conduct a detailed assessment of primary studies. Consequently, the study relied on the methods and quality of included SRs, many of which were of moderate quality. Most reviews used a narrative synthesis approach, and only a few meta-analyses and one qualitative evidence synthesis reported on the quality of the evidence. Second, it was not possible to perform a meta-analysis of quantitative review findings due to the heterogeneous inclusion criteria and outcome definitions. Third, it is possible that some relevant sources were missed, as grey literature was not included. Finally, the assessment of each record was performed by a single reviewer only, which may yield a lower sensitivity.

This study summarized the complex network of factors associated with decisional and outcome uncertainty in medical decision-making in TB through a synthesis and thematic analysis of the systematic review literature. Different sources of uncertainty were found to influence provider choices around testing and treatment initiation, often resulting in diagnostic and treatment delays or missed diagnoses and treatment opportunities. Further, the application of a multi-level framework to classify uncertainty revealed the extent to which findings pertaining to different sources and types of uncertainty were intertwined. Gaps in TB diagnosis and treatment suggest the need to integrate evidence from studies that consider variations in healthcare systems and end-users’ attitudes, preferences, and experiences with interventions introducing new diagnostic tools. Such considerations are important to improve TB diagnosis and treatment and quality of patient care and to allow impactful introduction of novel diagnostic aids in clinical practice worldwide.

The figure summarizes multi-level (patient, provider, health systems, diagnostic tests) factors associated with TB clinical decision-making, identified through thematic content analysis of the SRs. The factors were classified as barriers or facilitators for testing or treatment decisions, and represented using the threshold model [ 10 ]. Several facilitators positively influenced providers’ decisions to test (lower testing threshold), including the presence of typical symptoms and patient history, providers’ personal attributes and experiences, workplace (public/urban facility), and available test characteristics. Barriers to testing were the presence of confounding/atypical symptoms, inadequate TB knowledge and staff training, fear of infection, lack of resources, and challenges of respiratory specimen collection. Empiric treatment decisions (treatment threshold) were facilitated by the presence of factors generally associated with an increased risk of severe disease or negative outcomes (young age, severe symptoms), unavailability or inaccessibility (e.g., because of costs) to diagnostic tests, and lack of confidence in tests with low sensitivity. Providers were inclined to withhold treatment decisions if facing with certain elements of patient history (e.g., unknown TB exposure), waiting for test results, and in the presence of negative test results (without considering the possibility of a low negative predictive value).

During the clinical encounter, the provider assesses the patient’s clinical variables (clinical uncertainty) to determine the disease probability and evaluate therapeutic benefit-harm trade-offs. Disease probability estimates depend on the provider’s knowledge and experience (epistemic uncertainty). Provider’s ability to conduct an informative, high-quality clinical assessment is also influenced by patient-provider relation and communication strategies (relational and knowledge exchange uncertainty) as well as by provider’s attitudes and beliefs (personal uncertainty). When a decision is made to test, the probability of disease is adjusted based on diagnostic test results (post-test probability). However, a negative test result may be insufficient to withhold therapy, considering the low sensitivity of currently available diagnostic tests and the potential benefit of empiric treatment (test uncertainty). Additionally, the provider may decide not to proceed with invasive specimen collection and testing because of individual risk assessments such as fear of infection (personal uncertainty). Thus, the characteristics of diagnostic tests can impact decision-making. Clinical decisions are further limited by healthcare setting constraints such as lack of skilled staff, poor infrastructure, and scarcity of diagnostic tools (health system uncertainty), and absence of local guidelines (epistemic uncertainty).

Supporting information

S1 appendix. search strategy (july 2023)..

The search strategy was refined and tested in the MEDLINE database, and then adapted to the other databases. To restrict the search to capture SRs, while simultaneously minimizing the capture of non-SR publications, search terms and MeSH specific to SR study designs (e.g. ‘SR’, ‘qualitative evidence synthesis’) were included. A search of grey literature was not conducted. Details on the search strategy across all databases are presented in S1 Appendix.

https://doi.org/10.1371/journal.pgph.0003429.s001

S1 Table. Eligibility criteria- PICOS framework.

SRs were selected on predefined inclusion and exclusion criteria guided by the Population, Intervention, Comparison, Outcome and Study design/setting (PICOS) framework. References were pre-emptively de-duplicated in Endnote. The selected references were imported in Covidence ( https://www.covidence.org/home ) and re-screened for duplicates. Full texts of all potentially eligible reviews were obtained. The full text of selected papers was then examined for inclusion in the UR, based on the predefined criteria. The reason for the exclusion of each article was documented in the software for transparency and auditing purposes.

https://doi.org/10.1371/journal.pgph.0003429.s002

S2 Table. Critical appraisal of methodological quality of included reviews.

JBI Critical Appraisal Checklist for Systematic Reviews and Research Syntheses.

https://doi.org/10.1371/journal.pgph.0003429.s003

S1 Fig. Heatmap showing pairwise calculation of the CCA.

CCA was interpreted in banded thresholds: values below 5% indicated slight citations overlap, between 6–10% indicated moderate overlap, between 11–15% indicated high overlap and values above 15% indicated very high overlap.

https://doi.org/10.1371/journal.pgph.0003429.s004

S1 Checklist. PRISMA 2020 checklist.

https://doi.org/10.1371/journal.pgph.0003429.s005

S2 Checklist. Research checklist, PRISMA 2020 checklist.

https://doi.org/10.1371/journal.pgph.0003429.s006

1. World Health Organization. WHO Global Tuberculosis Report 2023. Geneva; 2023. Available from: https://www.who.int/publications/i/item/9789240083851
View Article
PubMed/NCBI
Google Scholar
4. World Health Organization. WHO consolidated guidelines on tuberculosis. Module 3: Diagnosis ‐ Rapid diagnostics for tuberculosis detection 2021 update [Internet]. 2021 [cited 2022 Feb 1]. Available from: https://www.who.int/publications/i/item/9789240029415
22. WHO global lists of high burden countries for tuberculosis (TB), TB/HIV and multidrug/rifampicin-resistant TB (MDR/RR-TB), 2021–2025 Background document. 2021; Available from: https://iris.who.int/handle/10665/341980
30. Methodology for JBI Umbrella Reviews. 2014; Available from: https://ro.uow.edu.au/cgi/viewcontent.cgi?article=4367&context=smhpapers
31. Chapter V: Overviews of Reviews | Cochrane Training [Internet]. [cited 2022 Jul 7]. Available from: https://training.cochrane.org/handbook/current/chapter-v
73. World Health Organization. Tuberculosis in women. 2018. Availble from: https://www.who.int/publications/m/item/tuberculosis-in-women
74. StopTB Partnership. Gender and TB- investment package. Available from: https://stoptb.org/assets/documents/communities/TB%20Gender%20Investment%20Package.pdf
85. World Health Organization. WHO consolidated guidelines on tuberculosis Module 5: Management of tuberculosis in children and adolescents [Internet]. 2022 [cited 2024 May 27]. Available from: https://www.who.int/publications/i/item/9789240046764

Open access
Published: 31 July 2024

Ozone therapy in musculoskeletal medicine: a comprehensive review

Madhan Jeyaraman ORCID: orcid.org/0000-0002-9045-9493 1 ,
Naveen Jeyaraman ORCID: orcid.org/0000-0002-4362-3326 1 ,
Swaminathan Ramasubramanian ORCID: orcid.org/0000-0001-8845-8427 2 ,
Sangeetha Balaji ORCID: orcid.org/0000-0002-1566-1333 2 ,
Arulkumar Nallakumarasamy ORCID: orcid.org/0000-0002-2445-2883 3 ,
Bishnu Prasad Patro ORCID: orcid.org/0000-0001-9497-9624 4 &
Filippo Migliorini ORCID: orcid.org/0000-0001-7220-1221 5 , 6 , 7

European Journal of Medical Research volume 29 , Article number: 398 ( 2024 ) Cite this article

64 Accesses

1 Altmetric

Metrics details

Musculoskeletal disorders encompass a wide range of conditions that impact the bones, joints, muscles, and connective tissues within the body. Despite the ongoing debate on toxicity and administration, ozone demonstrated promise in managing several musculoskeletal disorders, modulating pain and inflammation. A literature search was conducted. The research design, methods, findings, and conclusions of the studies were then examined to evaluate the physiological effects, clinical application, controversies, and safety of the application of ozone in musculoskeletal medicine. Ozone application demonstrates considerable therapeutic applications in the management of musculoskeletal disorders, including fractures, osteoarthritis, and chronic pain syndromes. Despite these advantages, studies have raised concerns regarding its potential toxicity and emphasized the importance of adhering to stringent administration protocols to ensure safety. Additionally, heterogeneities in patient reactions and hazards from oxidizing agents were observed. Given its anti-inflammatory and analgesic qualities, ozone therapy holds potential in the management of several musculoskeletal disorders. Additional high-quality research with long follow-up is required to refine indications, efficacy and safety profile. Finally, for wider clinical acceptability and utilization, the development of international recommendations is essential.

Introduction

Ozone therapy is an innovative modality which gained growing attention in medical science, especially in musculoskeletal medicine [ 1 ]. Administered as a mixture of oxygen and ozone gases, it stimulates physiological responses, particularly anti-inflammatory and analgesic effects [ 2 , 3 ]. By doing so, ozone therapy is administered in various acute and chronic musculoskeletal disorders such as fractures [ 4 , 5 , 6 ], osteoarthritis (OA) [ 7 , 8 , 9 , 10 ], low back ache [ 11 , 12 ], osteomyelitis [ 13 , 14 , 15 , 16 ], and chronic pain syndromes [ 17 , 18 , 19 ]. Beyond orthopaedic applications, ozone therapy has shown potential in treating conditions such as chronic fatigue syndrome (CFS) and myalgic encephalomyelitis (ME) using different modalities such as ozonated water, ozone oil, and ozone gas [ 20 , 21 , 22 , 23 , 24 ]. Tailored routes of administration, ranging from enteral ozonated oil [ingestion], enteral ozonated water [ingestion], parenteral oxygen/ozone gas mixture [intramuscular, subcutaneous, intratonsillar, intrathecal, intraperitoneal, periganglionic, intraforaminal, paravertebral, intra-articular, intradiscal, oral submucous, supralaminar, epidural, penile, and intravenous], parenteral ozonated water [intra-tumoral], systemic [minor autohemotherapy (MiAH), major autohemotherapy (MAH), extracorporeal blood oxygenation and ozonation (EBOO), intravenous, and rectal insufflation], topical ozonated water [wound wash, ozonized balneotherapy, mouth wash, irrigation (nasal, sinus, otological, vaginal, intrauterine, intestinal, and intravesical), and sauna therapy, topical oxygen/ozone gas mixture [insufflation (ontological, intrafistula, intrauterine, and intravesical) and hyperbaric bagging], topical ozonated oil [embrocation and inhalation], and topical ozonated saline solution [wound wash, mouth wash, irrigation (saline, ontological, vaginal, intrauterine, and intravesical)], highlight its versatility [ 18 , 25 , 26 , 27 ]. In orthopaedic practice, intramuscular, subcutaneous, intrathecal, periganglionic, intraforaminal, paravertebral, intra-articular, intradiscal, oral submucous, supralaminar, epidural, intravenous routes are used. However, besides its therapeutic potential, concerns about ozone toxicity and the importance of adhering to safe administration protocols cannot be overlooked [ 21 , 25 , 26 , 28 ]. Prolonged inhalation of tropospheric ozone has been linked to detrimental impacts on the respiratory system and vital organs, inducing chronic oxidative stress and inflammation across multiple organs [ 29 ]. However, in controlled therapeutic settings, ozone therapy has demonstrated calculated oxidative stress induction, which activates therapeutic benefits without resulting in acute or chronic toxicity [ 29 ]. Caution is warranted to ensure that ozone doses do not surpass the blood’s antioxidant capacity to prevent potential toxicity [ 30 ]. While ozone therapy has the potential to enhance erythrocyte characteristics, correct hypoxia in diseases, and increase ATP levels through glycolysis activation, precise application is essential to avoid toxicity concerns [ 30 ]. The existing evidence is divided between its clinical benefits and potential risks as an oxidizing agent, further nuanced by variability in patient responses [ 28 ]. Despite its environmental implications as a pollutant, when administered under controlled conditions, ozone therapeutic potential in musculoskeletal conditions, including OA and herniated discs, is documented [ 20 , 25 , 26 ]. Therefore, ozone therapy emerges as a potent yet intricate intervention, necessitating further in-depth scientific scrutiny.

Numerous studies have explored ozone therapy, yet discrepancies persist, yielding variable results. Existing literature on ozone therapy reveals research gaps: limited comparative studies between treatments, unclear safety and effectiveness in various musculoskeletal conditions and insufficient understanding of anti-inflammatory effects [ 3 , 18 , 31 ]. Reviews frequently concentrate on specialized areas, prompting the necessity for a comprehensive review to address lacunae. In light of these controversies, this review critically evaluates the current evidence on the utilization of ozone in musculoskeletal disorders.

Effects of ozone

Ozone therapy, with its multifaceted therapeutic capacities, has demonstrated notable physiological effects across various medical arenas. Its primary mechanism centres on enhancing tissue oxygenation, an indispensable component of cellular metabolism, by augmenting oxygen delivery to hypoxic tissues, thereby fostering optimal conditions for metabolic and repair processes [ 20 , 26 , 32 ]. For systemic administration of ozone, such as in major autohemotherapy (MAH), the concentration range should fall within 10–40 μg/mL, with 10–20 μg/mL of blood considered biologically relevant. It is advisable to administer a total ozone amount of 500–1000 μg per 50 mL of blood during MAH treatment. This induces the production of pharmacologically active hydroxy hydroperoxides, referred to as “ozone peroxides” [ 15 , 25 ]. These compounds play a pivotal role in modulating endogenous antioxidant systems, particularly through interactions with the glutathione system [ 15 ]. A salient feature of ozone therapy is its proficiency in managing oxidative stress (Fig. 1 ), amplifying cellular antioxidant mechanisms while tempering the formation of reactive oxygen species (ROS) [ 3 , 33 ]. Specifically, ozone engages with cysteine residues and glutathione, catalysing signal processes that support the bioregulation of enzymatic antioxidants [ 15 ]. This aptitude for oxidative stress modulation is important in conditions marked by persistent inflammation and immune system anomalies [ 3 , 17 ]. Notably, ozone moderates inflammation by curtailing pro-inflammatory cytokines, such as interleukin 6 (IL-6), and simultaneously promotes anabolic activity, as evident by elevated insulin-like growth factor 1 (IGF-1) levels [ 33 , 34 ]. In addition, ozone therapy exerts a regulatory effect on key transcription factors: it activates Nrf2, endorsing antioxidant and anti-inflammatory responses, while concurrently inhibiting NF-kB, integral in inflammation regulation [ 18 , 33 , 35 , 36 ]. The therapy also modulates the synthesis of prostaglandins, impacts bradykinin release [ 26 ], and optimizes immune functions, particularly enhancing the activity of macrophages pivotal to immune reactions [ 3 , 17 ]. From a tissue repair and angiogenesis standpoint, ozone therapy stimulates the expression of proteins such as collagen I, α-SMA, and TGF-β1, facilitating fibroblast activity [ 22 ]. It also upregulates growth factors, including vascular endothelial growth factor (VEGF) and transforming growth factor-beta (TGF-β), fostering enhanced tissue regeneration [ 23 , 33 ]. Moreover, the activation of cellular signalling pathways like the PI3K/Akt/mTOR axis underscores its role in promoting the epithelial–mesenchymal transition (EMT) process [ 22 , 37 ].

Physiological effects of ozone

Clinically, ozone therapy has been effective in pain management, addressing conditions such as low back pain [ 18 ] and rotator cuff calcific tendinitis [ 38 ] (Fig. 2 ). Its influence extends to ameliorating microcirculatory disturbances seen in bone necrosis [ 1 ] and positively adjusting serum physiological markers, including total antioxidant capacity and lactate dehydrogenase [ 14 ], alongside inflammatory markers like the erythrocyte sedimentation rate [ 13 ]. The nuanced mechanisms of action underlying ozone therapy, ranging from improved oxygenation and oxidative stress modulation to anti-inflammatory effects, immune system enhancements, and tissue repair, validate its rising interest as a potential adjunctive treatment for several medical conditions. Its multifaceted physiological interactions render it an asset in contemporary clinical practice.

Applications of metamizole in musculoskeletal medicine

Clinical applications of ozone in orthopaedics

Fracture non-union posed challenges in orthopaedic medicine, prompting the exploration of novel approaches [ 39 ]. In this context, ozone therapy garnered attention for its potential to enhance bone healing. Experimental studies using animal models, especially those with critical size defects, have demonstrated its efficacy. Duman et al. documented improved aspects of bone healing in a rat femoral fracture model with ozone therapy, noting enhanced union, bone formation, and bone marrow organization [ 40 ]. These findings align with those of Irban et al., who observed increased periosteal thickness and trabecular formation, thereby solidifying ozone therapy prospective role in bone regeneration [ 41 ]. These preclinical insights are bolstered by clinical evidence underscoring the therapy benefits. Shah et al. detailed a case wherein ozone therapy expedited wound healing and infection resolution over a tibial area, resulting in the patient regained ability to walk within 20 months [ 42 ]. Beyond fracture healing, ozone therapy has demonstrated effectiveness in pain management. Sconza et al. identified a comparable efficacy between ozone therapy and hyaluronic acid injections for knee OA pain alleviation and functional improvement [ 8 ]. Additionally, in comparison with low-level laser therapy, ozone therapy evidenced superior performance in promoting bone formation in rat calvarial defects [ 5 ]. Irban et al. conducted a comprehensive examination revealing ozone therapy profound impact on bone healing. Beyond the observed increase in periosteal thickness and trabecular areas, there was a marked elevation in vital growth factors and proteins, such as VEGF, β-catenin, and TGF-β, in the ozone-administered group versus the control [ 41 ]. Nevertheless, while the existing evidence is promising, it is crucial to address the current methodological limitations. Bennett et al. systematic review highlighted concerns regarding the quality of studies on the prevention or treatment of fracture non-union, pinpointing issues like lack of method standardization, inconsistent outcome measures, and inadequate reporting, which currently hampers the feasibility of meta-analyses [ 4 ].

Osteoarthritis

The advent of ozone therapy modalities, specifically O 2 –O 3 therapy and oxygen–ozone therapy (OOT), has significantly enriched the therapeutic spectrum for knee OA. These interventions, recognized for their ability to modulate oxidative stress and suppress pro-inflammatory cytokines, offer pain relief and functional enhancement in patients with knee OA [ 3 , 43 , 44 ]. Ozone therapy yields outcomes in pain, stiffness, and overall function comparable to the traditionally favoured hyaluronic acid injections [ 8 ]. Fernandez et al. observed its effectiveness through intra-articular injections for knee OA, resulting in symptomatic relief and improved physical function [ 33 ]. Comparative trials, such as those juxtaposing the efficacy of ozone prolotherapy and intra-articular hypertonic saline, found both modalities to be therapeutically analogous [ 45 ]. Periodic intra-articular ozone injections have been advocated for their consistent pain amelioration, joint function enhancement, and overall quality of life improvement, alongside a commendable safety profile [ 10 ]. These favourable findings merit a balanced perspective. A systematic review of randomized controlled trials (RCTs) pinpointed the generally subpar methodological quality in the prevailing research on this topic [ 4 ]. A previous review on OOT for knee OA reinforced this finding, suggesting that while short to medium-term results were encouraging, the included RCTs exhibited discernible methodological constraints [ 8 ]. When juxtaposing ozone therapy with hyaluronic acid injections, both modalities manifested significant advancements in pain, stiffness, and function across various time points, including 1, 3, and 6 months of follow-ups [ 46 ]. The anti-inflammatory and anabolic attributes of ozone therapy have garnered notable attention. Fernandez-Cuadros et al. documented reductions in inflammatory markers such as interleukin-6 (IL-6), C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), and uric acid, coupled with pain alleviation and functional improvements spanning all grades of knee OA [ 33 ]. Farpour et al. findings echoed this, revealing that both ozone prolotherapy and intra-articular hypertonic saline injections offered parallel therapeutic benefits [ 45 ]. Moreover, a specific RCT highlighted the potential of weekly ozone injections over 8 weeks, emphasizing discernible pain reduction, improved joint functionality, and enhanced life quality [ 10 ]. While the prevailing literature accentuates the prospective benefits of ozone therapy in knee OA management, it simultaneously underscores methodological inadequacies in the current research landscape. To unequivocally establish ozone therapy standing as an efficacious knee OA treatment, there is an undeniable need for comprehensive, multicentre RCTs with extended follow-ups to corroborate its long-term benefits and reliability.

Osteonecrosis of the femoral head

Ozone therapy has emerged as a promising alternative in treating osteonecrosis of the femoral head, especially for patients unresponsive to traditional therapeutic approaches. The therapy effectiveness is rooted in its capacity to promote neovascularization and immunoregulation, both pivotal for repairing hypoxic-ischaemic necrosis in the femoral head [ 1 , 47 ]. Elucidating the molecular underpinnings of this treatment, An et al. highlighted the influence of differentially expressed genes, suggesting that individualized ozone therapy regimens, tailored to specific molecular markers, could be developed [ 1 ]. Complementary findings by An et al. showcased the augmented benefits of conjoining intra-articular O 2 –O 3 hip injections with O 3 -AHT (autohemotherapy). Such a combination not only ameliorated enduring pain, but also aided in mitigating bone marrow oedema across diverse stages of osteonecrosis of the femoral head. Importantly, differential clinical outcomes between patients responsive and non-responsive to ozone therapy underscored the significance of immunoregulatory pathways, particularly spotlighting the impact of mitotically active lymphocytes in modulating treatment efficacy [ 1 ]. Branching out from osteonecrosis of the femoral head, the therapeutic scope of ozone therapy has been assessed in other conditions like avascular bisphosphonate-induced jaw osteonecrosis [ 48 , 49 , 50 , 51 ]. Agrillo et al. indicated that over half of the participants experienced full lesion healing and symptom abatement, with another 30% witnessing substantial lesion diminution and symptom relief. Even though lesions persisted relatively unchanged in 16% of the patients, they still reported some alleviation in symptoms, suggesting that ozone therapy might be an effective alternative to more invasive interventions, such as bone ablation surgery, particularly for bisphosphonate-induced related jaw osteonecrosis cases [ 51 ].

Osteomyelitis

Ozone therapy potential as a treatment for osteomyelitis, a challenging bone infection often unyielding to conventional modalities, has garnered increasing attention in both clinical and preclinical domains. Yasheng et al. demonstrated the effectiveness of a treatment approach combining ozonated water lavage, physiological saline irrigation, and vacuum-sealed drainage for chronic osteomyelitis. This regimen yielded significant bacterial clearance and stimulated granulation tissue formation, attesting to ozone therapy potential as a reliable intervention [ 16 ]. Further, Shetty et al. highlighted its adaptability in specialized cases, showcasing its efficacy in neonatal mandibular osteomyelitis when combined with intravenous vancomycin antibiotics. The route of ozone administration involves the local application of ozonated water to the lesional site. The ozonated water is prepared by infusing three cycles of ozonated gas into 100 mL of saline over a 24-h p24 hours [ 52 ]. Animal studies bolster these clinical observations. Bilge et al., using a rat osteomyelitis model, noted improved histopathological parameters, suggesting ozone therapy has potent anti-inflammatory attributes [ 14 ]. Focusing on inflammatory markers, Nabi et al. reported the therapy positive impact on indices like the erythrocyte sedimentation rate, documenting a commendable 86.66% recovery in a patient cohort [ 13 ]. This line of exploration aligns with Shah et al. emphasis on ozone inherent disinfectant and antibacterial properties, positioning it as a beneficial option for extensive orthopaedic conditions, including osteomyelitis [ 42 ]. Furthermore, ozone therapy role extends to conditions such as bisphosphonate-related jaw osteonecrosis. Agrillo et al. evidenced complete healing in a significant patient proportion and substantial symptom relief in others [ 51 ]. An RCT delving into ozone therapy for chronic osteomyelitis showed the treatment group exhibiting marginally improved recovery rates and inflammatory indices, without notable adverse events, although the observed difference was not statistically significant [ 13 ].

Adhesive capsulitis

Recent randomized controlled trials have sought to discern the optimal treatment strategy for adhesive capsulitis of the shoulder. Foula et al. compared the therapeutic merits of ultrasound-guided intra-articular injections, specifically contrasting ozone, steroid, and pulsed radiofrequency. The equipment utilized in the study employed platelet-rich fibrin (PRF) generators, which emit oscillating pulses spanning a frequency range of 420–500 kHz. These pulses have an amplitude of 45 V and a duration of 20 ms, followed by a silent period lasting 480 ms. Their results indicated the superior long-term efficacy of pulsed radiofrequency in symptom alleviation compared to its counterparts, ozone and steroid injections [ 53 ]. Another study echoed the benefits of these ultrasound-guided intra-articular injections in treating shoulder adhesive capsulitis. It documented marked improvements in pain, disability, range of motion (ROM), and inflammation across all intervention groups. Notably, visual analogue scale (VAS) scores during movement exhibited significant enhancements from the second week onward, persisting through the fourth and eighth weeks for all interventions. The steroid group, in particular, showed early improvements with VAS scores at rest enhancing from the initial week. ROM and the Shoulder Pain and Disability Index (SPADI) scores also reported progress from the second week. Moreover, consistent reductions in inflammatory markers, intercellular adhesion molecule-1 (ICAM-1), and high-sensitivity C-reactive protein (hs-CRP), were observed across the groups [ 53 ]. While both studies validate the efficacy of ultrasound-guided intra-articular injections in managing adhesive capsulitis symptoms, Foula et al. findings distinctly spotlight pulsed radiofrequency advantage for sustained symptom relief [ 53 ]. Concurrently, the subsequent study provides a more holistic perspective, underscoring the significant short-term benefits of both ozone and steroid injections, in addition to pulsed radiofrequency. Thus, pulsed radiofrequency stands out for its long-term therapeutic potential. Still, all modalities, including ozone and steroids, remain significant contenders in the short-term therapeutic landscape for adhesive capsulitis of the shoulder.

Tendinopathies

Ozone therapy is progressively gaining prominence in managing various musculoskeletal conditions, especially tendinopathies [ 20 , 38 , 54 , 55 , 56 ]. Hidalgo-Tallon et al., spotlighted the therapy efficacy in treating subacromial tendinopathy, illustrating its advantage over mesotherapy and steroid injections [ 2 ]. Complementary research by Dong et al. [ 38 ] and Atar et al. [ 56 ] further substantiated these findings, suggesting ozone therapy potential in managing conditions like rotator cuff calcific tendinitis and chronic supraspinatus tendinopathy. These studies suggest that ozone injections are comparable to corticosteroid injections in alleviating symptoms. Further broadening the scope of ozone therapy, the modality matches corticosteroid injections in delivering significant relief from chronic plantar fasciitis [ 55 ]. This not only accentuates ozone therapy versatility, but also positions it as a potent alternative to established treatments. In the realm of sports medicine, the ozone therapy utility is becoming apparent. Hidalgo-Tallon et al. showcased ozone therapy benefits in managing hamstring injuries among professional athletes, noting pronounced pain reduction and functional improvements [ 57 ]. Seyam et al. highlighted positive outcomes from ultrasound-guided ozone therapy for partial supraspinatus tendon tears [ 20 ]. Furthermore, with an emphasis on compassionate care and treatment-resistant cases, Hidalgo-Tallon et al. proposed ozone therapy as an invaluable tool [ 2 ]. While ongoing research is geared towards elucidating ozone therapy comprehensive efficacy relative to corticosteroid injections in treating rotator cuff calcific tendinitis, definitive outcomes remain anticipated [ 38 ]. Additionally, animal studies such as the one by Gurger et al. spotlight the augmented effect of combining ozone therapy with platelet-rich plasma (PRP) on tendon-to-bone healing. This particular study revealed the duo potential in enhancing biomechanical strength, collagen fibre continuity, and alignment, thus hinting at the possibility of using ozonized PRP as a biological catalyst in tendon healing processes [ 58 ]. Lastly, Atar et al. randomized controlled trial buttressed the comparable therapeutic potency of ozone and corticosteroid injections for chronic supraspinatus tendinopathy in terms of pain relief, life quality, and functionality, though emphasizing the need for extended studies to gauge long-term outcomes [ 56 ].

Rheumatic pathologies

Ozone therapy has emerged as a noteworthy intervention in the realm of rheumatic diseases, driven primarily by its capacities to modulate oxidative stress and attenuate pro-inflammatory cytokines [ 3 ]. Rheumatoid arthritis stands as a significant context where ozone therapy is increasingly being validated for methotrexate (MTX) efficacy for clinical response and improves cellular redox balance [ 59 , 60 , 61 ]. In animal models induced by Freund complete adjuvant and another experimental model, studies by Bozbas et al. and Zhao et al., respectively, observed marked attenuation of symptoms, histopathological signs of inflammation, and reductions in pro-inflammatory cytokines [ 62 , 63 ]. Furthermore, Patel et al. postulate the potential dual application of ozone therapy, both as a primary treatment and synergistically alongside stem cell therapy or natural medicines [ 64 ]. Complementing its standalone utility, ozone therapy demonstrates promise as an adjunctive approach in conventional treatment protocols for rheumatoid arthritis. Fernandez et al. explored its potential combined with methotrexate (MTX), finding enhanced therapeutic outcomes of MTX in rheumatoid arthritis [ 61 ]. This combined approach not only mitigated disease activity, but also reinforced the antioxidant system, accentuating ozone therapeutic complementarity [ 61 ]. Beyond rheumatoid arthritis, the application spectrum of ozone therapy expands to other rheumatic conditions. Studies by Seyam et al. and Tartari et al. also spotlighted ozone therapy role in modulating oxidative damage in systemic sclerosis and its potential applicability in systemic autoimmune rheumatic diseases via cytokine profile adjustments [ 20 , 60 ]. However, nuances exist in the broader application of ozone. Zhao et al. study pointed to a correlation between fine particulate matter (PM2.5) exposure and systemic autoimmune rheumatic diseases, yet did not firmly link ozone exposure with the onset of these diseases [ 63 ]. While another study affirmed ozone therapy benefits in reducing inflammation and arthritis severity in an animal model with rheumatoid arthritis, the alterations in oxidative stress markers remained statistically inconclusive [ 62 ].

Chronic pain syndromes

Ozone therapy has solidified its position as an effective therapeutic modality in managing an array of chronic pain conditions. Particularly, intramuscular injections of the O 2 –O 3 mixture have shown significant efficacy in alleviating cervicobrachial pain, with discernible reductions in VAS pain scores [ 65 , 66 ]. Parallel to this, intramuscular paravertebral lumbar injections of the mixture have proven fruitful in addressing low back pain, as reflected by reductions in VAS scores and enhancements in the Oswestry Disability Index (ODI) scores [ 11 ]. This is further substantiated by studies by Andrade et al. and Biazzo et al., which emphasize the minimally invasive and effective nature of ozone therapy in lumbar pain management [ 11 , 12 ]. A meta-analysis considering ozone therapy role in lumbar pain underscores its therapeutic potential but also signals the necessity for caution due to the high or uncertain risk of bias in some of the assessed trials [ 12 ]. Beyond specific pain syndromes, ozone therapy exhibits broader mechanisms, such as oxygenation, immune modulation, and anti-inflammatory action, and has been applied to other chronic conditions like ME/CFS and CFS [ 17 , 67 , 68 , 69 ]. Furthermore, its efficacy has been observed to be comparable, if not superior, to standard treatments in Pain Units, especially when conventional therapies fall short [ 2 ]. The treatment adaptability was further highlighted in a case report detailing its successful application in an 11-year-old with Complex Regional Pain Syndrome and pseudo-seizures, marking a full remission post-intervention [ 17 ]. Delving into synergistic applications, Patel presented the intriguing prospect of combining ozone therapy with stem cell interventions, particularly for rheumatoid arthritis. However, this proposal is nascent and requires in-depth exploration for both safety and effectiveness [ 64 ]. While ozone therapy therapeutic potential is increasingly acknowledged, there exists a clear mandate for more robust scientific investigation. Tartari et al. emphasize the need for clarity in indications, protocol optimization, and pinpointing the patient demographics most likely to benefit [ 60 ]. Thus, despite the growing advocacy for ozone therapy as a versatile and efficient treatment modality in chronic pain management, its broad clinical application demands further empirical rigour.

Fibromyalgia

Ozone therapy has emerged as a viable therapeutic option in addressing chronic conditions marked by fatigue and musculoskeletal pain, particularly myalgic encephalomyelitis (ME)/chronic fatigue syndrome (CFS) [ 67 , 68 , 69 , 70 ] and fibromyalgia [ 71 , 72 , 73 , 74 ]. In patients with ME/CFS, oxygen–ozone autohemotherapy (O 2 –O 3 -AHT) has been linked to significant clinical advantages. After O 2 –O 3 -AHT, approximately 43.5% of participants experienced a marked reduction in fatigue symptoms, with these benefits persisting for at least 3 months across diverse demographics [ 68 ]. The therapy impact on ME/CFS extends beyond symptom alleviation, showing potential in modulating metabolic pathways, oxidative stress, antioxidant systems, and immune and inflammatory responses [ 67 ]. Simultaneously, for fibromyalgia, a condition which shares many symptomatic parallels with ME/CFS, rectal insufflation ozone therapy has shown significant therapeutic potential [ 66 ]. Hidalgo-Tallon et al. observed substantial improvements in fibromyalgia patients’ physical symptoms, as reflected by decreased Fibromyalgia Impact Questionnaire scores within the initial 4 weeks of intervention [ 66 ]. In addition to physical improvements, the therapy also positively influenced psychological parameters, evident from reduced depression scores and enhanced Physical Summary Scores on the SF-12 questionnaire [ 66 ]. Subsequent research has affirmed these findings, with notable reductions in fatigue and psychological distress in fibromyalgia patients post-ozone therapy [ 25 , 41 ]. While transient meteorism was occasionally reported, the side effects did not negate the overall beneficial outcomes of the therapy [ 66 ]. Furthermore, in a distinct study focusing on CFS, an impressive 70% of patients reported significant symptom alleviation, with the treatment presenting no major adverse effects [ 67 ]. Collectively, ozone therapy, whether through O 2 –O 3 -AHT or rectal insufflation, provides a robust therapeutic approach to the comprehensive challenges presented by ME/CFS and fibromyalgia. The therapy multifaceted mechanisms of action, coupled with its evident safety profile [ 67 , 68 , 69 ], advocate for its consideration, especially for patients who have found limited relief with other treatments [ 69 ].

Miscellaneous

Ozone therapy has been identified as a versatile therapeutic modality, demonstrating efficacy across diverse medical conditions. For avascular bisphosphonate-related jaw osteonecrosis, approximately 54% of patients achieved complete lesion healing following an average treatment duration of 6.5 months, suggesting its potential in managing such conditions [ 51 ]. Furthermore, ozone therapy has demonstrated longer-lasting symptom relief for chronic plantar fasciitis compared to corticosteroids [ 55 ].

In orthopaedics, ozone therapy has been examined for its benefits in a range of conditions, from temporomandibular joint disorders to low back pain and carpal tunnel syndrome [ 8 , 11 , 15 ]. Its application has also been extended to chronic wound and ulcer management. Topical and injected ozone treatments have enhanced wound healing by reducing inflammation, facilitating rapid wound closure, and promoting angiogenesis and fibroblast proliferation, as evident from animal studies [ 22 , 23 ]. Case studies and systematic reviews, such as those by Fitzpatrick et al. and Romary et al., highlight the efficacy of ozone therapy in promoting wound healing, especially in chronic wound scenarios [ 24 , 75 ]. Particularly for diabetic foot ulcers, ozone therapy has been credited for reducing wound size and amputation rates [ 76 , 77 , 78 , 79 , 80 ].

The mechanistic attributes of ozone therapy in tissue repair have also been explored. Studies highlight its role in promoting fibroblast migration, stimulating epithelial–mesenchymal transition via the PI3K/Akt/mTOR pathway, and augmenting angiogenesis [ 22 , 23 ]. A notable case study indicated the potential of combined ozone therapy in promoting the healing of extensive tibial wounds, enhancing the patient mobility [ 42 ]. In sports medicine, ozone therapy has proven valuable in managing injuries such as those in professional athletes with hamstring injuries, resulting in diminished pain, improved mobility, and enhanced perfusion [ 57 , 77 , 78 ]. Its safety and effectiveness are also corroborated by systematic reviews in pain medicine for conditions like knee OA [ 2 ]. The significant research findings and merits and de-merits of ozone therapy for various musculoskeletal conditions are jotted in Tables 1 and 2 , respectively.

Complications of ozone therapy

Ozone therapy, increasingly recognized in the scientific literature for its therapeutic potential, has demonstrated a commendable safety profile across various clinical applications when administered following established guidelines and employing an atoxic system. Clinical trials predominantly document mild and self-limiting adverse effects such as abdominal distension, lower limb hypoesthesia, and transient pain exacerbation, which typically resolve without necessitating extensive medical intervention [ 4 , 21 , 25 , 26 , 66 ]. Specifically, within the realm of musculoskeletal disorders, ozone therapy, in conditions like knee OA, has showcased a safety profile commensurate with hyaluronic acid injections, with both modalities yielding only mild, transient adverse events [ 33 , 46 ]. This safety spectrum further extends to osteomyelitis, as evidenced in a rat model study, wherein ozone therapy augmented antioxidant mechanisms devoid of adverse reactions [ 14 ]. In treating chronic wounds and ulcers, studies have reiterated the minimal risk profile of ozone therapy, emphasizing its efficacy in wound healing [ 80 ]. Nonetheless, potential complications arising from specific administration methods, such as the use of polyvinyl chloride (PVC) auto-transfusion bags contaminated with excessive citrate–phosphate–dextrose (CPD), warrant vigilance [ 21 ]. Puncture accidents during therapy and risks in populations like the elderly or those with decompensated conditions merit careful consideration [ 10 , 12 ].

Contraindications to ozone therapy are unambiguously defined, encompassing conditions like latent hypoglycaemia, hyperthyroidism, favism (due to G-6PD deficiency), pregnancy, and sickle cell anaemia. The use of angiotensin-converting enzyme (ACE) inhibitors also mandates caution with ozone therapy [ 20 , 21 ]. These contraindications are rooted in potential risks such as the prospect of hemolysis in G-6PD deficiency, mutagenic concerns during early pregnancy, and asthmatic hypersensitivity.

Prolonged exposure to ozone may adversely affect the respiratory system, fostering the release of deleterious compounds into the bloodstream and potentially resulting in multiorgan damage [ 81 ]. Cells with deficient antioxidant activity are especially vulnerable to mutagenic alterations with sustained ozone exposure [ 82 ]. High concentrations of ozone can induce DNA oxidation and exhibit genotoxic effects [ 83 ]. Within the pulmonary milieu, ozone interaction with unsaturated fatty acids can yield lipid ozonation products, engendering lipid peroxidation, perturbed membrane permeability, and subsequent activation of inflammatory mediators [ 84 ]. When ozone commingles with nitrogen dioxide (NO 2 ), the ensuing photochemical smog can intensify detrimental effects. Fortunately, antioxidants like vitamins E and C, along with anti-inflammatory agents like indomethacin, can serve as protective countermeasures against such adversities [ 20 , 21 ].

Limitations of ozone therapy

Ozone therapy, emerging as a promising therapeutic modality across diverse medical conditions, including osteonecrosis of the femoral head (ONFH) and orthopaedic disorders, has garnered considerable attention. Despite promising clinical observations [ 1 , 5 , 20 , 28 , 51 , 69 ], the present study has methodological challenges which negatively impact the robustness of the findings. Notably, the prevalence of studies with small sample sizes underscores the imperative for investigations with expansive patient cohorts and prolonged follow-ups to affirm these preliminary insights [ 20 , 46 , 69 , 80 ]. The methodological quality of much extant research tempers the interpretability of results. A significant portion of these studies exhibit a high or uncertain risk of bias [ 8 , 12 ], highlighting the exigency for methodologically rigorous investigations to validate the therapeutic potential of ozone therapy [ 2 ]. A marked inconsistency in treatment protocols, as underscored by Fernandez-Cuadros et al. regarding knee OA [ 33 ], underlines the need for standardized approaches. Such consistency can lay the foundation for evidence-based guidelines, fostering the safe and efficacious deployment of ozone therapy in clinical contexts. A paramount lacuna in our current comprehension pertains to the elucidation of the exact biological and physiological mechanisms underpinning ozone therapy effects. This knowledge gap is accentuated in conditions like ONFH, where hypotheses around ozone role in neovascularization and immunoregulation remain speculative [ 1 ]. Sire et al. further highlight the necessity to broaden the research spectrum to encompass additional musculoskeletal disorders, enriching our understanding of ozone therapy applications in orthopaedics [ 18 ]. Deciphering these mechanisms is pivotal to refine and optimize treatment protocols. While the promise of ozone therapy in contexts such as wound healing in diabetic foot ulcers is evident [ 80 , 85 ], the call for high-quality randomized controlled trials (RCTs) reverberates, especially when venturing beyond specific wound types. This is emblematic of the overarching need for comprehensive, methodologically sound trials which shed light on ozone therapy influence on inflammatory mediators and its broader clinical implications [ 3 ]. Despite ozone therapy prospective benefits, it encounters significant barriers to its mainstream medical integration, most prominently the absence of FDA approval and a paucity of evidence corroborating its expansive efficacy [ 17 , 86 , 87 ]. Overcoming these challenges demands the undertaking of well-constructed research studies, adhering to rigorous scientific paradigms. By redressing these research inadequacies, the scientific community can proffer compelling evidence which vindicates the place of ozone therapy in contemporary healthcare, emphasizing its efficacy and safety vis-à-vis conventional treatments [ 24 , 75 , 80 ].

Level of evidence of ozone in orthopaedics

In this paper, the cited evidence is classified according to the Oxford Centre for Evidence-based Medicine hierarchy. Foremost, Level 2 evidence, encompassing randomized controlled trials (RCTs) and systematic reviews with meta-analyses, offers the most robust validation. This calibre of evidence prominently features in discussions on osteoarthritis, osteomyelitis, tendinopathies, rheumatic pathologies, and chronic pain syndromes. The methodological rigour of Level 2 studies engenders heightened confidence in the outcomes they present as mentioned in Table 3 . Subsequently, Level 3 evidence, characterized by non-randomized controlled trials and case–control studies, emerges in contexts such as avascular necrosis, osteomyelitis, adhesive capsulitis, chronic pain syndromes, and fibromyalgia. Though not mirroring the exacting standards of Level 2 research, these Level 3 studies furnish indispensable foundational knowledge, potentially guiding subsequent, more rigorous investigations.

Lastly, a solitary study, a systematic review assessing preclinical therapies for fracture non-union, is identified as Level 4 evidence. While systematic reviews are typically accorded a higher evidential standing, the specific methodological approach of this review positions it within the Level 4 category. In summary, the literature predominantly aligns with Level 2 and Level 3 evidence, reflecting a considerable degree of scientific rigour. The limited representation of Level 4 evidence, confined to a single systematic review of fractures, accentuates the depth of the findings while simultaneously highlighting domains warranting further high-level research to corroborate assertions across diverse medical contexts.

Future prospective

The advancement of ozone therapy scientific foundation in orthopaedics demands stringent, meticulously designed randomized controlled trials with expanded sample sizes and prolonged follow-up periods [ 4 , 8 , 11 , 15 , 22 ]. Such studies should prioritize consistent, objective outcome measures to elevate their methodological quality, enabling a clearer comparison of various treatments [ 4 ]. It is essential that research impartially delineates the pros and cons of ozone therapy. A focal area of inquiry should be the interplay between constant oxidative stress and the episodic acute stress invoked by ozone treatments, a realm presently marked by uncertainty [ 40 ]. Deepening our comprehension of the molecular underpinnings guiding therapeutic responses is pivotal, not only to academic discourse but also in refining and personalizing treatment regimens for specific orthopaedic afflictions [ 1 , 28 , 65 , 85 ]. For the therapy to achieve broader clinical acceptance, the establishment of standardized guidelines is paramount. These guidelines would encapsulate the best practices for ozone administration, thereby ensuring uniformity in therapeutic approaches and the consequent predictability of clinical outcomes [ 66 ]. The identification of differentially expressed genes between positive responders and non-responders to ozone therapy can offer insights into predicting therapeutic outcomes, facilitating a more personalized therapeutic strategy [ 1 ]. Research endeavours should encompass a detailed exploration of optimal dosages, session frequencies, and treatment durations, ensuring maximized therapeutic potential across varied orthopaedic conditions [ 3 , 66 ]. Furthermore, expansive multicentre prospective studies can enhance the validation of ozone therapy’s therapeutic and safety profiles, extending its reach to broader patient demographics with a multitude of clinical conditions [ 1 , 52 , 88 , 89 , 90 , 91 , 92 ]. Emphasis should also be placed on examining the specific impacts of topical ozone treatments, such as ozonated water and oils, which currently lack comprehensive evaluation in human studies [ 75 ]. The fortification of ozone therapy’s role in orthopaedics necessitates a multifaceted research approach. This encompasses the execution of rigorous trials, a deeper investigation into molecular mechanisms, and the formulation of standardized treatment protocols. Through such integrated efforts, the orthopaedic community can establish a solid evidence base, ensuring the effective and reliable clinical integration of ozone therapy.

Ozone therapy, with its multifaceted potential, is steadily gaining prominence across diverse clinical domains, notably within orthopaedics. At the physiological level, the therapy exhibits attributes crucial for orthopaedic interventions, including enhanced tissue oxygenation, modulation of oxidative stress, and anti-inflammatory properties. Such mechanisms provide potential relief in conditions such as knee OA, chronic osteomyelitis, and various chronic pain syndromes, positioning ozone therapy as a valuable adjunct or alternative when conventional treatments fall short. While the merits are evident, a comprehensive endorsement of ozone therapy in mainstream clinical practice awaits a more robust body of evidence. The imperative lies in rigorous research endeavours, particularly well-executed randomized controlled trials, to address current methodological limitations. Comprehensive studies must also delve into the precise mechanisms underpinning ozone therapy therapeutic impacts. Alongside this, the establishment of standardized treatment guidelines will be vital for its broader clinical acceptance and application. Until consolidated evidence emerges, clinicians should approach ozone therapy with circumspection, calibrating its use based on individual patient needs and clinical contexts. In essence, while ozone therapy offers considerable promise for a myriad of orthopaedic conditions, the onus is on the scientific community to further elucidate its efficacy, refine its application, and ascertain its safety profile.

Availability of data and materials

The datasets generated during and/or analysed during the current study are available throughout the manuscript.

An J-X, Wu G-P, Niu K, Wei Y-P, Liu H, Gao X-Y, et al. Treatment of femoral head osteonecrosis with ozone therapy: pilot trial of a new therapeutic approach. Pain Physician. 2022;25:E43–54.

PubMed Google Scholar

Hidalgo-Tallón FJ, Torres-Morera LM, Baeza-Noci J, Carrillo-Izquierdo MD, Pinto-Bonilla R. Updated review on ozone therapy in pain medicine. Front Physiol. 2022;13: 840623.

Article PubMed PubMed Central Google Scholar

de Sire A, Marotta N, Ferrillo M, Agostini F, Sconza C, Lippi L, et al. Oxygen–ozone therapy for reducing pro-inflammatory cytokines serum levels in musculoskeletal and temporomandibular disorders: a comprehensive review. Int J Mol Sci. 2022;23:2528.

Bennett PM, Stewart SK, Dretzke J, Bem D, Penn-Barwell JG. Preclinical therapies to prevent or treat fracture non-union: a systematic review. PLoS ONE. 2018;13: e0201077.

Kazancioglu HO, Ezirganli S, Aydin MS. Effects of laser and ozone therapies on bone healing in the calvarial defects. J Craniofac Surg. 2013;24:2141–6.

Article PubMed Google Scholar

Irban A, Uslu S, Gereli A, Aydinlar EI, Karyemez PE, Luleci N, et al. The effect of ozone therapy on experimental bone fracture healing in rats [abstract]. J Ozone Ther. 2019;3:29.

Article Google Scholar

Feng X, Beiping L. Therapeutic efficacy of ozone injection into the knee for the osteoarthritis patient along with oral celecoxib and glucosamine. J Clin Diagn Res. 2017;11:UC01–3.

CAS PubMed PubMed Central Google Scholar

Sconza C, Respizzi S, Virelli L, Vandenbulcke F, Iacono F, Kon E, et al. Oxygen–ozone therapy for the treatment of knee osteoarthritis: a systematic review of randomized controlled trials. Arthrosc J Arthrosc Relat Surg. 2020;36:277–86.

Aliyev D, Akkemik U, Asik I. Efficacy of an intra-articular ozone injection for chronic knee pain due to osteoarthritis. Altern Ther Health Med. 2023;29:24–8.

Lopes De Jesus CC, Dos Santos FC, De Jesus LMOB, Monteiro I, SantAna MSSC, Trevisani VFM. Comparison between intra-articular ozone and placebo in the treatment of knee osteoarthritis: a randomized, double-blinded, placebo-controlled study. PLoS ONE. 2017;12: e0179185.

Biazzo A, Corriero AS, Confalonieri N. Intramuscular oxygen–ozone therapy in the treatment of low back pain. Acta Bio Medica Atenei Parmensis. 2018;89:41–6.

De Andrade RR, De Oliveira-Neto OB, Barbosa LT, Santos IO, De Sousa-Rodrigues CF, Barbosa FT. Effectiveness of ozone therapy compared to other therapies for low back pain: a systematic review with meta-analysis of randomized clinical trials. Braz J Anesthesiol. 2019;69:493–501 ( English Edition ).

Nabi BN, Sedighinejad A, Mirbolouk AR, Farzi F, Haghighi M, Atrkarroushan Z, et al. The effectiveness of ozone therapy in chronic osteomyelitis: a randomized controlled clinical trial. Arch Clin Infect Dis. 2018;13: e61320.

Google Scholar

Bilge A, Öztürk Ö, Adali Y, Üstebay S. Could ozone treatment be a promising alternative for osteomyelitis? An experimental study. Acta Ortop Bras. 2018;26:67–71.

Viebahn-Hänsler R, León Fernández OS, Fahmy Z. Ozone in medicine: the low-dose ozone concept—guidelines and treatment strategies. Ozone Sci Eng. 2012;34:408–24.

Yasheng T, Mijiti A, Yushan M, Liu Z, Liu Y, Yusufu A. Ozonated water lavage and physiological saline irrigation combined with vacuum-sealed drainage in the treatment of 18 cases of chronic osteomyelitis. J Int Med Res. 2021;49:0300060521999530.

Article CAS PubMed PubMed Central Google Scholar

Rowen RJ, Robins H. Ozone therapy for complex regional pain syndrome: review and case report. Curr Pain Headache Rep. 2019;23:41.

De Sire A, Agostini F, Lippi L, Mangone M, Marchese S, Cisari C, et al. Oxygen–ozone therapy in the rehabilitation field: state of the art on mechanisms of action, safety and effectiveness in patients with musculoskeletal disorders. Biomolecules. 2021;11:356.

Alebouyeh M-R, Morsali SF, Zojaji F, Ebrahimi SA, Ahani A, Antar A. Refractory complex regional pain syndrome: a case report and review of literature. Anesth Pain Med. 2023;13: e135286.

Seyam O, Smith N, Reid I, Gandhi J, Jiang W, Khan S. Clinical utility of ozone therapy for musculoskeletal disorders. Med Gas Res. 2018;8:103.

Bocci V. The potential toxicity of ozone: side effects and contraindications of ozonetherapy. In: Ozone. Dordrecht: Springer; 2010. p. 75–84.

Chapter Google Scholar

Xiao W, Tang H, Wu M, Liao Y, Li K, Li L, et al. Ozone oil promotes wound healing by increasing the migration of fibroblasts via PI3K/Akt/mTOR signaling pathway. Biosci Rep. 2017;37:BSR20170658.

Pchepiorka R, Moreira MS, Lascane NADS, Catalani LH, Allegrini S Jr, De Lima NB, et al. Effect of ozone therapy on wound healing in the buccal mucosa of rats. Arch Oral Biol. 2020;119: 104889.

Article CAS PubMed Google Scholar

Fitzpatrick E, Holland OJ, Vanderlelie JJ. Ozone therapy for the treatment of chronic wounds: a systematic review. Int Wound J. 2018;15:633–44.

Serra MEG, Baeza-Noci J, Mendes Abdala CV, Luvisotto MM, Bertol CD, Anzolin AP. The role of ozone treatment as integrative medicine. An evidence and gap map. Front Public Health. 2023;10:1112296.

Akkawi I. Ozone therapy for musculoskeletal disorders Current concepts. Acta Biomed. 2020;91(4):e2020191.

Bocci V. How is ozone administered? Ozone. 2010. https://doi.org/10.1007/978-90-481-9234-2_5 .

Article PubMed Central Google Scholar

Re L, Mawsouf MN, Menéndez S, León OS, Sánchez GM, Hernández F. Ozone therapy: clinical and basic evidence of its therapeutic potential. Arch Med Res. 2008;39:17–26.

Bocci VA. Tropospheric ozone toxicity vs. usefulness of ozone therapy. Arch Med Res. 2007;38:265–7.

Bocci V. Ozone as Janus: this controversial gas can be either toxic or medically useful. Mediat Inflamm. 2004;13:3–11.

Article CAS Google Scholar

Li Q, Qi X, Zhang Z. Intra-articular oxygen–ozone versus hyaluronic acid in knee osteoarthritis: a meta-analysis of randomized controlled trials. Int J Surg. 2018;58:3–10.

Sagai M, Bocci V. Mechanisms of action involved in ozone therapy: is healing induced via a mild oxidative stress? Med Gas Res. 2011;1:29.

Fernández-Cuadros ME, Pérez-Moro OS, Albaladejo-Florín MJ, Tobar-Izquierdo MM, Magaña-Sánchez A, Jiménez-Cuevas P, et al. Intra articular ozone modulates inflammation and has anabolic effect on knee osteoarthritis: IL-6 and IGF-1 as pro-inflammatory and anabolic biomarkers. Processes. 2022;10:138.

Travagli V, Iorio EL. The biological and molecular action of ozone and its derivatives: state-of-the-art, enhanced scenarios, and quality insights. Int J Mol Sci. 2023;24:8465.

Galiè M, Covi V, Tabaracci G, Malatesta M. The role of Nrf2 in the antioxidant cellular response to medical ozone exposure. Int J Mol Sci. 2019;20:4009.

Viebahn-Haensler R, León Fernández OS. Ozone in medicine. The low-dose ozone concept and its basic biochemical mechanisms of action in chronic inflammatory diseases. Int J Mol Sci. 2021;22:7890.

Xu W, Yang Z, Lu N. A new role for the PI3K/Akt signaling pathway in the epithelial–mesenchymal transition. Cell Adhes Migr. 2015;9:317–24.

Dong J, Zhang L, Jia H, Zhu Y, Xiang R, Li P. Effects of adjuvant application of corticosteroid and ozone after ultrasound-guided puncture and lavage for the treatment of rotator cuff calcific tendinitis: study protocol for a non-inferiority randomized controlled trial. Trials. 2023;24:375.

Stewart S. Fracture non-union: a review of clinical challenges and future research needs. Malays Orthop J. 2019;13:1–10.

Duman IG, Davul S, Gokce H, Gonenci R, Özden R, Uruc V. Effects of gaseous ozone treatment on bone regeneration in femoral defect model in rats. J Hard Tissue Biol. 2017;26:7–12.

Irban A, Uslu S, Gereli A, Aydinlar EI, Karyemez PE, Luleci N, et al. The effect of ozone therapy on experimental bone fracture healing in rats. Int Res J Public Environ Health. 2015. https://doi.org/10.15739/irjpeh.034 .

Shah P, Shyam AK, Shah S. Adjuvant combined ozone therapy for extensive wound over tibia. IJOO. 2011;45:376–9.

Vaillant JD, Fraga A, Díaz MT, Mallok A, Viebahn-Hänsler R, Fahmy Z, et al. Ozone oxidative postconditioning ameliorates joint damage and decreases pro-inflammatory cytokine levels and oxidative stress in PG/PS-induced arthritis in rats. Eur J Pharmacol. 2013;714:318–24.

Sun P, Xu W, Zhao X, Zhang C, Lin X, Gong M, et al. Ozone induces autophagy by activating PPARγ/mTOR in rat chondrocytes treated with IL-1β. J Orthop Surg Res. 2022;17:351.

Farpour HR, Ashraf A, Hosseini SS. The efficacy of ozone prolotherapy compared to intra-articular hypertonic saline injection in reducing pain and improving the function of patients with knee osteoarthritis: a randomized clinical trial. Evid-Based Complement Altern Med. 2021;2021:1–7.

Sconza C, Di Matteo B, Queirazza P, Dina A, Amenta R, Respizzi S, et al. Ozone therapy versus hyaluronic acid injections for pain relief in patients with knee osteoarthritis: preliminary findings on molecular and clinical outcomes from a randomized controlled trial. IJMS. 2023;24:8788.

Iliakis E, Bonetti M, Iliakis A. Osteonecrosis of the femoral head: could oxygen–ozone therapy became a treatment option? J Ozone Ther. 2015;1(1).1-7.

Sacco R, Leeson R, Nissan J, Olate S, Bettoni Cruz de Castro CH, Acocella A, et al. A systematic review of oxygen therapy for the management of medication-related osteonecrosis of the jaw (MRONJ). Appl Sci. 2019;9:1026.

Ripamonti CI, Cislaghi E, Mariani L, Maniezzo M. Efficacy and safety of medical ozone (O 3 ) delivered in oil suspension applications for the treatment of osteonecrosis of the jaw in patients with bone metastases treated with bisphosphonates: preliminary results of a phase I–II study. Oral Oncol. 2011;47:185–90.

Di Fede O, Del Gaizo C, Panzarella V, La Mantia G, Tozzo P, Di Grigoli A, et al. Ozone infiltration for osteonecrosis of the jaw therapy: a case series. J Clin Med. 2022;11:5307.

Agrillo A, Ungari C, Filiaci F, Priore P, Iannetti G. Ozone therapy in the treatment of avascular bisphosphonate-related jaw osteonecrosis. J Craniofac Surg. 2007;18:1071–5.

Shetty L, Channe S, Londhe U, Bongulwar R. Potential use of local ozone therapy for neonatal mandibular osteomyelitis. World J Dent. 2018;9:343–4.

Foula AS, Sabry LS, Elmulla AF, Kamel MA, Hozien AI. Ultrasound-guided shoulder intraarticular ozone injection versus pulsed radiofrequency application for shoulder adhesive capsulitis: a randomized controlled trial. Pain Physician. 2023;26(4):E329.

Kizilkaya V, Uruc V, Levent A, Kanat O, Yildizgoren MT, Dogramaci Y, et al. Effectiveness of ozone therapy on tendon healing: an experimental study in generated Achilles tendon injury model in rats. J Hard Tissue Biol. 2018;27:309–14.

Babaei-Ghazani A, Karimi N, Forogh B, Madani SP, Ebadi S, Fadavi HR, et al. Comparison of ultrasound-guided local ozone (O 2 –O 3 ) injection vs corticosteroid injection in the treatment of chronic plantar fasciitis: a randomized clinical trial. Pain Med. 2019;20:314–22.

Atar MÖ, Korkmaz N, Aslan SG, Tezen Ö, Köylü SU, Demir Y, et al. Comparison of ultrasound-guided subacromial corticosteroid and ozone (O 2 –O 3 ) injections in the treatment of chronic rotator cuff tendinopathy: a randomized clinical trial. Korean J Pain. 2023;36:128–36.

Hidalgo-Tallón FJ, Pinto-Bonilla R, Baeza-Noci J, Menéndez-Cepero S, Cabizosu A. Medical ozone on hamstring injury in a professional athlete assessed by thermography: a clinical case report. BJR Case Rep. 2023;9:20220078.

PubMed PubMed Central Google Scholar

Gurger M, Once G, Yilmaz E, Demir S, Calik I, Say Y, et al. The effect of the platelet-rich plasma and ozone therapy on tendon-to-bone healing in the rabbit rotator cuff repair model. J Orthop Surg Res. 2021;16:202.

Bozbaş GT, Yilmaz M, Paşaoğlu E, Gürer G, İvgin R, Demirci B. Effect of ozone in Freund’s complete adjuvant-induced arthritis. Arch Rheumatol. 2018;33:137–42.

Tartari APS, Moreira FF, Pereira MCDS, Carraro E, Cidral-Filho FJ, Salgado AI, et al. Anti-inflammatory effect of ozone therapy in an experimental model of rheumatoid arthritis. Inflammation. 2020;43:985–93.

León Fernández OS, Viebahn-Haensler R, Cabreja GL, Espinosa IS, Matos YH, Roche LD, et al. Medical ozone increases methotrexate clinical response and improves cellular redox balance in patients with rheumatoid arthritis. Eur J Pharmacol. 2016;789:313–8.

Taşçı BG. Effect of ozone in Freund’s complete adjuvant-induced arthritis. Arch Rheumatol. 2018;33:137–42.

Zhao N, Smargiassi A, Jean S, Gamache P, Laouan-Sidi E-A, Chen H, et al. Long-term exposure to fine particulate matter and ozone and the onset of systemic autoimmune rheumatic diseases: an open cohort study in Quebec, Canada. Arthritis Res Ther. 2022;24:151.

[PDF] Ozone and stem cell a possible cure for Rheumatoid Arthritis | Semantic Scholar. https://www.semanticscholar.org/paper/Ozone-and-stem-cell-a-possible-cure-for-Rheumatoid-Nd-Cht/710b110e67457449bf0fca17e1056db608ddf32d . Accessed 9 Oct 2023.

Rania V, Marcianò G, Casarella A, Vocca C, Palleria C, Calabria E, et al. Oxygen–ozone therapy in cervicobrachial pain: a real-life experience. JCM. 2022;12:248.

Hidalgo-Tallón J, Menéndez-Cepero S, Vilchez JS, Rodríguez-López CM, Calandre EP. Ozone therapy as add-on treatment in fibromyalgia management by rectal insufflation: an open-label pilot study. J Altern Complement Med. 2013;19:238–42.

Tirelli U, Cirrito C, Pavanello M. Oxygen–ozone therapy is an effective therapy in chronic fatigue syndrome: results in 100 patients. BJSTR. 2019;21:15890–2.

Tirelli U, Franzini M, Valdenassi L, Pandolfi S, Berretta M, Ricevuti G, et al. Patients with myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) greatly improved fatigue symptoms when treated with oxygen–ozone autohemotherapy. J Clin Med. 2021;11:29.

Tirelli U, Cirrito C, Pavanello M. Ozone therapy is an effective therapy in chronic fatigue syndrome: result of an Italian study in 65 patients. Ozone Ther. 2018. https://doi.org/10.4081/ozone.2018.7812 .

Chirumbolo S, Valdenassi L, Franzini M, Pandolfi S, Ricevuti G, Tirelli U. Male vs. female differences in responding to oxygen–ozone autohemotherapy (O 2 –O 3 -AHT) in patients with myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). J Clin Med. 2022;11:173.

Tirelli U, Cirrito C, Pavanello M. Ozone therapy in 40 patients with fibromyalgia: an effective therapy. Ozone Ther. 2018. https://doi.org/10.4081/ozone.2018.7969 .

Gazioğlu Türkyılmaz G, Rumeli Ş, Bakır M. Effects of major ozone autohemotherapy on physical functionality and quality of life in fibromyalgia syndrome: a prospective cross-sectional study. Altern Ther Health Med. 2021;27:8–12.

Sucuoğlu H, Soydaş N. Efficacy of ozone therapy as an add-on treatment in fibromyalgia: a randomized double-blind placebo-controlled study. J Back Musculoskelet Rehabil. 2023;36:357–66.

Tirelli U, Cirrito C, Pavanello M, Piasentin C, Lleshi A, Taibi R. Ozone therapy in 65 patients with fibromyalgia: an effective therapy. Eur Rev Med Pharmacol Sci. 2019;23:1786–8.

CAS PubMed Google Scholar

Romary DJ, Landsberger SA, Bradner KN, Ramirez M, Leon BR. Liquid ozone therapies for the treatment of epithelial wounds: a systematic review and meta-analysis. Int Wound J. 2023;20:1235–52.

Faraji N, Goli R, Choobianzali B, Bahrami S, Sadeghian A, Sepehrnia N, et al. Ozone therapy as an alternative method for the treatment of diabetic foot ulcer: a case report. J Med Case Rep. 2021;15:234.

Astasio-Picado Á, Babiano AÁ, López-Sánchez M, Lozano RR, Cobos-Moreno P, Gómez-Martín B. Use of ozone therapy in diabetic foot ulcers. J Pers Med. 2023;13:1439.

Kushmakov R, Gandhi J, Seyam O, Jiang W, Joshi G, Smith NL, et al. Ozone therapy for diabetic foot. Med Gas Res. 2018;8:111–5.

Izadi M, Kheirjou R, Mohammadpour R, Aliyoldashi MH, Moghadam SJ, Khorvash F, et al. Efficacy of comprehensive ozone therapy in diabetic foot ulcer healing. Diabetes Metab Syndr. 2019;13:822–5.

Wen Q, Liu D, Wang X, Zhang Y, Fang S, Qiu X, et al. A systematic review of ozone therapy for treating chronically refractory wounds and ulcers. Int Wound J. 2022;19:853–70.

Kim S-Y, Kim E, Kim WJ. Health effects of ozone on respiratory diseases. Tuberc Respir Dis. 2020;83(Suppl 1):S6–11.

Lobo V, Patil A, Phatak A, Chandra N. Free radicals, antioxidants and functional foods: impact on human health. Pharmacogn Rev. 2010;4:118–26.

Palli D, Sera F, Giovannelli L, Masala G, Grechi D, Bendinelli B, et al. Environmental ozone exposure and oxidative DNA damage in adult residents of Florence. Italy Environ Pollut. 2009;157:1521–5.

Ballinger CA, Cueto R, Squadrito G, Coffin JF, Velsor LW, Pryor WA, et al. Antioxidant-mediated augmentation of ozone-induced membrane oxidation. Free Radic Biol Med. 2005;38:515–26.

Manoto SL, Maepa MJ, Motaung SK. Medical ozone therapy as a potential treatment modality for regeneration of damaged articular cartilage in osteoarthritis. Saudi J Biol Sci. 2018;25:672–9.

Secondary direct food additives permitted in food for human consumption (21 CFR 173.5-173.405). | FAOLEX. https://www.fao.org/faolex/results/details/en/c/LEX-FAOC156335/ . Accessed 14 Nov 2023.

Grangeat AM, de LA Erario M. The use of medical ozone in chronic intervertebral disc degeneration can be an etiological and conservative treatment. Int J Mol Sci. 2023;24:6538.

El Meligy OA, Elemam NM, Talaat IM. Ozone therapy in medicine and dentistry: a review of the literature. Dent J. 2023;11:187.

Bocci V, Zanardi I, Borrelli E, Travagli V. Reliable and effective oxygen–ozone therapy at a crossroads with ozonated saline infusion and ozone rectal insufflation. J Pharm Pharmacol. 2012;64:482–9.

Shah M, Captain J, Vaidya V, Kulkarni A, Valsangkar K, Nair PMK, et al. Safety and efficacy of ozone therapy in mild to moderate COVID-19 patients: a phase 1/11 randomized control trial (SEOT study). Int Immunopharmacol. 2021;91: 107301.

Epelle EI, Macfarlane A, Cusack M, Burns A, Okolie JA, Mackay W, et al. Ozone application in different industries: a review of recent developments. Chem Eng J. 2023;454: 140188.

Steppan J, Meaders T, Muto M, Murphy KJ. A metaanalysis of the effectiveness and safety of ozone treatments for herniated lumbar discs. J Vasc Interv Radiol. 2010;21:534–48.

Download references

Acknowledgements

The authors received no financial support for the research, authorship, and/or publication of this article.

Author information

Authors and affiliations.

Department of Orthopaedics, ACS Medical College and Hospital, Dr. MGR Educational and Research Institute, Chennai, Tamil Nadu, 600077, India

Madhan Jeyaraman & Naveen Jeyaraman

Department of Orthopaedics, Government Medical College, Omandurar Government Estate, Chennai, Tamil Nadu, 600002, India

Swaminathan Ramasubramanian & Sangeetha Balaji

Department of Orthopaedics, Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER)–Karaikal, Puducherry, 605006, India

Arulkumar Nallakumarasamy

Department of Orthopaedics, All India Institute of Medical Sciences, Bhubaneswar, Odisha, 751019, India

Bishnu Prasad Patro

Department of Orthopaedic, Trauma, and Reconstructive Surgery, RWTH University Medical Centre, Pauwelsstraße 30, 52074, Aachen, Germany

Filippo Migliorini

Department of Orthopaedic and Trauma Surgery, Academic Hospital of Bolzano (SABES-ASDAA), 39100, Bolzano, Italy

Department of Life Sciences, Health, and Health Professions, Link Campus University, Rome, Italy

You can also search for this author in PubMed Google Scholar

Contributions

MJ: conception and design, drafting (original and revision); MJ, FM: supervision, drafting (revision); MJ, FM: drafting (revision); NJ, SB, SR, AN, BPP: drafting (original). All authors have agreed to the final version to be published and agree to be accountable for all aspects of the work.

Corresponding author

Correspondence to Filippo Migliorini .

Ethics declarations

Ethics approval and consent to participate.

This study complies with ethical standards. Registration and protocol: the present review was not registered.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have any competing interests in this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Jeyaraman, M., Jeyaraman, N., Ramasubramanian, S. et al. Ozone therapy in musculoskeletal medicine: a comprehensive review. Eur J Med Res 29 , 398 (2024). https://doi.org/10.1186/s40001-024-01976-4

Download citation

Received : 28 November 2023

Accepted : 14 July 2024

Published : 31 July 2024

DOI : https://doi.org/10.1186/s40001-024-01976-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Orthopaedics
Traumatology
Musculoskeletal medicine

European Journal of Medical Research

ISSN: 2047-783X

Submission enquiries: [email protected]
General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
J Grad Med Educ
v.14(5); 2022 Oct

Steps for Conducting a Scoping Review

Susanne mak.

Both authors are with McGill University, Montreal, Quebec, Canada

Susanne Mak, MSc, is an Assistant Professor, School of Physical and Occupational Therapy, and an Associate Member, Institute of Health Sciences Education, Faculty of Medicine and Health Sciences

Aliki Thomas

Aliki Thomas, PhD, is an Associate Professor, School of Physical and Occupational Therapy, and an Associate Member, Institute of Health Sciences Education, Faculty of Medicine and Health Sciences

A scoping review is a type of knowledge synthesis that uses a systematic and iterative approach to identify and synthesize an existing or emerging body of literature on a given topic. 1 While there are several reasons for conducting a scoping review, the main reasons are to map the extent, range, and nature of the literature, as well as to determine possible gaps in the literature on a topic. 1 - 3 Scoping reviews are not limited to peer-reviewed literature. 3 , 4

Identifying a Team

Before conducting the review, it is important to consider the composition of the research team: scoping reviews are not conducted by a single individual. The team should include someone with content expertise and an individual with experience conducting scoping reviews. 1 , 3 , 5 Adding a librarian who can assist with building the search strategy is also extremely helpful. 1 , 3 Thoughtful planning of the team membership will result in the right knowledge, skills, and expertise to successfully complete the review and ensure that the findings make a notable contribution to the field.

An overview of the steps involved in conducting scoping reviews is provided below.

Step 1: Identifying the Research Question

Creating the research question is a vital first step. 1 , 3 - 5 A question that is too broad increases the number of papers for consideration, which may affect the feasibility of the review. 5 A question that is too narrow may compromise the breadth and depth of the review. Therefore, a preliminary search of the literature may be helpful in determining: (1) the breadth of your question; (2) whether a scoping review on the topic has already been conducted; and (3) if there is sufficient literature to warrant a scoping review. Consulting with a librarian can help in deciding if a scoping review is the appropriate review method. 1 , 3 In particular, a librarian may confirm that there is insufficient literature or that there is too much, which will necessitate a more targeted research question.

Step 2: Identifying Relevant Studies

Early consultation with a librarian should occur to build the search strategy—keywords, Medical Subject Headings, databases—and further refine the strategy based on the papers found. For example, you may find too many irrelevant papers. In this case you may need to review your search strategy to identify the terms which introduce too much “noise.”

You will also need to define the inclusion and exclusion criteria. 1 , 3 - 5 Discussions with your team are important to ensure diverse perspectives and that the inclusion criteria are aligned with the research question. 5 , 6

Step 3: Selecting Studies to Be Included in the Review

Tools such as Covidence and Rayyan can be helpful in organizing papers and making the screening process more efficient ( Box ). Once you have collected the citations from the search, you can import these from reference management software (eg, EndNote) into Rayyan. After selecting papers for inclusion, the citations of the included papers can be exported to reference management software for the next stage of the review. Other helpful features of management software can include the identification of duplicates, proportion of an abstract that resembles another, and documentation of reasons for inclusion or exclusion. Both Covidence and Rayyan allow for blinding the results of team members' reviews to each other.

▪ Covidence: www.covidence.org
▪ NVivo: https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home
▪ Rayyan: https://rayyan.qcri.org/welcome

Having additional reviewers will accelerate the pace of the review but will require calibration between reviewers. 1 , 3 , 5 A calibration exercise consists of selecting 5% to 10% of the papers for independent screening by each reviewer. 1 If a high level of agreement among reviewers is not achieved (eg, lower than 90%), 7 , 8 the reviewers should discuss their points of disagreement and review (and possibly revise) the inclusion criteria. 1 Another 10% of the papers are then selected for a second calibration exercise to test the modified inclusion criteria. If having 2 reviewers for each paper is not feasible, one reviewer can conduct an independent review, with a second reviewer verifying a portion of the papers, with the goal of 90% or better agreement.

The actual screening of papers should consist of reading not only the title of the paper, but the abstract as well. If an abstract is not available, a full-text review of the paper is required. Screening papers by title alone is insufficient, as the contents of a paper are not always well reflected in the title.

Step 4: Charting the Data

The team develops the data extraction form collaboratively. Although the extraction categories vary depending on the research question and review purpose, common categories are: author, year, geographical location, study population, main results, study limitations, and future directions. 4 , 5 More specific categories will be needed to capture the data for a given research question.

The extraction form will need to be pilot tested for further refinements and undergo a calibration exercise as well. 1 , 3 , 5 This entails a dyad of reviewers independently extracting data from a small number of papers (eg, 5-10), and meeting afterward to discuss any discrepancies, with further refinement of the form if a high level of agreement between reviewers is not obtained.

Step 5: Collating, Summarizing, and Reporting the Results

Once the data have been extracted from all papers, numerical and thematic analyses are conducted. 5 The findings from the numerical analysis can be presented in a table or chart to showcase the most salient aspects of the review. Readers should be able to see alignment of findings with objectives for conducting the review. 1 , 3 Thematic analysis 9 consists of examining excerpts of text and asking how this text relates to the research question, as well as creating a code (label) that best reflects that text. A list of tentative codes (a codebook) is created and modified iteratively as the team engages in data analysis. Once codes are developed, a review of the codes and how they relate to each other can help to identify patterns among them, which leads to the creation of categories (collections of similar data in one place) 10 and themes (patterns across the dataset). 9

Reflexivity is essential throughout the review process but especially during thematic analysis, with use of memos, to capture the thoughts that arise from examining and interpreting the data. Once the codes are generated, the research team will further refine them through discussion. 6 The team should discuss not only the clarity of the operational definitions of the codes, but also how the codes are named and how they may relate to each other. As the codes are grouped together, the team will develop themes. 5

Step 6: Consulting Stakeholders (Optional)

Reasons for stakeholder consultation may be to obtain input on the research question and sources of information, and to provide insights on a topic. Other purposes may include obtaining feedback to help shed light on the review findings and pinpoint gaps not explored in the literature. While a stakeholder consultation has been named as the final step of a review, it can be incorporated throughout the review stages and can occur through focus groups, individual interviews, or surveys. 1 , 5

A scoping review is useful to map the literature on evolving or emerging topics and to identify gaps. It may be a step before undertaking research or conducting another type of review, such as a systematic review. Before conducting a scoping review, it is important to consider how the research team will implement each step and who will be involved at each stage, while being mindful that the methodological approach provides teams with the opportunity to move back to earlier stages as the review evolves.

IMAGES

(PDF) Exploring Research Methodology: Review Article
Peer Review Process
Systematic Literature Review Methodology
Methodology for the review process through Journal articles
PPT
Types of Research Methodology: Uses, Types & Benefits

VIDEO

How to write Literature Review
TR Webinar -Writing High-Quality Manuscripts and Publishing Your Research
Methodological Reviews
Unlocking the Secrets of arxiv
Positive Academy Session 6 What is Literature Survey and Review, What is Journal
Warning: The dangers of excessive PhD journal publications

COMMENTS

Literature review as a research methodology: An overview and guidelines
Her research interest relates to service innovation, customer creativity, deviant customer behavior, and value co-creation as well as a special interest in literature review methodology. She has published in the Journal of Business Research, European Journal of Marketing, Journal of Service Management and International Journal of Nursing Studies.
Guidance on Conducting a Systematic Literature Review
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
Methodological Approaches to Literature Review
The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature.
Systematic reviews: Structure, form and content
The systematic review methodology originated in medical and healthcare research, but it has now been adopted by other disciplines, such as engineering, education, economics and business studies. ... co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews Journal of Clinical ...
An overview of methodological approaches in systematic reviews
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...
Writing a Scientific Review Article: Comprehensive Insights for
2. Benefits of Review Articles to the Author. Analysing literature gives an overview of the "WHs": WHat has been reported in a particular field or topic, WHo the key writers are, WHat are the prevailing theories and hypotheses, WHat questions are being asked (and answered), and WHat methods and methodologies are appropriate and useful [].For new or aspiring researchers in a particular ...
Reviewing the research methods literature: principles and strategies
Overviews of methods are potentially useful means to increase clarity and enhance collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that demand ...
Systematic reviews: Structure, form and content
Topic selection and planning. In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions.Systematic reviews can be inadvisable for a variety of reasons.
How-to conduct a systematic literature review: A quick guide for
Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...
Scoping reviews: reinforcing and advancing the methodology and
Scoping reviews are an increasingly common approach to evidence synthesis with a growing suite of methodological guidance and resources to assist review authors with their planning, conduct and reporting. The latest guidance for scoping reviews includes the JBI methodology and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses—Extension for Scoping Reviews.
Guidelines for writing a systematic review
The journal's scope is included on their web pages, and the author/s need to take into account the international relevance. ... Rapid review: Utilises Systematic Review methods but may be time limited. (Randles and Finnegan, 2022) Meta-analysis: Technique of combining the results of quantitative research studies and statistically analysing the ...
PDF METHODOLOGY OF THE LITERATURE REVIEW
O D E S. Data); (f) Step 6: Analyze and Synthesize Information; and (g) Step 7: Present the CLR Report. These seven steps are multidimensional, interactive, emergent, iterative, dynamic, holistic, and synergistic. By multidimensional, we mean that each of the steps has multiple components or dimensions.
How to Do a Systematic Review: A Best Practice Guide for ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
How to do a systematic review
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings.13 Systematic reviews may combine data from different.
Full article: Rapid literature review: definition and methodology
Definition. Cochrane Rapid Reviews Methods Group developed methods guidance based on scoping review of the underlying evidence, primary methods studies conducted, as well as surveys sent to Cochrane representative and discussion among those with expertise [Citation 11].They analyzed over 300 RLRs or RLR method papers and based on the methodology of those studies, constructed a broad definition ...
A tutorial on methodological studies: the what, when, how and why
This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. Journal characteristics: Certain journals' characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [63, 64], and Journal Impact Factor (JIF) have been shown to be associated with ...
How to Review a Journal Article
Before getting started on the critique, it is important to review the article thoroughly and critically. To do this, we recommend take notes, annotating, and reading the article several times before critiquing. As you read, be sure to note important items like the thesis, purpose, research questions, hypotheses, methods, evidence, key findings ...
Literature Review Research
Literature Review is a comprehensive survey of the works published in a particular field of study or ... Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches. Indicates potential directions for future research. ... Australian and New Zealand Journal of Public Health, 39: 103-103. doi:10. ...
Steps for Conducting a Scoping Review
When not to choose a scoping review methodology: ... Peer-reviewed journal papers were included if they were: published between the period of 2000-2016, written in English, involved human participants and described a measure for burden of treatment, e.g. including single measurements, measuring and/or incorporating one or two dimensions of ...
Chapter 9 Methods for Literature Reviews
9.3. Types of Review Articles and Brief Illustrations. EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic.
How to write methodology review when in article there is no
If you meant that you want to create a methods section in a review article that you are writing, i.e., to describe the methodology you followed to review the literature and write your review article, you can simply introduce a section detailing this. In this section, you would typically start by mentioning how you formulated the research problem.
What Is Peer Review?
What Is Peer Review? | Types & Examples. Published on December 17, 2021 by Tegan George.Revised on June 22, 2023. Peer review, sometimes referred to as refereeing, is the process of evaluating submissions to an academic journal.Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.
Finding and Reading Journal Articles
Peer review simply means other experts believe the methods, the evidence, the conclusions of an article have met important standards of legitimacy, reliability, and intellectual honesty. Searching the journal literature is part of being a responsible researcher at any level: professor, grad student, concentrator, first-year.
Reviewing Methods Contributions: From Current & Previous Editors of
Previous work suggests the Academy strengthen its emphasis on reviewer training (e.g., Raelin, 2008). In response to this call, we hope to provide advice to scholars regarding 1) what makes a methodological contribution and 2) how to apply this knowledge when reviewing methods papers. Authors and reviewers alike are given little guidance regarding what makes for a methodological contribution ...
Deep learning based network intrusion detection system: a ...
This SLR represents the subsequent planning protocol of review consisting of background definition of the review, identifying the requirement for SLR, revealing of research questions (RQs), Searching methodology of papers, selection of key papers, inclusion criteria, exclusion criteria, Quality assessment(QA) of selected research papers Quality ...
A tutorial on methodological studies: the what, when, how and why
Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. ... analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC ...
Uncertainty in tuberculosis clinical decision-making: An umbrella
Study design rationale and methodology. An initial scoping search was conducted on MEDLINE (via OVID) for terms related to "tuberculosis" and "decision-making", and identified several reviews relevant to our research question [23-25].Since most records evaluated either qualitative or quantitative primary studies, and often reported complementary findings, we chose an umbrella review ...
Ozone therapy in musculoskeletal medicine: a comprehensive review
Ozone therapy is an innovative modality which gained growing attention in medical science, especially in musculoskeletal medicine [].Administered as a mixture of oxygen and ozone gases, it stimulates physiological responses, particularly anti-inflammatory and analgesic effects [2, 3].By doing so, ozone therapy is administered in various acute and chronic musculoskeletal disorders such as ...
Generative AI in Undergraduate Medical Education: A Rapid Review
A total of 521 relevant articles were screened during this review. Forty-one articles underwent full-text review and data extraction. The majority of the articles were opinion pieces (9), case reports (8), letters to the editor (5), editorials (5), and commentaries (3) about the use of generative AI while 7 articles used qualitative and/or quantitative methods.
Steps for Conducting a Scoping Review
A scoping review is a type of knowledge synthesis that uses a systematic and iterative approach to identify and synthesize an existing or emerging body of ... advancing the methodology. Implement Sci. 2010; 5:69. doi: 10.1186/1748-5908-5-69. [PMC free ... Articles from Journal of Graduate Medical Education are provided here courtesy of ...

Scoping reviews: reinforcing and advancing the methodology and application

Introduction

What is a scoping review and how is it different from other evidence syntheses?

Guidance to improve the quality of reporting of scoping reviews

Reasons for conducting a scoping review

Future directions in scoping reviews

Availability of data and materials

Acknowledgements

Author information

Contributions

Corresponding author

Ethics declarations

Consent for publication

Additional information

Rights and permissions

About this article

Share this article

Systematic Reviews

Save citation to file

Add to My Bibliography

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Similar articles

LinkOut - more resources

Other Literature Sources

Miscellaneous

A tutorial on methodological studies: the what, when, how and why

What is a methodological study?

When should we conduct a methodological study?

How often are methodological studies conducted?

Why do we conduct methodological studies?

Where can we find methodological studies?

Some frequently asked questions about methodological studies

A proposed framework

Conclusions

Availability of data and materials

Abbreviations

Acknowledgements

Author information

Contributions

Corresponding author

Ethics declarations

Consent for publication

Additional information

Rights and permissions

About this article

Share this article

BMC Medical Research Methodology

How to Review a Journal Article

Questions to Consider

Evaluating Purpose and Argument

Evaluating the Presentation/Organization of Information

Evaluating Methods

Evaluating Data

Research Methods

Literature Review

Steps for Conducting a Scoping Review

What is a Scoping Review?

When a scoping review methodology would be appropriate:

When not to choose a scoping review methodology:

Methodological Guidelines

Methodological guidelines for scoping reviews have evolved over time:

1. Developing review objective(s) & question(s)

Defining Objectives:

Developing Research Questions:

2. Write A Research Protocol

Example Protocols:

Report in the Methods Section

3. Developing eligibility criteria

When specifying the inclusion and exclusion criteria, consider the following aspects:

Iterative Process

4. Information Sources

5. Searching for the evidence

Ovid Search Strategy (Table 1)

Citation Chasing Process

6. Selecting the evidence

The search results are screened against pre-defined eligibility criteria to determine inclusion in the review.

7. Extracting the evidence

8. Analyzing the evidence

9. Presenting the results

Narrative Summaries