how to review qualitative research article

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Critically appraising...

Critically appraising qualitative research

• Related content
• Peer review
• Ayelet Kuper , assistant professor 1 ,
• Lorelei Lingard , associate professor 2 ,
• Wendy Levinson , Sir John and Lady Eaton professor and chair 3
• 1 Department of Medicine, Sunnybrook Health Sciences Centre, and Wilson Centre for Research in Education, University of Toronto, 2075 Bayview Avenue, Room HG 08, Toronto, ON, Canada M4N 3M5
• 2 Department of Paediatrics and Wilson Centre for Research in Education, University of Toronto and SickKids Learning Institute; BMO Financial Group Professor in Health Professions Education Research, University Health Network, 200 Elizabeth Street, Eaton South 1-565, Toronto
• 3 Department of Medicine, Sunnybrook Health Sciences Centre
• Correspondence to: A Kuper ayelet94{at}post.harvard.edu

Six key questions will help readers to assess qualitative research

Summary points

Appraising qualitative research is different from appraising quantitative research

Qualitative research papers should show appropriate sampling, data collection, and data analysis

Transferability of qualitative research depends on context and may be enhanced by using theory

Ethics in qualitative research goes beyond review boards’ requirements to involve complex issues of confidentiality, reflexivity, and power

Over the past decade, readers of medical journals have gained skills in critically appraising studies to determine whether the results can be trusted and applied to their own practice settings. Criteria have been designed to assess studies that use quantitative methods, and these are now in common use.

In this article we offer guidance for readers on how to assess a study that uses qualitative research methods by providing six key questions to ask when reading qualitative research (box 1). However, the thorough assessment of qualitative research is an interpretive act and requires informed reflective thought rather than the simple application of a scoring system.

Box 1 Key questions to ask when reading qualitative research studies

Was the sample used in the study appropriate to its research question.

Were the data collected appropriately?

Were the data analysed appropriately?

Can I transfer the results of this study to my own setting?

Does the study adequately address potential ethical issues, including reflexivity?

Overall: is what the researchers did clear?

One of the critical decisions in a qualitative study is whom or what to include in the sample—whom to interview, whom to observe, what texts to analyse. An understanding that qualitative research is based in experience and in the construction of meaning, combined with the specific research question, should guide the sampling process. For example, a study of the experience of survivors of domestic violence that examined their reasons for not seeking help from healthcare providers might focus on interviewing a …

Log in using your username and password

BMA Member Log In

If you have a subscription to The BMJ, log in:

• Need to activate
• Log in via institution
• Log in via OpenAthens

Log in through your institution

Subscribe from £184 *.

Subscribe and get access to all BMJ articles, and much more.

* For online subscription

Access this article for 1 day for: £33 / $40 / €36 ( excludes VAT )

You can download a PDF version for your personal record.

Buy this article

Beyond the pages of academic medicine

10 Tips for Reviewing a Qualitative Paper

Editor’s Note: The following post is part of a series of Peer Reviewer Resources written by some of  Academic Medicine ‘s top peer reviewers. Read other peer review posts .

By: Carol-anne Moulton, MD, FRACS, MEd, PhD, Department of Surgery, University of Toronto, and Priyanka Patel, MSc, Wilson Center, University Health Network, University of Toronto

This is a tough task. Let us say that off the bat. We have been involved in qualitative research for a long time now and the complexity of it never ceases to amaze us…so there is no “how to” guide that will suit all qualitative research.

Having said that, we think there are some guiding principles that can help us begin to understand the rigor of qualitative research and consequently the review process.

• Question/Purpose : This should be clearly stated, as in all research studies. There are generally no hypotheses statements in qualitative research as we are not testing but rather exploring. Ideally, the questions are framed around how and why type questions, rather than how often, is there a difference, or what are the factors type questions.
• Rationale of study : We like to make sure that the study was built upon a well justified and referenced rationale. It may not be our area of study but we think it is important for the authors to provide rationale for their study by building up the arguments from the literature. Theories or pre-existing frameworks that informed the research question should be described up front. Some work claims to be atheoretical. Traditionally, grounded theorists claimed their work to be atheoretical, but nowadays many grounded theorists are acknowledging being informed by particular perspectives, frameworks, or theories. This should be made explicit.
• Methodology described : What type of research was this? Ethnography? Grounded theory? Phenomenology? Discourse analysis? It’s important that the researchers describe their research journey in a clear and detailed enough way to give the readers an understanding for how the analyses evolved. This should include an explanation of why the methodological approach was used, as well as the key principles from the methodology that guided the study.
• Epistemology : Researchers come from all paradigms and it is important to identify within which paradigm the authors are situated. Sometimes they might state deliberately “We have used constructivist grounded theory,” but it might be a matter of reading between the lines to figure it out. If from the positivist paradigm, authors might use the terms valid or verified to imply they are making statements of truth. The paradigm helps us understand what the authors mean by “truth” and informs how they went about creating knowledge and constructing meaning from their results.
• Context described satisfactorily : Qualitative research is not meant to imply generalizability. In fact, we celebrate the importance of context. We recognize that the phenomena we study are often different in meaningful ways when taken to a different context. For example, the experiences of physicians coping with burnout may be unique based on specialty and/or institution (i.e. type of systems-level support available, differing demands in academic or community institutions). A good qualitative study should therefore describe sufficient details of context (i.e. physical, cultural, social, and/or environmental context) in which the research was conducted to allow the reader to make judgments of whether the results might be transferable to another (possibly their own) setting.
• Data collection and analysis : Do they provide enough information to understand the collection and analysis process? As reviewers, we often ask ourselves whether the data collection and analyses are clear and detailed enough for us to gain a sense of how the analysis of the phenomena evolved. For example, who made up the research team? Because most knowledge is viewed as a co-construction between researcher and participants, each individual (e.g. a sociologist versus a surgeon) will analyze the results differently, but both meaningfully, based on their unique position and perspective.
• Sampling strategies : These are very important to understand whether the question was aligned with the data collection process. The sample reflects the type of results achieved and helps the reader understand from which perspective the data was collected. Some common sampling strategies include theoretical sampling and negative case sampling. Researchers may theoretically sample by selecting participants that in someway inform their understanding of an emergent theme or idea. Negative case sampling may be used to search for instances that may challenge the emergent patterns from the data for the purpose of refining the analysis. Negative case sampling is used to ensure that the researchers are not specifically selecting cases that confirm their findings.
• Analysis elevated beyond description : Results might be descriptive in nature (e.g. “One surgeon felt upset and isolated after he experienced a hernia complication in his first month of independent practice”) or they might be elevated to create more abstract concepts and ideas removed from the primary dataset (e.g. characterizing the phases of surgeons’ reactions to complications). In either case, the researcher should ensure that the way they present their findings are aligned with principles of the methodology used.
• Proof of an iterative process : Qualitative research is usually done in an iterative manner where ideas and concepts are built up over time and occur through cycles of data collection and data analysis. This is demonstrated through statements like “Our interview template was altered over time to reflect the emergent ideas through the analysis process,” or “As we became interested in this concept, we began to sample for…”.
• Reflexivity : This is tough to understand, especially for those of us who come from the positivist paradigm where it is of utmost importance to “prove” that the results are “true” and untainted by bias. The aim of qualitative research is to understand meaning rather than assuming that there is a singular truth or reality. A good qualitative researcher recognizes that the way they make sense of and attach meaning to the data is partly shaped by the characteristics of the researcher (i.e. age, gender, social class, ethnicity, professional status, etc.) and the assumptions they hold. The researcher should make explicit the perspectives they are coming from so that the readers can interpret the data appropriately. Consider a study exploring the pressures surgical trainees experience in residency conducted by a staff surgeon versus a non-surgical anthropologist. You can imagine the findings may differ based on the types of questions the two interviewers decide to ask, what they each find interesting or important, or how comfortable the resident feels discussing sensitive information with an outsider (anthropologist) as opposed to an insider (surgeon). We like to see that a researcher has reflected on how her or his unique position, preconceptions, and biases influenced the findings.

19 thoughts on “ 10 Tips for Reviewing a Qualitative Paper ”

Pingback: Qualitative Review in Nursing - Nursing Papers 247

Pingback: Qualitative Review in Nursing - Essay Don

Pingback: Qualitative Review in Nursing - Assignmentnerds.net

Pingback: Qualitative Review in Nursing - Academia Essays

Pingback: qualitative review in nursing - Superb Papers

Pingback: Qualitative Review in Nursing - Allessaysexpert

Pingback: Qualitative Review in Nursing - Chip Writers

Pingback: Science - Technical Assignments

Pingback: Qualitative Review in Nursing – fastwriting

Pingback: Qualitative Review in Nursing – EssaySolutions.net

Pingback: Qualitative Review in Nursing - Homework Market

Pingback: Qualitative Review in Nursing - Midterm Essays

Pingback: Qualitative Review in Nursing - Tutoring Beast

Pingback: Qualitative Review in Nursing - StudyCore

Pingback: Qualitative Review in Nursing - Custom Nursing Essays

Pingback: qualitative review in nursing - Gradesmine.com

Pingback: Qualitative Review in Nursing - Master My Course

Pingback: Qualitative Review in Nursing - Proficient Essay Help

Pingback: Qualitative Review in Nursing - Eliteprofessionalwriters.com

Comments are closed.

Discover more from AM Rounds

Subscribe now to keep reading and get access to the full archive.

Type your email…

Continue reading

Archer Library

Qualitative research: literature review .

• Archer Library This link opens in a new window
• Schedule a Reference Appointment This link opens in a new window
• Qualitative Research Handout This link opens in a new window
• Locating Books
• ebook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Stats
• Literature Review Resources
• Citation & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

AU Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: Scholarly & Practitioner Journals

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

The Chronicle of Higher Education has the nation’s largest newsroom dedicated to covering colleges and universities.  Source of news, information, and jobs for college and university faculty members and administrators

The Chronicle features complete contents of the latest print issue; daily news and advice columns; current job listings; archive of previously published content; discussion forums; and career-building tools such as online CV management and salary databases. Dates covered: 1970-present.

Offers in-depth coverage of national and international business and finance as well as first-rate coverage of hard news--all from America's premier financial newspaper. Covers complete bibliographic information and also subjects, companies, people, products, and geographic areas. 

Comprehensive coverage back to 1984 is available from the world's leading financial newspaper through the ProQuest database. 

Newspaper Source provides cover-to-cover full text for hundreds of national (U.S.), international and regional newspapers. In addition, it offers television and radio news transcripts from major networks.

Provides complete television and radio news transcripts from CBS News, CNN, CNN International, FOX News, and more.

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: EBSCOhost Search Tips
• EBSCO Connect: Basic Searching with EBSCO
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Stats
• Next: Literature Review Resources >>
• Last Updated: Jun 27, 2024 11:14 AM
• URL: https://libguides.ashland.edu/qualitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

• Open access
• Published: 27 May 2020

How to use and assess qualitative research methods

• Loraine Busetto   ORCID: orcid.org/0000-0002-9228-7875 1 ,
• Wolfgang Wick 1 , 2 &
• Christoph Gumbinger 1  

Neurological Research and Practice volume  2 , Article number:  14 ( 2020 ) Cite this article

759k Accesses

332 Citations

90 Altmetric

Metrics details

This paper aims to provide an overview of the use and assessment of qualitative research methods in the health sciences. Qualitative research can be defined as the study of the nature of phenomena and is especially appropriate for answering questions of why something is (not) observed, assessing complex multi-component interventions, and focussing on intervention improvement. The most common methods of data collection are document study, (non-) participant observations, semi-structured interviews and focus groups. For data analysis, field-notes and audio-recordings are transcribed into protocols and transcripts, and coded using qualitative data management software. Criteria such as checklists, reflexivity, sampling strategies, piloting, co-coding, member-checking and stakeholder involvement can be used to enhance and assess the quality of the research conducted. Using qualitative in addition to quantitative designs will equip us with better tools to address a greater range of research problems, and to fill in blind spots in current neurological research and practice.

The aim of this paper is to provide an overview of qualitative research methods, including hands-on information on how they can be used, reported and assessed. This article is intended for beginning qualitative researchers in the health sciences as well as experienced quantitative researchers who wish to broaden their understanding of qualitative research.

What is qualitative research?

Qualitative research is defined as “the study of the nature of phenomena”, including “their quality, different manifestations, the context in which they appear or the perspectives from which they can be perceived” , but excluding “their range, frequency and place in an objectively determined chain of cause and effect” [ 1 ]. This formal definition can be complemented with a more pragmatic rule of thumb: qualitative research generally includes data in form of words rather than numbers [ 2 ].

Why conduct qualitative research?

Because some research questions cannot be answered using (only) quantitative methods. For example, one Australian study addressed the issue of why patients from Aboriginal communities often present late or not at all to specialist services offered by tertiary care hospitals. Using qualitative interviews with patients and staff, it found one of the most significant access barriers to be transportation problems, including some towns and communities simply not having a bus service to the hospital [ 3 ]. A quantitative study could have measured the number of patients over time or even looked at possible explanatory factors – but only those previously known or suspected to be of relevance. To discover reasons for observed patterns, especially the invisible or surprising ones, qualitative designs are needed.

While qualitative research is common in other fields, it is still relatively underrepresented in health services research. The latter field is more traditionally rooted in the evidence-based-medicine paradigm, as seen in " research that involves testing the effectiveness of various strategies to achieve changes in clinical practice, preferably applying randomised controlled trial study designs (...) " [ 4 ]. This focus on quantitative research and specifically randomised controlled trials (RCT) is visible in the idea of a hierarchy of research evidence which assumes that some research designs are objectively better than others, and that choosing a "lesser" design is only acceptable when the better ones are not practically or ethically feasible [ 5 , 6 ]. Others, however, argue that an objective hierarchy does not exist, and that, instead, the research design and methods should be chosen to fit the specific research question at hand – "questions before methods" [ 2 , 7 , 8 , 9 ]. This means that even when an RCT is possible, some research problems require a different design that is better suited to addressing them. Arguing in JAMA, Berwick uses the example of rapid response teams in hospitals, which he describes as " a complex, multicomponent intervention – essentially a process of social change" susceptible to a range of different context factors including leadership or organisation history. According to him, "[in] such complex terrain, the RCT is an impoverished way to learn. Critics who use it as a truth standard in this context are incorrect" [ 8 ] . Instead of limiting oneself to RCTs, Berwick recommends embracing a wider range of methods , including qualitative ones, which for "these specific applications, (...) are not compromises in learning how to improve; they are superior" [ 8 ].

Research problems that can be approached particularly well using qualitative methods include assessing complex multi-component interventions or systems (of change), addressing questions beyond “what works”, towards “what works for whom when, how and why”, and focussing on intervention improvement rather than accreditation [ 7 , 9 , 10 , 11 , 12 ]. Using qualitative methods can also help shed light on the “softer” side of medical treatment. For example, while quantitative trials can measure the costs and benefits of neuro-oncological treatment in terms of survival rates or adverse effects, qualitative research can help provide a better understanding of patient or caregiver stress, visibility of illness or out-of-pocket expenses.

How to conduct qualitative research?

Given that qualitative research is characterised by flexibility, openness and responsivity to context, the steps of data collection and analysis are not as separate and consecutive as they tend to be in quantitative research [ 13 , 14 ]. As Fossey puts it : “sampling, data collection, analysis and interpretation are related to each other in a cyclical (iterative) manner, rather than following one after another in a stepwise approach” [ 15 ]. The researcher can make educated decisions with regard to the choice of method, how they are implemented, and to which and how many units they are applied [ 13 ]. As shown in Fig.  1 , this can involve several back-and-forth steps between data collection and analysis where new insights and experiences can lead to adaption and expansion of the original plan. Some insights may also necessitate a revision of the research question and/or the research design as a whole. The process ends when saturation is achieved, i.e. when no relevant new information can be found (see also below: sampling and saturation). For reasons of transparency, it is essential for all decisions as well as the underlying reasoning to be well-documented.

Iterative research process

While it is not always explicitly addressed, qualitative methods reflect a different underlying research paradigm than quantitative research (e.g. constructivism or interpretivism as opposed to positivism). The choice of methods can be based on the respective underlying substantive theory or theoretical framework used by the researcher [ 2 ].

Data collection

The methods of qualitative data collection most commonly used in health research are document study, observations, semi-structured interviews and focus groups [ 1 , 14 , 16 , 17 ].

Document study

Document study (also called document analysis) refers to the review by the researcher of written materials [ 14 ]. These can include personal and non-personal documents such as archives, annual reports, guidelines, policy documents, diaries or letters.

Observations

Observations are particularly useful to gain insights into a certain setting and actual behaviour – as opposed to reported behaviour or opinions [ 13 ]. Qualitative observations can be either participant or non-participant in nature. In participant observations, the observer is part of the observed setting, for example a nurse working in an intensive care unit [ 18 ]. In non-participant observations, the observer is “on the outside looking in”, i.e. present in but not part of the situation, trying not to influence the setting by their presence. Observations can be planned (e.g. for 3 h during the day or night shift) or ad hoc (e.g. as soon as a stroke patient arrives at the emergency room). During the observation, the observer takes notes on everything or certain pre-determined parts of what is happening around them, for example focusing on physician-patient interactions or communication between different professional groups. Written notes can be taken during or after the observations, depending on feasibility (which is usually lower during participant observations) and acceptability (e.g. when the observer is perceived to be judging the observed). Afterwards, these field notes are transcribed into observation protocols. If more than one observer was involved, field notes are taken independently, but notes can be consolidated into one protocol after discussions. Advantages of conducting observations include minimising the distance between the researcher and the researched, the potential discovery of topics that the researcher did not realise were relevant and gaining deeper insights into the real-world dimensions of the research problem at hand [ 18 ].

Semi-structured interviews

Hijmans & Kuyper describe qualitative interviews as “an exchange with an informal character, a conversation with a goal” [ 19 ]. Interviews are used to gain insights into a person’s subjective experiences, opinions and motivations – as opposed to facts or behaviours [ 13 ]. Interviews can be distinguished by the degree to which they are structured (i.e. a questionnaire), open (e.g. free conversation or autobiographical interviews) or semi-structured [ 2 , 13 ]. Semi-structured interviews are characterized by open-ended questions and the use of an interview guide (or topic guide/list) in which the broad areas of interest, sometimes including sub-questions, are defined [ 19 ]. The pre-defined topics in the interview guide can be derived from the literature, previous research or a preliminary method of data collection, e.g. document study or observations. The topic list is usually adapted and improved at the start of the data collection process as the interviewer learns more about the field [ 20 ]. Across interviews the focus on the different (blocks of) questions may differ and some questions may be skipped altogether (e.g. if the interviewee is not able or willing to answer the questions or for concerns about the total length of the interview) [ 20 ]. Qualitative interviews are usually not conducted in written format as it impedes on the interactive component of the method [ 20 ]. In comparison to written surveys, qualitative interviews have the advantage of being interactive and allowing for unexpected topics to emerge and to be taken up by the researcher. This can also help overcome a provider or researcher-centred bias often found in written surveys, which by nature, can only measure what is already known or expected to be of relevance to the researcher. Interviews can be audio- or video-taped; but sometimes it is only feasible or acceptable for the interviewer to take written notes [ 14 , 16 , 20 ].

Focus groups

Focus groups are group interviews to explore participants’ expertise and experiences, including explorations of how and why people behave in certain ways [ 1 ]. Focus groups usually consist of 6–8 people and are led by an experienced moderator following a topic guide or “script” [ 21 ]. They can involve an observer who takes note of the non-verbal aspects of the situation, possibly using an observation guide [ 21 ]. Depending on researchers’ and participants’ preferences, the discussions can be audio- or video-taped and transcribed afterwards [ 21 ]. Focus groups are useful for bringing together homogeneous (to a lesser extent heterogeneous) groups of participants with relevant expertise and experience on a given topic on which they can share detailed information [ 21 ]. Focus groups are a relatively easy, fast and inexpensive method to gain access to information on interactions in a given group, i.e. “the sharing and comparing” among participants [ 21 ]. Disadvantages include less control over the process and a lesser extent to which each individual may participate. Moreover, focus group moderators need experience, as do those tasked with the analysis of the resulting data. Focus groups can be less appropriate for discussing sensitive topics that participants might be reluctant to disclose in a group setting [ 13 ]. Moreover, attention must be paid to the emergence of “groupthink” as well as possible power dynamics within the group, e.g. when patients are awed or intimidated by health professionals.

Choosing the “right” method

As explained above, the school of thought underlying qualitative research assumes no objective hierarchy of evidence and methods. This means that each choice of single or combined methods has to be based on the research question that needs to be answered and a critical assessment with regard to whether or to what extent the chosen method can accomplish this – i.e. the “fit” between question and method [ 14 ]. It is necessary for these decisions to be documented when they are being made, and to be critically discussed when reporting methods and results.

Let us assume that our research aim is to examine the (clinical) processes around acute endovascular treatment (EVT), from the patient’s arrival at the emergency room to recanalization, with the aim to identify possible causes for delay and/or other causes for sub-optimal treatment outcome. As a first step, we could conduct a document study of the relevant standard operating procedures (SOPs) for this phase of care – are they up-to-date and in line with current guidelines? Do they contain any mistakes, irregularities or uncertainties that could cause delays or other problems? Regardless of the answers to these questions, the results have to be interpreted based on what they are: a written outline of what care processes in this hospital should look like. If we want to know what they actually look like in practice, we can conduct observations of the processes described in the SOPs. These results can (and should) be analysed in themselves, but also in comparison to the results of the document analysis, especially as regards relevant discrepancies. Do the SOPs outline specific tests for which no equipment can be observed or tasks to be performed by specialized nurses who are not present during the observation? It might also be possible that the written SOP is outdated, but the actual care provided is in line with current best practice. In order to find out why these discrepancies exist, it can be useful to conduct interviews. Are the physicians simply not aware of the SOPs (because their existence is limited to the hospital’s intranet) or do they actively disagree with them or does the infrastructure make it impossible to provide the care as described? Another rationale for adding interviews is that some situations (or all of their possible variations for different patient groups or the day, night or weekend shift) cannot practically or ethically be observed. In this case, it is possible to ask those involved to report on their actions – being aware that this is not the same as the actual observation. A senior physician’s or hospital manager’s description of certain situations might differ from a nurse’s or junior physician’s one, maybe because they intentionally misrepresent facts or maybe because different aspects of the process are visible or important to them. In some cases, it can also be relevant to consider to whom the interviewee is disclosing this information – someone they trust, someone they are otherwise not connected to, or someone they suspect or are aware of being in a potentially “dangerous” power relationship to them. Lastly, a focus group could be conducted with representatives of the relevant professional groups to explore how and why exactly they provide care around EVT. The discussion might reveal discrepancies (between SOPs and actual care or between different physicians) and motivations to the researchers as well as to the focus group members that they might not have been aware of themselves. For the focus group to deliver relevant information, attention has to be paid to its composition and conduct, for example, to make sure that all participants feel safe to disclose sensitive or potentially problematic information or that the discussion is not dominated by (senior) physicians only. The resulting combination of data collection methods is shown in Fig.  2 .

Possible combination of data collection methods

Attributions for icons: “Book” by Serhii Smirnov, “Interview” by Adrien Coquet, FR, “Magnifying Glass” by anggun, ID, “Business communication” by Vectors Market; all from the Noun Project

The combination of multiple data source as described for this example can be referred to as “triangulation”, in which multiple measurements are carried out from different angles to achieve a more comprehensive understanding of the phenomenon under study [ 22 , 23 ].

Data analysis

To analyse the data collected through observations, interviews and focus groups these need to be transcribed into protocols and transcripts (see Fig.  3 ). Interviews and focus groups can be transcribed verbatim , with or without annotations for behaviour (e.g. laughing, crying, pausing) and with or without phonetic transcription of dialects and filler words, depending on what is expected or known to be relevant for the analysis. In the next step, the protocols and transcripts are coded , that is, marked (or tagged, labelled) with one or more short descriptors of the content of a sentence or paragraph [ 2 , 15 , 23 ]. Jansen describes coding as “connecting the raw data with “theoretical” terms” [ 20 ]. In a more practical sense, coding makes raw data sortable. This makes it possible to extract and examine all segments describing, say, a tele-neurology consultation from multiple data sources (e.g. SOPs, emergency room observations, staff and patient interview). In a process of synthesis and abstraction, the codes are then grouped, summarised and/or categorised [ 15 , 20 ]. The end product of the coding or analysis process is a descriptive theory of the behavioural pattern under investigation [ 20 ]. The coding process is performed using qualitative data management software, the most common ones being InVivo, MaxQDA and Atlas.ti. It should be noted that these are data management tools which support the analysis performed by the researcher(s) [ 14 ].

From data collection to data analysis

Attributions for icons: see Fig. 2 , also “Speech to text” by Trevor Dsouza, “Field Notes” by Mike O’Brien, US, “Voice Record” by ProSymbols, US, “Inspection” by Made, AU, and “Cloud” by Graphic Tigers; all from the Noun Project

How to report qualitative research?

Protocols of qualitative research can be published separately and in advance of the study results. However, the aim is not the same as in RCT protocols, i.e. to pre-define and set in stone the research questions and primary or secondary endpoints. Rather, it is a way to describe the research methods in detail, which might not be possible in the results paper given journals’ word limits. Qualitative research papers are usually longer than their quantitative counterparts to allow for deep understanding and so-called “thick description”. In the methods section, the focus is on transparency of the methods used, including why, how and by whom they were implemented in the specific study setting, so as to enable a discussion of whether and how this may have influenced data collection, analysis and interpretation. The results section usually starts with a paragraph outlining the main findings, followed by more detailed descriptions of, for example, the commonalities, discrepancies or exceptions per category [ 20 ]. Here it is important to support main findings by relevant quotations, which may add information, context, emphasis or real-life examples [ 20 , 23 ]. It is subject to debate in the field whether it is relevant to state the exact number or percentage of respondents supporting a certain statement (e.g. “Five interviewees expressed negative feelings towards XYZ”) [ 21 ].

How to combine qualitative with quantitative research?

Qualitative methods can be combined with other methods in multi- or mixed methods designs, which “[employ] two or more different methods [ …] within the same study or research program rather than confining the research to one single method” [ 24 ]. Reasons for combining methods can be diverse, including triangulation for corroboration of findings, complementarity for illustration and clarification of results, expansion to extend the breadth and range of the study, explanation of (unexpected) results generated with one method with the help of another, or offsetting the weakness of one method with the strength of another [ 1 , 17 , 24 , 25 , 26 ]. The resulting designs can be classified according to when, why and how the different quantitative and/or qualitative data strands are combined. The three most common types of mixed method designs are the convergent parallel design , the explanatory sequential design and the exploratory sequential design. The designs with examples are shown in Fig.  4 .

Three common mixed methods designs

In the convergent parallel design, a qualitative study is conducted in parallel to and independently of a quantitative study, and the results of both studies are compared and combined at the stage of interpretation of results. Using the above example of EVT provision, this could entail setting up a quantitative EVT registry to measure process times and patient outcomes in parallel to conducting the qualitative research outlined above, and then comparing results. Amongst other things, this would make it possible to assess whether interview respondents’ subjective impressions of patients receiving good care match modified Rankin Scores at follow-up, or whether observed delays in care provision are exceptions or the rule when compared to door-to-needle times as documented in the registry. In the explanatory sequential design, a quantitative study is carried out first, followed by a qualitative study to help explain the results from the quantitative study. This would be an appropriate design if the registry alone had revealed relevant delays in door-to-needle times and the qualitative study would be used to understand where and why these occurred, and how they could be improved. In the exploratory design, the qualitative study is carried out first and its results help informing and building the quantitative study in the next step [ 26 ]. If the qualitative study around EVT provision had shown a high level of dissatisfaction among the staff members involved, a quantitative questionnaire investigating staff satisfaction could be set up in the next step, informed by the qualitative study on which topics dissatisfaction had been expressed. Amongst other things, the questionnaire design would make it possible to widen the reach of the research to more respondents from different (types of) hospitals, regions, countries or settings, and to conduct sub-group analyses for different professional groups.

How to assess qualitative research?

A variety of assessment criteria and lists have been developed for qualitative research, ranging in their focus and comprehensiveness [ 14 , 17 , 27 ]. However, none of these has been elevated to the “gold standard” in the field. In the following, we therefore focus on a set of commonly used assessment criteria that, from a practical standpoint, a researcher can look for when assessing a qualitative research report or paper.

Assessors should check the authors’ use of and adherence to the relevant reporting checklists (e.g. Standards for Reporting Qualitative Research (SRQR)) to make sure all items that are relevant for this type of research are addressed [ 23 , 28 ]. Discussions of quantitative measures in addition to or instead of these qualitative measures can be a sign of lower quality of the research (paper). Providing and adhering to a checklist for qualitative research contributes to an important quality criterion for qualitative research, namely transparency [ 15 , 17 , 23 ].

Reflexivity

While methodological transparency and complete reporting is relevant for all types of research, some additional criteria must be taken into account for qualitative research. This includes what is called reflexivity, i.e. sensitivity to the relationship between the researcher and the researched, including how contact was established and maintained, or the background and experience of the researcher(s) involved in data collection and analysis. Depending on the research question and population to be researched this can be limited to professional experience, but it may also include gender, age or ethnicity [ 17 , 27 ]. These details are relevant because in qualitative research, as opposed to quantitative research, the researcher as a person cannot be isolated from the research process [ 23 ]. It may influence the conversation when an interviewed patient speaks to an interviewer who is a physician, or when an interviewee is asked to discuss a gynaecological procedure with a male interviewer, and therefore the reader must be made aware of these details [ 19 ].

Sampling and saturation

The aim of qualitative sampling is for all variants of the objects of observation that are deemed relevant for the study to be present in the sample “ to see the issue and its meanings from as many angles as possible” [ 1 , 16 , 19 , 20 , 27 ] , and to ensure “information-richness [ 15 ]. An iterative sampling approach is advised, in which data collection (e.g. five interviews) is followed by data analysis, followed by more data collection to find variants that are lacking in the current sample. This process continues until no new (relevant) information can be found and further sampling becomes redundant – which is called saturation [ 1 , 15 ] . In other words: qualitative data collection finds its end point not a priori , but when the research team determines that saturation has been reached [ 29 , 30 ].

This is also the reason why most qualitative studies use deliberate instead of random sampling strategies. This is generally referred to as “ purposive sampling” , in which researchers pre-define which types of participants or cases they need to include so as to cover all variations that are expected to be of relevance, based on the literature, previous experience or theory (i.e. theoretical sampling) [ 14 , 20 ]. Other types of purposive sampling include (but are not limited to) maximum variation sampling, critical case sampling or extreme or deviant case sampling [ 2 ]. In the above EVT example, a purposive sample could include all relevant professional groups and/or all relevant stakeholders (patients, relatives) and/or all relevant times of observation (day, night and weekend shift).

Assessors of qualitative research should check whether the considerations underlying the sampling strategy were sound and whether or how researchers tried to adapt and improve their strategies in stepwise or cyclical approaches between data collection and analysis to achieve saturation [ 14 ].

Good qualitative research is iterative in nature, i.e. it goes back and forth between data collection and analysis, revising and improving the approach where necessary. One example of this are pilot interviews, where different aspects of the interview (especially the interview guide, but also, for example, the site of the interview or whether the interview can be audio-recorded) are tested with a small number of respondents, evaluated and revised [ 19 ]. In doing so, the interviewer learns which wording or types of questions work best, or which is the best length of an interview with patients who have trouble concentrating for an extended time. Of course, the same reasoning applies to observations or focus groups which can also be piloted.

Ideally, coding should be performed by at least two researchers, especially at the beginning of the coding process when a common approach must be defined, including the establishment of a useful coding list (or tree), and when a common meaning of individual codes must be established [ 23 ]. An initial sub-set or all transcripts can be coded independently by the coders and then compared and consolidated after regular discussions in the research team. This is to make sure that codes are applied consistently to the research data.

Member checking

Member checking, also called respondent validation , refers to the practice of checking back with study respondents to see if the research is in line with their views [ 14 , 27 ]. This can happen after data collection or analysis or when first results are available [ 23 ]. For example, interviewees can be provided with (summaries of) their transcripts and asked whether they believe this to be a complete representation of their views or whether they would like to clarify or elaborate on their responses [ 17 ]. Respondents’ feedback on these issues then becomes part of the data collection and analysis [ 27 ].

Stakeholder involvement

In those niches where qualitative approaches have been able to evolve and grow, a new trend has seen the inclusion of patients and their representatives not only as study participants (i.e. “members”, see above) but as consultants to and active participants in the broader research process [ 31 , 32 , 33 ]. The underlying assumption is that patients and other stakeholders hold unique perspectives and experiences that add value beyond their own single story, making the research more relevant and beneficial to researchers, study participants and (future) patients alike [ 34 , 35 ]. Using the example of patients on or nearing dialysis, a recent scoping review found that 80% of clinical research did not address the top 10 research priorities identified by patients and caregivers [ 32 , 36 ]. In this sense, the involvement of the relevant stakeholders, especially patients and relatives, is increasingly being seen as a quality indicator in and of itself.

How not to assess qualitative research

The above overview does not include certain items that are routine in assessments of quantitative research. What follows is a non-exhaustive, non-representative, experience-based list of the quantitative criteria often applied to the assessment of qualitative research, as well as an explanation of the limited usefulness of these endeavours.

Protocol adherence

Given the openness and flexibility of qualitative research, it should not be assessed by how well it adheres to pre-determined and fixed strategies – in other words: its rigidity. Instead, the assessor should look for signs of adaptation and refinement based on lessons learned from earlier steps in the research process.

Sample size

For the reasons explained above, qualitative research does not require specific sample sizes, nor does it require that the sample size be determined a priori [ 1 , 14 , 27 , 37 , 38 , 39 ]. Sample size can only be a useful quality indicator when related to the research purpose, the chosen methodology and the composition of the sample, i.e. who was included and why.

Randomisation

While some authors argue that randomisation can be used in qualitative research, this is not commonly the case, as neither its feasibility nor its necessity or usefulness has been convincingly established for qualitative research [ 13 , 27 ]. Relevant disadvantages include the negative impact of a too large sample size as well as the possibility (or probability) of selecting “ quiet, uncooperative or inarticulate individuals ” [ 17 ]. Qualitative studies do not use control groups, either.

Interrater reliability, variability and other “objectivity checks”

The concept of “interrater reliability” is sometimes used in qualitative research to assess to which extent the coding approach overlaps between the two co-coders. However, it is not clear what this measure tells us about the quality of the analysis [ 23 ]. This means that these scores can be included in qualitative research reports, preferably with some additional information on what the score means for the analysis, but it is not a requirement. Relatedly, it is not relevant for the quality or “objectivity” of qualitative research to separate those who recruited the study participants and collected and analysed the data. Experiences even show that it might be better to have the same person or team perform all of these tasks [ 20 ]. First, when researchers introduce themselves during recruitment this can enhance trust when the interview takes place days or weeks later with the same researcher. Second, when the audio-recording is transcribed for analysis, the researcher conducting the interviews will usually remember the interviewee and the specific interview situation during data analysis. This might be helpful in providing additional context information for interpretation of data, e.g. on whether something might have been meant as a joke [ 18 ].

Not being quantitative research

Being qualitative research instead of quantitative research should not be used as an assessment criterion if it is used irrespectively of the research problem at hand. Similarly, qualitative research should not be required to be combined with quantitative research per se – unless mixed methods research is judged as inherently better than single-method research. In this case, the same criterion should be applied for quantitative studies without a qualitative component.

The main take-away points of this paper are summarised in Table 1 . We aimed to show that, if conducted well, qualitative research can answer specific research questions that cannot to be adequately answered using (only) quantitative designs. Seeing qualitative and quantitative methods as equal will help us become more aware and critical of the “fit” between the research problem and our chosen methods: I can conduct an RCT to determine the reasons for transportation delays of acute stroke patients – but should I? It also provides us with a greater range of tools to tackle a greater range of research problems more appropriately and successfully, filling in the blind spots on one half of the methodological spectrum to better address the whole complexity of neurological research and practice.

Availability of data and materials

Not applicable.

Abbreviations

Endovascular treatment

Randomised Controlled Trial

Standard Operating Procedure

Standards for Reporting Qualitative Research

Philipsen, H., & Vernooij-Dassen, M. (2007). Kwalitatief onderzoek: nuttig, onmisbaar en uitdagend. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Qualitative research: useful, indispensable and challenging. In: Qualitative research: Practical methods for medical practice (pp. 5–12). Houten: Bohn Stafleu van Loghum.

Chapter   Google Scholar  

Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches . London: Sage.

Kelly, J., Dwyer, J., Willis, E., & Pekarsky, B. (2014). Travelling to the city for hospital care: Access factors in country aboriginal patient journeys. Australian Journal of Rural Health, 22 (3), 109–113.

Article   Google Scholar  

Nilsen, P., Ståhl, C., Roback, K., & Cairney, P. (2013). Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Science, 8 (1), 1–12.

Howick J, Chalmers I, Glasziou, P., Greenhalgh, T., Heneghan, C., Liberati, A., Moschetti, I., Phillips, B., & Thornton, H. (2011). The 2011 Oxford CEBM evidence levels of evidence (introductory document) . Oxford Center for Evidence Based Medicine. https://www.cebm.net/2011/06/2011-oxford-cebm-levels-evidence-introductory-document/ .

Eakin, J. M. (2016). Educating critical qualitative health researchers in the land of the randomized controlled trial. Qualitative Inquiry, 22 (2), 107–118.

May, A., & Mathijssen, J. (2015). Alternatieven voor RCT bij de evaluatie van effectiviteit van interventies!? Eindrapportage. In Alternatives for RCTs in the evaluation of effectiveness of interventions!? Final report .

Google Scholar  

Berwick, D. M. (2008). The science of improvement. Journal of the American Medical Association, 299 (10), 1182–1184.

Article   CAS   Google Scholar  

Christ, T. W. (2014). Scientific-based research and randomized controlled trials, the “gold” standard? Alternative paradigms and mixed methodologies. Qualitative Inquiry, 20 (1), 72–80.

Lamont, T., Barber, N., Jd, P., Fulop, N., Garfield-Birkbeck, S., Lilford, R., Mear, L., Raine, R., & Fitzpatrick, R. (2016). New approaches to evaluating complex health and care systems. BMJ, 352:i154.

Drabble, S. J., & O’Cathain, A. (2015). Moving from Randomized Controlled Trials to Mixed Methods Intervention Evaluation. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry (pp. 406–425). London: Oxford University Press.

Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science : IS, 8 , 117.

Hak, T. (2007). Waarnemingsmethoden in kwalitatief onderzoek. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Observation methods in qualitative research] (pp. 13–25). Houten: Bohn Stafleu van Loghum.

Russell, C. K., & Gregory, D. M. (2003). Evaluation of qualitative research studies. Evidence Based Nursing, 6 (2), 36–40.

Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating qualitative research. Australian and New Zealand Journal of Psychiatry, 36 , 717–732.

Yanow, D. (2000). Conducting interpretive policy analysis (Vol. 47). Thousand Oaks: Sage University Papers Series on Qualitative Research Methods.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 , 63–75.

van der Geest, S. (2006). Participeren in ziekte en zorg: meer over kwalitatief onderzoek. Huisarts en Wetenschap, 49 (4), 283–287.

Hijmans, E., & Kuyper, M. (2007). Het halfopen interview als onderzoeksmethode. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [The half-open interview as research method (pp. 43–51). Houten: Bohn Stafleu van Loghum.

Jansen, H. (2007). Systematiek en toepassing van de kwalitatieve survey. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Systematics and implementation of the qualitative survey (pp. 27–41). Houten: Bohn Stafleu van Loghum.

Pv, R., & Peremans, L. (2007). Exploreren met focusgroepgesprekken: de ‘stem’ van de groep onder de loep. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Exploring with focus group conversations: the “voice” of the group under the magnifying glass (pp. 53–64). Houten: Bohn Stafleu van Loghum.

Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. J. (2014). The use of triangulation in qualitative research. Oncology Nursing Forum, 41 (5), 545–547.

Boeije H: Analyseren in kwalitatief onderzoek: Denken en doen, [Analysis in qualitative research: Thinking and doing] vol. Den Haag Boom Lemma uitgevers; 2012.

Hunter, A., & Brewer, J. (2015). Designing Multimethod Research. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry (pp. 185–205). London: Oxford University Press.

Archibald, M. M., Radil, A. I., Zhang, X., & Hanson, W. E. (2015). Current mixed methods practices in qualitative research: A content analysis of leading journals. International Journal of Qualitative Methods, 14 (2), 5–33.

Creswell, J. W., & Plano Clark, V. L. (2011). Choosing a Mixed Methods Design. In Designing and Conducting Mixed Methods Research . Thousand Oaks: SAGE Publications.

Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. BMJ, 320 (7226), 50–52.

O'Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine : Journal of the Association of American Medical Colleges, 89 (9), 1245–1251.

Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018). Saturation in qualitative research: Exploring its conceptualization and operationalization. Quality and Quantity, 52 (4), 1893–1907.

Moser, A., & Korstjens, I. (2018). Series: Practical guidance to qualitative research. Part 3: Sampling, data collection and analysis. European Journal of General Practice, 24 (1), 9–18.

Marlett, N., Shklarov, S., Marshall, D., Santana, M. J., & Wasylak, T. (2015). Building new roles and relationships in research: A model of patient engagement research. Quality of Life Research : an international journal of quality of life aspects of treatment, care and rehabilitation, 24 (5), 1057–1067.

Demian, M. N., Lam, N. N., Mac-Way, F., Sapir-Pichhadze, R., & Fernandez, N. (2017). Opportunities for engaging patients in kidney research. Canadian Journal of Kidney Health and Disease, 4 , 2054358117703070–2054358117703070.

Noyes, J., McLaughlin, L., Morgan, K., Roberts, A., Stephens, M., Bourne, J., Houlston, M., Houlston, J., Thomas, S., Rhys, R. G., et al. (2019). Designing a co-productive study to overcome known methodological challenges in organ donation research with bereaved family members. Health Expectations . 22(4):824–35.

Piil, K., Jarden, M., & Pii, K. H. (2019). Research agenda for life-threatening cancer. European Journal Cancer Care (Engl), 28 (1), e12935.

Hofmann, D., Ibrahim, F., Rose, D., Scott, D. L., Cope, A., Wykes, T., & Lempp, H. (2015). Expectations of new treatment in rheumatoid arthritis: Developing a patient-generated questionnaire. Health Expectations : an international journal of public participation in health care and health policy, 18 (5), 995–1008.

Jun, M., Manns, B., Laupacis, A., Manns, L., Rehal, B., Crowe, S., & Hemmelgarn, B. R. (2015). Assessing the extent to which current clinical research is consistent with patient priorities: A scoping review using a case study in patients on or nearing dialysis. Canadian Journal of Kidney Health and Disease, 2 , 35.

Elsie Baker, S., & Edwards, R. (2012). How many qualitative interviews is enough? In National Centre for Research Methods Review Paper . National Centre for Research Methods. http://eprints.ncrm.ac.uk/2273/4/how_many_interviews.pdf .

Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18 (2), 179–183.

Sim, J., Saunders, B., Waterfield, J., & Kingstone, T. (2018). Can sample size in qualitative research be determined a priori? International Journal of Social Research Methodology, 21 (5), 619–634.

Download references

Acknowledgements

no external funding.

Author information

Authors and affiliations.

Department of Neurology, Heidelberg University Hospital, Im Neuenheimer Feld 400, 69120, Heidelberg, Germany

Loraine Busetto, Wolfgang Wick & Christoph Gumbinger

Clinical Cooperation Unit Neuro-Oncology, German Cancer Research Center, Heidelberg, Germany

Wolfgang Wick

You can also search for this author in PubMed   Google Scholar

Contributions

LB drafted the manuscript; WW and CG revised the manuscript; all authors approved the final versions.

Corresponding author

Correspondence to Loraine Busetto .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Busetto, L., Wick, W. & Gumbinger, C. How to use and assess qualitative research methods. Neurol. Res. Pract. 2 , 14 (2020). https://doi.org/10.1186/s42466-020-00059-z

Download citation

Received : 30 January 2020

Accepted : 22 April 2020

Published : 27 May 2020

DOI : https://doi.org/10.1186/s42466-020-00059-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Mixed methods
• Quality assessment

Neurological Research and Practice

ISSN: 2524-3489

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Research article
• Open access
• Published: 18 May 2020

What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis

• Oliver Rudolf HERBER   ORCID: orcid.org/0000-0003-3041-4098 1 ,
• Caroline BRADBURY-JONES 2 ,
• Susanna BÖLING 3 ,
• Sarah COMBES 4 ,
• Julian HIRT 5 ,
• Yvonne KOOP 6 ,
• Ragnhild NYHAGEN 7 ,
• Jessica D. VELDHUIZEN 8 &
• Julie TAYLOR 2 , 9  

BMC Medical Research Methodology volume  20 , Article number:  122 ( 2020 ) Cite this article

15k Accesses

16 Citations

34 Altmetric

Metrics details

Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central ( BMC ) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers’ comments.

A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January – 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports.

A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes “writing criteria” (dimension II) is the top ranking theme, followed by comments in relation to the “methods” (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts.

Conclusions

This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called “Degree of data transformation” to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.

Peer Review reports

Peer review is at the heart of the scientific process. Reviewers independently examine a submitted manuscript and then recommend acceptance, rejection or – most frequently – revisions to be made before it gets published [ 1 ]. Editors rely on peer review to make decisions on which submissions warrant publication and to enhance quality standards. Typically, each manuscript is reviewed by two or three reviewers [ 2 ] who are chosen for their knowledge and expertise regarding the subject or methodology [ 3 ]. The history of peer review, often regarded as a “touchstone of modern evaluation of scientific quality” [ 4 ] is relatively short. For example, the British Medical Journal (now the BMJ ) was a pioneer when it established a system of external reviewers in 1893. But it was in the second half of the twentieth century that employing peers as reviewers became custom [ 5 ]. Then, in 1973 the prestigious scientific weekly Nature introduced a rigorous formal peer review system for every paper it printed [ 6 ].

Despite ever-growing concerns about its effectiveness, fairness and reliability [ 4 , 7 ], peer review as a central part of academic self-regulation is still considered the best available practice [ 8 ]. With the advent of digitisation in the late 1990s, scholarly publishing has changed dramatically with many journals starting to offer print as well as electronic articles or publishing online only [ 9 ]. The latter category includes for-profit journals such as BioMed Central ( BMC ) that have been online since their inception in 1999, with an ever evolving portfolio of currently over 300 peer-reviewed journals.

As compared to traditional print journals where individuals or libraries need to pay a fee for an annual subscription or for reading a specific article, open access journals such as BMC, PLoS ONE or BMJ Open are permanently free for everyone to read and download since the cost of publishing is paid by the author or an entity such as the university. Many, but not all, open access journals impose an article processing charge on the author, also known as the gold open access route, to cover the cost of publication. Depending on the journal and the publisher, article processing charges can range significantly between US$100 and US$5200 per article [ 10 , 11 ].

In the digital age, a new philosophy regarding the peer review process found its way into academia, questioning the anonymity of the closed system of peer-review as contrary to the demands for transparency [ 1 ]. The issue of reviewer bias, especially concerning gender and affiliation [ 12 ], led not only to the establishment of double-blind review but also to its extreme opposite: the open peer review system [ 8 ]. Although the term ‘open peer review’ has no standardised definition, scholars use the term to indicate that the identities of the authors and reviewers are disclosed and that reviewer reports are openly available [ 13 ]. In the late 1990s, the BMJ changed from a closed system of peer review to an open system [ 14 , 15 ]. During the same time, other publishers such as some journals in BMC followed the example of opening up their peer review.

While peer review reports have long been hidden from the public gaze [ 16 , 17 ], opening up the closed peer review system allows researchers to access reviewer comments, thus making it possible to study them. Since then, a number of articles have been published to assess reviews using quantitative research methods. For example, Landkroon et al. [ 18 ] assessed the quality of 247 reviews of 119 original articles using a 5-point Likert scale. Similarly, Henly and Dougherty [ 19 ] developed and applied a grading scale to assess the narrative portion of 464 reviews of 203 manuscripts using descriptive statistics. The retrospective cohort study by van Lent et al. [ 20 ] assessed peer review comments on drug trials from 246 manuscripts to investigate whether there is a relationship between the content of these comments and sponsorship using a generalised linear mixed model. Most recently, Davis et al. [ 21 ] evaluated reviewer grading forms for surgical journals with higher impact factors and compared them to surgical journals with lower impact factors using Fisher’s exact test.

Despite the readily available reviewer comments that are published alongside the final article of many open access journals, to the best of our knowledge no studies exist to date that used – besides quantitative methods – also qualitative methods to analyse the content of reviewers’ comments. Identifying (negative) reviewer comments will help authors to pay particular attention to these aspects and assist prospective qualitative researchers to understand the most common pitfalls when preparing their manuscript for submission. Thus, the aim of the study was to appraise the quality and nature of reviewers’ feedback in order to understand how reviewers engage with and influence the development of a qualitative manuscript. Our focus on qualitative research can be explained by the fact that we are passionate qualitative researchers with a history in determining the state of qualitative research in health and social science literature [ 22 ]. The following research questions were answered: (1) What are the frequencies of certain commentary types in manuscripts reporting on qualitative research? and (2) What are the nature of reviewers’ comments made on manuscripts reporting on qualitative research?

We conducted a focused mapping review and synthesis (FMRS) [ 22 , 23 , 24 , 25 ]. Most forms of review aim for breadth and exhaustive searches, but the FMRS searches within specific, pre-determined journals. While Platt [ 26 ] observed that ‘a number of studies have used samples of journal articles’, the distinctive feature of the FMRS is the purposive selection of journals. These are chosen for their likelihood to contain articles relevant to the field of inquiry – in this case qualitative research published in open access journals that operate an open peer-review process that involves posting the reviewer’s reports. It is these reports that we have analysed using thematic analysis techniques [ 27 ].

Currently there are over 70 BMC journals that have adopted open peer-review. The FMRS focused on reviewers’ reports published during the first quarter of 2018. Journals were selected using a three-stage process. First, we produced a list with all BMC journals that operate an open peer review process and will publish qualitative research articles ( n  = 62). Second, from this list we selected journals that are general fields of practice and non-disease specific ( n  = 15). Third, to ensure a sufficient number of qualitative articles, we excluded journals with less than 25 hits on the search term “qualitative” for the year 2018 (search date: 16 July 2018) because chances were considered too slim to contain sufficient articles of interest. At the end of the selection process, the following nine BMC journals were included in our synthesis: (1) BMC Complementary and Alternative Medicine , (2) BMC Family Practice , (3) BMC Health Services Research , (4) BMC Medical Education , (5) BMC Medical Ethics , (6) BMC Nursing , (7) BMC Public Health , (8) Health Research Policy and Systems , and (9) Implementation Science . Since these journals represent different subjects, a variety of qualitative papers written for different audiences was captured. Every article published within the timeframe was scrutinised against the inclusion and exclusion criteria (Table  1 ).

Development of the data extraction sheet

A validated instrument for the classification of reviewer comments does not exist [ 20 ]. Hence, a detailed classification system was developed and pilot tested considering previous research [ 20 ]. Our newly developed data extraction sheet consists of a 77-item classification system organised according to three dimensions: (1) scientific/technical content, (2) writing criteria/representation, and (3) technical criteria. It represents themes and sub-themes identified by reading reviewer comments from twelve articles published in open peer-review journals. For the development of the data extraction sheet, we randomly selected four articles containing qualitative research from each of the following three journals published between 2017 and 2018: BMC Nursing , BMC Family Practice and BMJ Open . We then analysed the reviews of manuscripts by systematically coding and categorising the reviewers’ free-text comments. Following the recommendation by Shashok [ 28 ], we initially organised the reviewer’s comments along two main dimensions, i.e., scientific content and writing criteria. Shashok [ 28 ] argues that when peer reviewers confuse content and writing, their feedback can be misunderstood by authors who may modify texts in unintentional ways to the detriment of the manuscript.

To check the comprehensiveness of our classification system, provisional themes and sub-themes were piloted using reviewer comments we had previously received from twelve of our own manuscripts that had been submitted to journals that operate blind peer-review. We wanted to account for potential differences in reviewers’ feedback (open vs. blind review). As a result of this quality enhancement procedure, three sub-themes and a further dimension (‘technical criteria’) were added. For reasons of clarity and comprehensibility, the dimension ‘scientific content’ was subdivided following the IMRaD structure. IMRaD is the most common organisational structure of an original research article comprising I ntroduction, M ethods, R esults a nd D iscussion [ 29 ]. Anchoring examples were provided for each theme/sub-theme. To account for reviewer comments unrelated to the IMRaD structure, a sub-category called ‘generic codes’ was created to collect more general comments. When reviewer comments could not be assigned to any of the existing themes/sub-themes, they were noted as “Miscellaneous”. Table  2 shows the final data extraction sheet including anchoring examples.

Data extraction procedure

Data extraction was accomplished by six doctoral students (coders). On average, each coder was allocated 18 articles. After reading the reviews, coders independently classified each comment using the classification system. In line with Day et al. [ 30 ] a reviewer comment was defined as “ a distinct statement or idea found in a review, regardless of whether that statement was presented in isolation or was included in a paragraph that contained several statements. ” Editor comments were not included. Reviewers’ comments were copied and pasted into the most appropriate item of the classification system following a set of pre-defined guidelines. For example, a reviewer comment could only be coded once by assigning it to the most appropriate theme/sub-theme. A separate data extraction sheet was used for each article. For the purpose of calibration, the first completed data extraction sheet from each coder together with the reviewer’s comments was sent to the study coordinator (ORH) who provided feedback on classifying the reviewer comments. The aim of the calibration was to ensure that all coders were working within the same parameters of understanding, to discuss the subtleties of the judgement process and create consensus regarding classifications. Although the assignment to specific themes/sub-themes is, by nature, a subjective process, difficult to assign comments were classified following discussion and agreement between coder and study coordinator to ensure reliability. Once all data extraction was completed, two experienced qualitative researchers (CB-J, JT) independently undertook a further calibration exercise of a random sub-sample of 20% of articles ( n  = 22) to ensure consistency across coders. Articles were selected using a random number generator. For these 22 articles, classification discrepancies were resolved by consensus between coders and experienced researchers. Finally, all individual data extraction sheets were collated to create a comprehensive Excel spreadsheet with over 8000 cells that allowed tallying the reviewer’s comments across manuscripts for the purpose of data analysis. For each manuscript, a reviewer could have several remarks related to one type of comment. However, each type of comment was scored only once per category.

Finally, reviewer comments were ‘quantitized’ [ 31 ] by applying programming language (Python) to Jupyter Notebook, an open-source web application, to perform frequency counts of free-text comments regarding the 77 items. Among other data manipulation, we sorted elements of arrays in descending order of frequency using Pandas, counted the number of studies in which a certain theme/sub-theme occurred, conducted distinct word searches using NLTK 3 or grouped data according to certain criteria. The calculation of frequencies is a way to unite the empirical precision of quantitative research with the descriptive precision of qualitative research [ 32 ]. This quantitative transformation of qualitative data allowed extracting more meaning from our spreadsheet through revealing patterns across themes/sub-themes, thus giving indicators about which of them to analyse using thematic analysis.

A total of 109 manuscripts submitted to nine open-access journals were included in the FMRS. When scrutinising the peer review reports, we noticed that on one occasion the reviewer’s comments were missing [ 33 ]. For the remaining 108 manuscripts, reviewer comments were accessible via the journal’s pre-publication history. On close inspection, however, it became apparent that one article did not contain qualitative research, thus leaving ultimately 107 articles to work with ( supplementary file ). Considering that each manuscript could potentially be reviewed by multiple reviewers and underwent at least one round of revision, the total number of reviewer reports analysed amounted to 347 containing collectively 1703 reviewer comments. The level of inter-rater agreement for the 22 articles included in the calibration exercise was 97%. Disagreement was, for example, in relation to coding a comment as “miscellaneous” or as “confirmation/approval (from reviewer)”. For 18 out of 22 articles, there was 100% agreement for all types of comments.

Variation in number of reviewers

The number of reviewers invited by the editor to review a submitted manuscript varied greatly within and among journals. While the majority of manuscripts across journals had been reviewed by two to three reviewers, there were also significant variations. For example, the manuscript submitted to BMC Medical Education by Burgess et al. [ 34 ] had been reviewed by five reviewers whereas the manuscript submitted to BMC Public Health by Lee and Lee [ 35 ] had been reviewed by one reviewer only. Even within journals there was a huge variation. Among our sample, BMC Public Health had the greatest variance ranging from one to four reviewers. Besides, it was noted that additional reviewers were called in not until the second or even third revision of the manuscript. A summary of key information on journals included in the FMRS is provided in Table  3 .

“Quantitizing” reviewer comments

The frequency analysis revealed that the number of articles in which a certain theme/sub-theme occurred ranged from 1 to 79. Across all 107 articles, the types of comments most frequently reported were in relation to generic themes. Reviewer comments regarding “Adding information/detail/nuances”, “Clarification needed”, “Further explanation required” and “Confirmation/approval (from reviewer)” were used in 79, 79, 66 and 63 articles, respectively. The four most frequently used themes/sub-themes are composed of generic codes from dimension I (“Scientific/technical content”). Leaving all generic codes aside, it became apparent that among the 30 most frequently employed themes “Writing criteria” (dimension II) is the top ranking theme, followed by comments in relation to the “Methods” (dimension I) (Table  4 ).

Subsequently, we present key qualitative findings regarding “Confirmation/approval from reviewers” (generic), “Sampling” and “Analysis process” (methods), “Robust/rich data analysis and “Themes/sub-themes” (results) as well as findings that suggest an underlying quantitative mindset of the reviewers.

Confirmation/approval from reviewers (generic)

The theme “confirmation/approval from reviewers” ranks third among the top 30 categories. A total of 63 manuscripts contained at least one reviewer comment related to this theme. Overall, reviewers maintained a respectful and affirmative rhetoric when providing feedback. The vast majority of reviewers began their report by stating that the manuscript was well written. The following is a typical example:

“Overall, the paper is well written, and theoretically informed.” Article #14.

Reviewers then continued to add explicit praise for aspects or sections that were particularly innovative and/or well constructed before they started to put forward any negative feedback.

Sampling (methods)

Across all 107 articles there were 34 reviewer comments in relation to the sampling technique(s). Two major categories were identified: (1) composition of the sample and (2) identification and justification of selected participants. Regarding the former, reviewers raised several concerns about how the sample was composed. For instance, one reviewer wanted to know the reason for female predominance in the study and why an entire focus group was composed of females only. Another reviewer expressed strong criticism on the composition of the sample since only young, educated and non-minority white British participants were included in the study. The reviewer commented:

“ So a typical patient was young, educated and non-minority White British? The research studies these days should be inclusive of diverse types of patients and excluding patients because of their age and ethnicity is extremely concerning to me. This assumption that these individuals will “find it more difficult to complete questionnaires” is concerning ” Article #40.

This raised concerns of potentially excluding important diverse perspectives – such as extreme or deviant cases – from other participants. Similarly, some reviewers expressed concerns that relevant groups of people were not interviewed, calling into question that the findings were theoretically saturated. In terms of the identification of participants, reviewers raised questions regarding how the authors obtained the necessary characteristics to achieve purposive sampling or why only certain groups of people were included for interviews. Besides that, reviewers criticised that some authors did not mention their inclusion/exclusion criteria for selecting participants or did not specify their sampling method. For example:

“The authors state that they recruited a purposive sample of patients for the interviews. Concerning which variables was this sampling purposive? Are there any studies informing the patient selection process?” Article #61.

Hence, reviewers requested more detailed information on how participants were selected and to clearly state the type of sampling. Apart from the two key categories, reviewers made additional comments in relation to data saturation, transferability of findings, limitations of certain sampling methods and criticised the lack of description of participants who were approached but refused to participate in the study.

Details of analysis process (methods)

In 60 out of 107 articles, reviewers made comments in relation to the data analysis. The vast majority of comments stressed that authors provided scarce information about the analysis process. Hence, reviewers requested a more detailed description of the specific analysis techniques employed so that readers can obtain a better understanding of how the analysis was done to judge the trustworthiness of the findings. To this end, reviewers frequently requested an explicit statement on whether the analysis was inductive or deductive or iterative or sequential. One reviewer wrote the following comment:

“Please elaborate more on the qualitative analysis. The authors indicate that they used ‘iterative’ approaches. While this is certainly laudable, it is important to know how they moved from codes to themes (e.g. inductively? deductively?)” Article #5.

Since there are many approaches to analysing qualitative data, reviewers demanded sufficient detail in relation to the underlying theoretical framework used to develop the coding scheme, the analytic process, the researchers’ background (e.g. profession), the number of coders, data handling, length of interviews and whether data saturation occurred. Over a dozen reviewer comments were specifically in relation to the identification of themes/sub-themes. Reviewers requested a more detailed description on how the themes/sub-themes were derived from codes and whether they were developed by a second researcher working independently from each other.

“I would have liked to read how their themes were generated, what they were and how they assured robust practices in qualitative data analysis”. Article #43.

Besides that, some reviewers were in the opinion that the approach to analysis has led to a surface-level penetration of the data which was reflected in the Results section where themes were underexplored (for more detail see “ Robust/rich data analysis” below). Finally, reviewer comments that occurred infrequently included questions concerning the inter-rater reliability, competing interpretations of data, the use of computer software or the original interview language.

Robust/rich data analysis (results)

Among the 30 reviewer comments related to this theme/sub-theme, three key facets were observed: (1) greater analytical depth required, (2) suggestions for further analysis, and (3) themes are underexplored. In relation to the first point, reviewers requested more in-depth data analysis to strengthen the quality of the manuscript. Reviewers were in the opinion that authors reproduced interview data (raw data) in a reduced form with minimal or no interpretation, thus leaving the interpretation to the reader. Other reviewers referred to manuscripts as preliminary drafts that need to be further analysed to achieve greater analytical depth of themes, make links between themes or identify variations between respondents. In relation to the second point, several reviewers offered suggestions for further analysis. They provided detailed information on how to further explore the data and what additional results they would like to see in the revised version (e.g. group comparison, gender analysis). The latter aspect goes hand in hand with the third point. Several reviewers pointed out that the findings were shallow, simplistic or superficial at best; lacking the detailed descriptions of complex accounts from participants. For example:

“The results of the study are mostly descriptive and there is limited analysis. There is also absence of thick description, which one would expect in a qualitative study”. Article #34.

Even after the first revision, some manuscripts still lacked detailed analysis as the following comment from the same reviewer illustrates:

“I believe that the results in the revised version are still mostly descriptive and that there is limited analysis”. Article #34, R1.

Other, less frequently mentioned reviewer comments included lack of deviant cases or absence of relationships between themes.

Themes/sub-themes (results)

In total, there were 24 reviewer comments in relation to themes/sub-themes. More than half of the comments fell into one of the three categories: (1) themes/sub-themes are not sufficiently supported by data, (2) example/excerpt does not fit the stated theme, and (3) use of insufficient quotes to support theme/sub-theme. In relation to the first category, reviewers largely criticised that the data provided were insufficient to warrant being called a theme. Reviewers requested to provide data “from more than just one participant” to substantiate a certain theme or criticised that only a short excerpt was provided to support a theme. The second category dealt with reviewer comments that questioned whether the excerpts provided actually reflected the essence of a theme/sub-theme presented in the results section. The following reviewer comment exemplifies the issue:

“The data themes seem valid, but the data and narratives used to illustrate that don’t seem to fit entirely under each sub-heading”. Article #99.

Some reviewers provided alternative suggestions on how to call a theme/sub-theme or advised the authors to rethink if excerpts might be better placed under a different theme. The third category concerns themes/sub-themes that are not sufficiently supported by participants’ quotes. Reviewers perceived direct quotes as evidence to support a certain theme or as a means to add strength to the theme as the following example illustrates:

“Please provide at least one quote from each school leader and one quote from children to support this theme, if possible. It would seem that most, if not all, themes should reflect data from each participant group”. Article #88.

Hence, the absence of quotes prompted reviewers to request at least one quote to justify the existence of that theme. The inclusion of a rich set of quotes was perceived as strength of a manuscript. Finally, less frequently raised reviewer comments related to the discrimination of similar themes, the presentation of quotes in tables (rather than under the appropriate theme headings), the lack of defining a theme and reducing the number of themes.

Quantitative mindset

Some reviewers who were appointed by journal editors to review a manuscript containing qualitative research evaluated the quality of the manuscript from a perspective of a quantitative research paradigm. Some reviewers not only used terminology that is attuned to quantitative research, but also their judgements were based on a quantitative mindset. In particular, there were a number of reviewer comments published in BMC Health Services Research , BMC Medical Education and BMC Family Practice that demonstrated an apparent lack of understanding of the principles underlying qualitative inquiry of the person providing the review. First, several reviewers seemed to have confused the concept of generalisability with the concept of representativeness inherently associated with the positivist tradition. For instance, reviewers erroneously raised concerns about whether interviewees were “representative” of the “final target population” and requested the provision of detailed demographic characteristics.

“Need to better describe how the patients are representative of patients with chronic heart failure in the Netherlands generally. The declaration that “a representative group of patients were recruited” would benefit from stating what they were representative of.” Article # 66.

Similarly, another reviewer wanted to know from the authors how they ensured that the qualitative analysis was done objectively.

“The reader would benefit from a detailed description of […] how did the investigators ensure that they were objective in their analysis – objectivity and trustworthiness?” Article #22.

Furthermore, despite the fact that the paradigm wars have largely come to an end, hostility has not ceased on all fronts. In some reviewers the dominance and superiority of the quantitative paradigm over the qualitative paradigm is still present as the following comment illustrates:

“The main question and methods of this article is largely qualitative and does not seem to have significant implications for clinical practice, thus it may not be suitable to publish in this journal.” Article #45.

Finally, one reviewer apologised at the outset of the reviewer’s report for being unable to judge the data analysis due to the absence of sufficient knowledge in qualitative research.

Overall, in this FMRS we found that reviewers maintained a respectful and affirmative rhetoric when providing feedback. Yet, the positive feedback did not overshadow any key negative points that needed to be addressed in order to increase the quality of the manuscript. However, it should not be taken for granted that all reviewers are as courteous and generous as the ones included in our particular review, because as Taylor and Bradbury-Jones [ 36 ] observed there are many examples where reviewers can be unhelpful and destructive in their comments.

A key finding of this FMRS is that reviewers are more inclined to comment on the writing rather than the methodological rigour of a manuscript. This is a matter of concern, because Altman [ 37 ] – the originator of the EQUATOR (Enhancing the Quality and Transparency of Health Research) Network – has pointed out: “Unless methodology is described the conclusions must be suspect”. If we are to advance the quality of qualitative research then we need to encourage clarity and depth in reporting the rigour of research.

When reviewers did comment on the methodological aspects of an article, issues frequently commented on by reviewers were in relation to sampling, data analysis, robust/rich data analysis as reflected in the findings and themes/sub-themes that are insufficiently supported. Considerable work has been undertaken over the past decade trying to improve the reporting standards of qualitative research through the dissemination of qualitatively oriented reporting guidelines such as the ‘Standards for Reporting Qualitative Research’ (SRQR) [ 38 ] or the ‘Consolidated Criteria for Reporting Qualitative Research’ (COREQ) [ 39 ] with the aim of improving transparency of qualitative research. Although these guidelines appear to be comprehensive, some important issues identified in our study are not mentioned or only dealt with somewhat superficially: sampling for example. Neither COREQ nor SRQR shed light on the appropriateness of the sample composition, i.e., to critically question whether all relevant groups of people have been identified as potential participants or whether extreme or deviant cases were sought.

Similarly, lack of in-depth data analysis has been identified as another weakness where uninterpreted (raw) data were presented as if they were findings. However, existing reporting guidelines are not sharp enough to distinguish between findings and data. While findings are researchers’ interpretations of the data they collected, data consist of empirical, uninterpreted material researchers offer as their findings [ 32 ]. Hence, we suggest modifying the current reporting guidelines by including a further item to the checklist called “Degree of data transformation”. The suggested checklist item might prompt both authors and reviewers to make a judgment about the degree to which data have been transformed, i.e., interpretively removed from data as given. The rationale for the new item is to raise authors’ and reviewers’ awareness for the appropriateness of the degree of data transformation in relation to the chosen analysis method. For example, findings derived from content analysis remain close to the data as they were given to the research; they are often organised into surface classification systems and summarised in brief text. Findings derived from grounded theory, however, should offer a coherent model or line of argument which addresses causality or the fundamental nature of events or experiences [ 32 ].

Besides that, some reviewers put forward comments that we refer to as aligning with a ‘quantitative mindset’. Such reviewers did not appear to understand that rather than aspiring to statistical representativeness, in qualitative research participants are selected purposefully for the contribution they can make towards the phenomenon under study [ 40 ]. Hence, the generalisability of qualitative findings beyond an immediate group of participants is judged by similarities between the time, place, people or other social contexts [ 41 ] rather than in relation to the comparability of the demographic variables. It is the fit of the topic or the comparability of the problem that is of concern [ 40 ].

The majority of issues that reviewers picked up on are already mentioned in reporting guidelines, so there is no reason why these were omitted by researchers. Many journals now insist on alignment with COREQ criteria, so there is an important question to be asked as to why this is not always happening. We suggest that completion of an established reporting checklist (e.g. COREQ, SRQR) on submission becomes a requirement.

In this FMRS we have made judgements about fellow peer reviewers and found their feedback to be constructive, but also, among some, we found some lack of grasp of the essence of the qualitative endeavor. Some reviewers did not seem to understand that objectivity and representative sampling are the antithesis of subjectivity, reflexivity and data saturation. We acknowledge though, that individual reviewers might have varying levels of experience and competence both in terms of qualitative research, but also in the reviewing process. We found one reviewer who apologised at the outset of the reviewer’s report for being unable to judge the data analysis due to their absence of sufficient knowledge in qualitative research. In line with Spigt and Arts [ 42 ], we appreciate the honesty of that reviewer for being transparent about their skillset. The lessons here we feel are for more experienced reviewers to offer support and reviewing mentorship to those who are less experienced and for reviewers to emulate the honesty of the reviewer as discussed here, by being open about their capabilities within the review process.

Based on our findings, we have a number of recommendations for both researchers and reviewers. For researchers reporting qualitative studies, we suggest that particular attention is paid to reporting of sampling techniques, both in the characteristics and composition of the sample, and how participants were selected. This is an issue that the reviewers in our FMRS picked up on, so forewarned is forearmed. But it is also crucially important that sampling matters are not glossed over, so this constitutes good practice in research reporting as well. Second, it seems that qualitative researchers do not give sufficient detail about analytic techniques and underlying theoretical frameworks. The latter has been pointed out before [ 25 ], but both these aspects were often the subject of reviewer comments.

Our recommendation for reviewers is simply to be honest. If qualitative research is not an area of expertise, then it is better to decline to undertake the review, than to apply a quantitative lens in the assessment of a qualitative piece of work. It is inappropriate to ask for details about validity and generalisability and shows a lack of respect to qualitative researchers. We are well beyond the arguments about quantitative versus qualitative [ 43 ]. It is totally appropriate to comment on background and findings and any obvious deficiencies. Finally, our recommendation to editors is a difficult one, because as editors ourselves we know how challenging it can be to find willing reviewers. When selecting reviewers however, it is as important to bear in mind the methodological aspects of an article and its subject, and to select reviewers with appropriate methodological expertise. Some journals make it a requirement for quantitative articles to be reviewed by a statistical expert and we think this is good practice. When it comes to qualitative articles however, the methodological expertise of reviewers may not be so stringently noted and applied. Editors could make a difference here and help to push up the quality of qualitative reviews.

Strengths and weaknesses

Since we had only access to reviewer’s comments of articles that were finally published in open access journals, we are unable to compare them to types of comments related to rejected submissions. Thus, this study was limited to manuscripts that were sent out for external peer review and were finally published. Furthermore, the chosen study design of analysing only reviewer comments of published articles with an open system of peer review did not allow direct comparison with reviewer comments derived from blind-review.

FMRS provides a snap-shot of a particular issue at one particular time [ 23 ]. To that end, findings might be different in another review undertaken in a different time period. However, as a contemporary profile of reviewing within qualitative research, the current findings provide useful insights for authors of qualitative reports and reviewers alike. Further research should focus on comparing reviewer comments taken from an open and closed system of peer review in order to identify similarities and differences between the two models of peer review.

A limitation is that we reviewed open access journals because this was the only way of accessing a range of comments. The alternative that we did consider was to use the feedback provided by reviewers on our own manuscripts. However, this would have lacked the transparency and traceability associated with this current FMRS, which we consider to be a strength. That said, there may be an inherent problem in having reviewed open access peer review comments, where both the author and reviewer are known. Reviewers are unable to ‘hide behind’ the anonymity of blind peer review and this might reflect, at least in part, why their comments as analysed for this review were overwhelmingly courteous and constructive. This is at odds with the comments that one of us has received as part of a blind peer review: ‘silly, silly, silly’ [ 36 ].

This FMRS has highlighted some important issues in the field of qualitative reviewing that hold lessons for authors, reviewers and editors. Authors of qualitative reports are called upon to follow guidelines on reporting and any amendments that these might contain as recommended by the findings of our review. Humility and transparency are required among reviewers when it comes to accepting to undertake a review and an honest appraisal of their capabilities in understanding the qualitative endeavor. Journal editors can assist this by thoughtful and judicious selection of reviewers. Ultimately, all those involved with the publication process can drive up the quality of individual qualitative articles and the synergy is such that this can make a significant impact on quality across the field.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

BioMed central

British medical journal

Consolidated criteria for reporting qualitative research

Enhancing the quality and transparency of health research

Focused mapping review and synthesis

Introduction, methods, results and discussion

Natural language toolKit

Standards for reporting qualitative research

Gannon F. The essential role of peer review (editorial). EMBO Rep. 2001;21(91):743.

Article   Google Scholar  

Mungra P, Webber P. Peer review process in medical research publications: language and content comments. Engl Specif Purp. 2010;29:43–53.

Turcotte C, Drolet P, Girard M. Study design, originality, and overall consistency influence acceptance or rejection of manuscripts submitted to the journal. Can J Anaesth. 2004;51:549–56.

Van der Wall EE. Peer review under review: room for improvement? Neth Heart J. 2009;17:187.

Burnham JC. The evolution of editorial peer review. JAMA. 1990;263:1323–9.

Article   CAS   Google Scholar  

Baldwin M. Credibility, peer review, and Nature , 1945-1990. Notes Rec R Soc Lond. 2015;69:337–52.

Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Assoc Inf Sci Technol. 2013;64:2–17.

Horbach SPJM, Halffman W. The changing forms and expectations of peer review. Res Integr Peer Rev. 2018;3:8.

Oermann MH, Nicoll LH, Chinn PL, Ashton KS, Conklin JL, Edie AH, et al. Quality of articles published in predatory nursing journals. Nurs Outlook. 2018;66:4–10.

University of Cambridge. How much do publishers charge for Open Access? (2019) https://www.openaccess.cam.ac.uk/paying-open-access/how-much-do-publishers-charge-open-access Accessed 26 Jun 2019.

Elsevier. Open access journals. (2018) https://www.elsevier.com/about/open-science/open-access/open-access-journals Accessed 28 Oct 2018.

Peters DP, Ceci SJ. Peer-review practices of psychological journals: the fate of published articles, submitted again. Behav Brain Sci. 1982;5:187–95.

Ross-Hellauer T. What is open peer review? A systematic review. F1000 Res. 2017;6:588.

Smith R. Opening up BMJ peer review. A beginning that should lead to complete transparency. BMJ. 1999;318:4–5.

Brown HM. Peer review should not be anonymous. BMJ. 2003;326:824.

Gosden H. “Thank you for your critical comments and helpful suggestions”: compliance and conflict in authors’ replies to referees’ comments in peer reviews of scientific research papers. Iberica. 2001;3:3–17.

Google Scholar  

Swales J. Occluded genres in the academy. In: Mauranen A, Ventola E, editors. Academic writing: intercultural and textual issues. Amsterdam: John Benjamins Publishing Company; 1996. p. 45–58.

Chapter   Google Scholar  

Landkroon AP, Euser AM, Veeken H, Hart W, Overbeke AJ. Quality assessment of reviewers' reports using a simple instrument. Obstet Gynecol. 2006;108:979–85.

Henly SJ, Dougherty MC. Quality of manuscript reviews in nursing research. Nurs Outlook. 2009;57:18–26.

Van Lent M, IntHout J, Out HJ. Peer review comments on drug trials submitted to medical journals differ depending on sponsorship, results and acceptance: a retrospective cohort study. BMJ Open. 2015. https://doi.org/10.1136/bmjopen-2015-007961 .

Davis CH, Bass BL, Behrns KE, Lillemoe KD, Garden OJ, Roh MS, et al. Reviewing the review: a qualitative assessment of the peer review process in surgical journals. Res Integr Peer Rev. 2018;3:4.

Bradbury-Jones C, Breckenridge J, Clark MT, Herber OR, Wagstaff C, Taylor J. The state of qualitative research in health and social science literature: a focused mapping review and synthesis. Int J Soc Res Methodol. 2017;20:627–45.

Bradbury-Jones C, Breckenridge J, Clark MT, Herber OR, Jones C, Taylor J. Advancing the science of literature reviewing in social research: the focused mapping review and synthesis. Int J Soc Res Methodol. 2019. https://doi.org/10.1080/13645579.2019.1576328 .

Taylor J, Bradbury-Jones C, Breckenridge J, Jones C, Herber OR. Risk of vicarious trauma in nursing research: a focused mapping review and synthesis. J Clin Nurs. 2016;25:2768–77.

Bradbury-Jones C, Taylor J, Herber OR. How theory is used and articulated in qualitative research: development of a new typology. Soc Sci Med. 2014;120:135–41.

Platt J. Using journal articles to measure the level of quantification in national sociologies. Int JSoc Res Methodol. 2016;19:31–49.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.

Shashok K. Content and communication: how can peer review provide helpful feedback about the writing? BMC Med Res Methodol. 2008;8:3.

Hall GM. How to write a paper. 2nd ed. London: BMJ Publishing Group; 1998.

Day FC, Dl S, Todd C, Wears RL. The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Ann Emerg Med. 2002;40:329–33.

Tashakkori A, Teddlie C. Mixed methodology: combining qualitative and quantitative approaches. London: Sage Publications; 1998.

Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. New York: Springer Publishing Company; 2007.

Jonas K, Crutzen R, Krumeich A, Roman N, van den Borne B, Reddy P. Healthcare workers’ beliefs, motivations and behaviours affecting adequate provision of sexual and reproductive healthcare services to adolescents in Cape Town, South Africa: a qualitative study. BMC Health Serv Res. 2018;18:109.

Burgess A, Roberts C, Sureshkumar P, Mossman K. Multiple mini interview (MMI) for general practice training selection in Australia: interviewers’ motivation. BMC Med Educ. 2018;18:21.

Lee S-Y, Lee EE. Cancer screening in Koreans: a focus group approach. BMC Public Health. 2018;18:254.

Taylor J, Bradbury-Jones C. Writing a helpful journal review: application of the 6 C’s. J Clin Nurs. 2014;23:2695–7.

Altman D. My journey to EQUATOR: There are no degrees of randomness. EQUATOR Network. 2016 https://www.equator-network.org/2016/02/16/anniversary-blog-series-1/ Accessed 17 Jun 2019.

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89:1245–51.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57.

Morse JM. Editorial: Qualitative generalizability. Qual Health Res. 1999;9:5–6.

Leung L. Validity, reliability, and generalizability in qualitative research. J Family Med Prim Care. 2015;4:324–7.

Spigt M, Arts ICW. How to review a manuscript. J Clin Epidemiol. 2010;63:1385–90.

Griffiths P, Norman I. Qualitative or quantitative? Developing and evaluating complex interventions: time to end the paradigm war. Int J Nurs Stud. 2013;50:583–4.

Download references

Acknowledgments

The support of Daniel Rütter in compiling data and providing technical support is gratefully acknowledged. Furthermore, we would like to thank Holger Hönings for applying general-purpose programming language to allow for a quantification of reviewer comments in the MS Excel spreadsheet.

Author information

Authors and affiliations.

Institute of General Practice, Centre for Health and Society, Medical Faculty of the Heinrich Heine University Düsseldorf, Moorenstr. 5, 40225, Düsseldorf, Germany

Oliver Rudolf HERBER

School of Nursing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK

Caroline BRADBURY-JONES & Julie TAYLOR

Institute of Health and Care Sciences, Sahlgrenska Academy at the University of Gothenburg, Gothenburg, Sweden

Susanna BÖLING

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Sarah COMBES

Institute of Applied Nursing Sciences, Department of Health, University of Applied Sciences FHS St.Gallen, St. Gallen, Switzerland

Julian HIRT

Cardiology department, Radboud University Medical Centre, Nijmegen, the Netherlands

Yvonne KOOP

Division of Emergencies and Critical Care, Oslo University Hospital/Institute of Health and Society, Faculty of Medicine, University of Oslo, Oslo, Norway

Ragnhild NYHAGEN

Hogeschool Utrecht, Utrecht, the Netherlands

Jessica D. VELDHUIZEN

Birmingham Women’s and Children’s Hospitals NHS Foundation Trust, Birmingham, UK

Julie TAYLOR

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have made an intellectual contribution to this research paper. ORH conducted the qualitative analysis and wrote the first draft of the paper. SB, SC, JH, YK, RN and JDV extracted and classified each comment using the classification system. CB-J and JT independently undertook a calibration exercise of a random sub-sample of articles ( n  = 22) to ensure consistency across coders. All co-authors (CB-J, SB, SC, JH, YK, RN, JDV and JT) have input into drafts and have read and approved the final version of the manuscript.

Corresponding author

Correspondence to Oliver Rudolf HERBER .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

References of all manuscripts included in the analysis ( n  = 107).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

HERBER, O.R., BRADBURY-JONES, C., BÖLING, S. et al. What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis. BMC Med Res Methodol 20 , 122 (2020). https://doi.org/10.1186/s12874-020-01005-y

Download citation

Received : 27 August 2019

Accepted : 04 May 2020

Published : 18 May 2020

DOI : https://doi.org/10.1186/s12874-020-01005-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Open access publishing
• Peer review
• Manuscript review
• reviewer’s report
• Qualitative analysis
• Qualitative research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 22, Issue 1
• How to appraise qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Calvin Moorley 1 ,
• Xabi Cathala 2
• 1 Nursing Research and Diversity in Care, School of Health and Social Care , London South Bank University , London , UK
• 2 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/ebnurs-2018-103044

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

In order to make a decision about implementing evidence into practice, nurses need to be able to critically appraise research. Nurses also have a professional responsibility to maintain up-to-date practice. 1 This paper provides a guide on how to critically appraise a qualitative research paper.

What is qualitative research?

• View inline

Useful terms

Some of the qualitative approaches used in nursing research include grounded theory, phenomenology, ethnography, case study (can lend itself to mixed methods) and narrative analysis. The data collection methods used in qualitative research include in depth interviews, focus groups, observations and stories in the form of diaries or other documents. 3

Authenticity

Title, keywords, authors and abstract.

In a previous paper, we discussed how the title, keywords, authors’ positions and affiliations and abstract can influence the authenticity and readability of quantitative research papers, 4 the same applies to qualitative research. However, other areas such as the purpose of the study and the research question, theoretical and conceptual frameworks, sampling and methodology also need consideration when appraising a qualitative paper.

Purpose and question

The topic under investigation in the study should be guided by a clear research question or a statement of the problem or purpose. An example of a statement can be seen in table 2 . Unlike most quantitative studies, qualitative research does not seek to test a hypothesis. The research statement should be specific to the problem and should be reflected in the design. This will inform the reader of what will be studied and justify the purpose of the study. 5

Example of research question and problem statement

An appropriate literature review should have been conducted and summarised in the paper. It should be linked to the subject, using peer-reviewed primary research which is up to date. We suggest papers with a age limit of 5–8 years excluding original work. The literature review should give the reader a balanced view on what has been written on the subject. It is worth noting that for some qualitative approaches some literature reviews are conducted after the data collection to minimise bias, for example, in grounded theory studies. In phenomenological studies, the review sometimes occurs after the data analysis. If this is the case, the author(s) should make this clear.

Theoretical and conceptual frameworks

Most authors use the terms theoretical and conceptual frameworks interchangeably. Usually, a theoretical framework is used when research is underpinned by one theory that aims to help predict, explain and understand the topic investigated. A theoretical framework is the blueprint that can hold or scaffold a study’s theory. Conceptual frameworks are based on concepts from various theories and findings which help to guide the research. 6 It is the researcher’s understanding of how different variables are connected in the study, for example, the literature review and research question. Theoretical and conceptual frameworks connect the researcher to existing knowledge and these are used in a study to help to explain and understand what is being investigated. A framework is the design or map for a study. When you are appraising a qualitative paper, you should be able to see how the framework helped with (1) providing a rationale and (2) the development of research questions or statements. 7 You should be able to identify how the framework, research question, purpose and literature review all complement each other.

There remains an ongoing debate in relation to what an appropriate sample size should be for a qualitative study. We hold the view that qualitative research does not seek to power and a sample size can be as small as one (eg, a single case study) or any number above one (a grounded theory study) providing that it is appropriate and answers the research problem. Shorten and Moorley 8 explain that three main types of sampling exist in qualitative research: (1) convenience (2) judgement or (3) theoretical. In the paper , the sample size should be stated and a rationale for how it was decided should be clear.

Methodology

Qualitative research encompasses a variety of methods and designs. Based on the chosen method or design, the findings may be reported in a variety of different formats. Table 3 provides the main qualitative approaches used in nursing with a short description.

Different qualitative approaches

The authors should make it clear why they are using a qualitative methodology and the chosen theoretical approach or framework. The paper should provide details of participant inclusion and exclusion criteria as well as recruitment sites where the sample was drawn from, for example, urban, rural, hospital inpatient or community. Methods of data collection should be identified and be appropriate for the research statement/question.

Data collection

Overall there should be a clear trail of data collection. The paper should explain when and how the study was advertised, participants were recruited and consented. it should also state when and where the data collection took place. Data collection methods include interviews, this can be structured or unstructured and in depth one to one or group. 9 Group interviews are often referred to as focus group interviews these are often voice recorded and transcribed verbatim. It should be clear if these were conducted face to face, telephone or any other type of media used. Table 3 includes some data collection methods. Other collection methods not included in table 3 examples are observation, diaries, video recording, photographs, documents or objects (artefacts). The schedule of questions for interview or the protocol for non-interview data collection should be provided, available or discussed in the paper. Some authors may use the term ‘recruitment ended once data saturation was reached’. This simply mean that the researchers were not gaining any new information at subsequent interviews, so they stopped data collection.

The data collection section should include details of the ethical approval gained to carry out the study. For example, the strategies used to gain participants’ consent to take part in the study. The authors should make clear if any ethical issues arose and how these were resolved or managed.

The approach to data analysis (see ref  10 ) needs to be clearly articulated, for example, was there more than one person responsible for analysing the data? How were any discrepancies in findings resolved? An audit trail of how the data were analysed including its management should be documented. If member checking was used this should also be reported. This level of transparency contributes to the trustworthiness and credibility of qualitative research. Some researchers provide a diagram of how they approached data analysis to demonstrate the rigour applied ( figure 1 ).

• Download figure
• Open in new tab
• Download powerpoint

Example of data analysis diagram.

Validity and rigour

The study’s validity is reliant on the statement of the question/problem, theoretical/conceptual framework, design, method, sample and data analysis. When critiquing qualitative research, these elements will help you to determine the study’s reliability. Noble and Smith 11 explain that validity is the integrity of data methods applied and that findings should accurately reflect the data. Rigour should acknowledge the researcher’s role and involvement as well as any biases. Essentially it should focus on truth value, consistency and neutrality and applicability. 11 The authors should discuss if they used triangulation (see table 2 ) to develop the best possible understanding of the phenomena.

Themes and interpretations and implications for practice

In qualitative research no hypothesis is tested, therefore, there is no specific result. Instead, qualitative findings are often reported in themes based on the data analysed. The findings should be clearly linked to, and reflect, the data. This contributes to the soundness of the research. 11 The researchers should make it clear how they arrived at the interpretations of the findings. The theoretical or conceptual framework used should be discussed aiding the rigour of the study. The implications of the findings need to be made clear and where appropriate their applicability or transferability should be identified. 12

Discussions, recommendations and conclusions

The discussion should relate to the research findings as the authors seek to make connections with the literature reviewed earlier in the paper to contextualise their work. A strong discussion will connect the research aims and objectives to the findings and will be supported with literature if possible. A paper that seeks to influence nursing practice will have a recommendations section for clinical practice and research. A good conclusion will focus on the findings and discussion of the phenomena investigated.

Qualitative research has much to offer nursing and healthcare, in terms of understanding patients’ experience of illness, treatment and recovery, it can also help to understand better areas of healthcare practice. However, it must be done with rigour and this paper provides some guidance for appraising such research. To help you critique a qualitative research paper some guidance is provided in table 4 .

Some guidance for critiquing qualitative research

• ↵ Nursing and Midwifery Council . The code: Standard of conduct, performance and ethics for nurses and midwives . 2015 https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21 Aug 18 ).
• Barrett D ,
• Cathala X ,
• Shorten A ,

Patient consent for publication Not required.

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

A Guide to Writing a Qualitative Systematic Review Protocol to Enhance Evidence-Based Practice in Nursing and Health Care

Affiliations.

• 1 PhD candidate, School of Nursing and Midwifey, Monash University, and Clinical Nurse Specialist, Adult and Pediatric Intensive Care Unit, Monash Health, Melbourne, Victoria, Australia.
• 2 Lecturer, School of Nursing and Midwifery, Monash University, Melbourne, Victoria, Australia.
• 3 Senior Lecturer, School of Nursing and Midwifery, Monash University, Melbourne, Victoria, Australia.
• PMID: 26790142
• DOI: 10.1111/wvn.12134

Background: The qualitative systematic review is a rapidly developing area of nursing research. In order to present trustworthy, high-quality recommendations, such reviews should be based on a review protocol to minimize bias and enhance transparency and reproducibility. Although there are a number of resources available to guide researchers in developing a quantitative review protocol, very few resources exist for qualitative reviews.

Aims: To guide researchers through the process of developing a qualitative systematic review protocol, using an example review question.

Methodology: The key elements required in a systematic review protocol are discussed, with a focus on application to qualitative reviews: Development of a research question; formulation of key search terms and strategies; designing a multistage review process; critical appraisal of qualitative literature; development of data extraction techniques; and data synthesis. The paper highlights important considerations during the protocol development process, and uses a previously developed review question as a working example.

Implications for research: This paper will assist novice researchers in developing a qualitative systematic review protocol. By providing a worked example of a protocol, the paper encourages the development of review protocols, enhancing the trustworthiness and value of the completed qualitative systematic review findings.

Linking evidence to action: Qualitative systematic reviews should be based on well planned, peer reviewed protocols to enhance the trustworthiness of results and thus their usefulness in clinical practice. Protocols should outline, in detail, the processes which will be used to undertake the review, including key search terms, inclusion and exclusion criteria, and the methods used for critical appraisal, data extraction and data analysis to facilitate transparency of the review process. Additionally, journals should encourage and support the publication of review protocols, and should require reference to a protocol prior to publication of the review results.

Keywords: guidelines; meta synthesis; qualitative; systematic review protocol.

© 2016 Sigma Theta Tau International.

PubMed Disclaimer

Similar articles

• How has the impact of 'care pathway technologies' on service integration in stroke care been measured and what is the strength of the evidence to support their effectiveness in this respect? Allen D, Rixson L. Allen D, et al. Int J Evid Based Healthc. 2008 Mar;6(1):78-110. doi: 10.1111/j.1744-1609.2007.00098.x. Int J Evid Based Healthc. 2008. PMID: 21631815
• Procedures and methods of benefit assessments for medicines in Germany. Bekkering GE, Kleijnen J. Bekkering GE, et al. Eur J Health Econ. 2008 Nov;9 Suppl 1:5-29. doi: 10.1007/s10198-008-0122-5. Eur J Health Econ. 2008. PMID: 18987905
• [Procedures and methods of benefit assessments for medicines in Germany]. Bekkering GE, Kleijnen J. Bekkering GE, et al. Dtsch Med Wochenschr. 2008 Dec;133 Suppl 7:S225-46. doi: 10.1055/s-0028-1100954. Epub 2008 Nov 25. Dtsch Med Wochenschr. 2008. PMID: 19034813 German.
• Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Manchikanti L. Manchikanti L. Pain Physician. 2008 Mar-Apr;11(2):161-86. Pain Physician. 2008. PMID: 18354710 Review.
• An example of the use of systematic reviews to answer an effectiveness question. Forbes DA. Forbes DA. West J Nurs Res. 2003 Mar;25(2):179-92. doi: 10.1177/0193945902250036. West J Nurs Res. 2003. PMID: 12666642 Review.
• Factors affecting implementation of mindfulness in hospital settings: A qualitative meta-synthesis of healthcare professionals' experiences. Knudsen RK, Skovbjerg S, Pedersen EL, Nielsen CL, Storkholm MH, Timmermann C. Knudsen RK, et al. Int J Nurs Stud Adv. 2024 Mar 27;6:100192. doi: 10.1016/j.ijnsa.2024.100192. eCollection 2024 Jun. Int J Nurs Stud Adv. 2024. PMID: 38746813 Free PMC article. Review.
• Experiences of nurses and midwives in policy development in low- and middle-income countries: Qualitative systematic review. Etowa J, Vukic A, Aston M, Iduye D, Mckibbon S, George A, Nkwocha C, Thapa B, Abrha G, Dol J. Etowa J, et al. Int J Nurs Stud Adv. 2022 Dec 17;5:100116. doi: 10.1016/j.ijnsa.2022.100116. eCollection 2023 Dec. Int J Nurs Stud Adv. 2022. PMID: 38746557 Free PMC article. Review.
• Couples' experiences of pregnancy resulting from assisted reproductive technologies: A qualitative meta-synthesis. René C, Landry I, de Montigny F. René C, et al. Int J Nurs Stud Adv. 2021 Dec 17;4:100059. doi: 10.1016/j.ijnsa.2021.100059. eCollection 2022 Dec. Int J Nurs Stud Adv. 2021. PMID: 38745619 Free PMC article. Review.
• Barriers and facilitators of adherence to low-dose aspirin during pregnancy: A co-produced systematic review and COM-B framework synthesis of qualitative evidence. Vinogradov R, Holden E, Patel M, Grigg R, Errington L, Araújo-Soares V, Rankin J. Vinogradov R, et al. PLoS One. 2024 May 3;19(5):e0302720. doi: 10.1371/journal.pone.0302720. eCollection 2024. PLoS One. 2024. PMID: 38701053 Free PMC article.
• The ethical challenges of diversifying genomic data: A qualitative evidence synthesis. Hardcastle F, Lyle K, Horton R, Samuel G, Weller S, Ballard L, Thompson R, De Paula Trindade LV, Gómez Urrego JD, Kochin D, Johnson T, Tatz-Wieder N, Redrup Hill E, Robinson Adams F, Eskandar Y, Harriss E, Tsosie KS, Dixon P, Mackintosh M, Nightingale L, Lucassen A. Hardcastle F, et al. Camb Prism Precis Med. 2023 Sep 12;2:e1. doi: 10.1017/pcm.2023.20. eCollection 2024. Camb Prism Precis Med. 2023. PMID: 38549845 Free PMC article. Review.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• Open access
• Published: 12 July 2024

Unlocking potential: a qualitative exploration guiding the implementation and evaluation of professional role substitution models in healthcare

• Rumbidzai N. Mutsekwa   ORCID: orcid.org/0000-0001-8200-3026 1 , 2 , 3 ,
• Katrina L. Campbell   ORCID: orcid.org/0000-0002-4479-1284 3 , 4 , 5 ,
• Russell Canavan   ORCID: orcid.org/0000-0002-0155-3219 6 ,
• Rebecca L. Angus   ORCID: orcid.org/0000-0002-1839-9641 1 , 7 ,
• Liza-Jane McBride   ORCID: orcid.org/0000-0002-6336-5934 8 &
• Joshua M. Byrnes   ORCID: orcid.org/0000-0001-6562-711X 3 , 4  

Implementation Science Communications volume  5 , Article number:  73 ( 2024 ) Cite this article

175 Accesses

3 Altmetric

Metrics details

As role substitution models gain prominence in healthcare, understanding the factors shaping their effectiveness is paramount. This study aimed to investigate factors that impact the implementation and performance evaluation of professional role substitution models in healthcare, with a focus on understanding the variables that determine their success or failure in adoption, execution, continuity, and outcomes.

The exploratory qualitative study used semi-structured interviews with key opinion leaders, decision makers, facilitators, recipients, and frontline implementers, who had influence and involvement in the implementation of professional role substitution models. Data analysis was guided by the Consolidated Framework for Implementation Research (CFIR).

Between November 2022 and April 2023, 39 stakeholders were interviewed. Factors influencing implementation and evaluation of allied health professional role substitution models of care aligned with the five core CFIR domains (innovation, outer setting, inner setting, individuals, implementation process) and outcome domain incorporating implementation and innovation outcomes. The six themes identified within these CFIR domains were, respectively; i) Examining the dynamics of innovation catalysts, evidence, advantages, and disadvantages; ii) Navigating the complex landscape of external factors that influence implementation and evaluation; iii) Impact of internal structural, political, and cultural contexts; iv) The roles and contributions of individuals in the process; v) Essential phases and strategies for effective implementation; and vi) The assessment of outcomes derived from allied health professional role substitution models.

Conclusions

The study highlights the complex interplay of contextual and individual factors that influence the implementation and performance evaluation of professional role substitution models. It emphasises the need for collaboration among diverse stakeholders to navigate the challenges and leverage the opportunities presented by expanded healthcare roles. Understanding these multifaceted factors can contribute to the development of an empowered workforce and a healthcare system that is more efficient, effective, safe, and sustainable, ultimately benefiting patients.

Peer Review reports

Contributions to literature

• There is limited understanding of the complex interplay of contextual and individual factors that influence implementation and performance evaluation of professional role substitution models of care.

• This study provides comprehensive guidance on successful implementation and evaluation of new models of care which influences efficient use of resources in healthcare.

• This study contributes to recognised gaps in literature, seeking to demonstrate value proposition of professional role substitution models of care. This study has identified outcome measures that can determine the successful implementation and impact of these models of care

The healthcare sector plays a crucial role in ensuring the well-being of individuals and society, but it is facing challenges due to a growing and ageing population. The demand for high-quality healthcare has increased significantly, while the shortage of healthcare workers has become a pressing concern [ 1 , 2 ]. Workforce reforms are now being prioritised in healthcare to shape the future of healthcare delivery. These reforms include initiatives to increase the number of healthcare workers, enhance the quality and duration of healthcare education and training, and diversify the healthcare workforce.

One key strategy to address healthcare challenges is the expanded scope of practice for non-medical healthcare professionals [ 3 ]. This expansion entails a discrete knowledge and skill base beyond the recognised scope of practice within a specific jurisdiction's regulatory framework [ 4 ]. It empowers healthcare practitioners such as nurse practitioners, allied health professionals, and physician assistants to practice to the full extent of their training and education, or to extend their scope of practice beyond traditional boundaries [ 5 , 6 , 7 ]. Consequently, they can perform a broader range of tasks, including those previously reserved for medical doctors.

Professional role substitution models have improved patients' access to healthcare services [ 8 , 9 , 10 ]. Moreover, there is a growing body of evidence suggesting that these alternative healthcare delivery models can provide safe and effective care that patients find acceptable. Nurse practitioners and advanced nurses in the US, Canada, the UK, and Australia expand primary care roles, including diagnosis, prescribing, patient education, managing long-term conditions, and minor surgeries [ 9 , 11 , 12 , 13 ]. Physician assistants (PAs) in countries like the US, Canada, and the Netherlands work closely with physicians, conducting assessments, diagnosing, treating common illnesses, and providing patient education. PAs improve healthcare access, especially in underserved and rural areas with physician shortages [ 14 , 15 , 16 ].

In developing countries with limited healthcare resources, professional role substitution models are vital for addressing shortages of skilled healthcare providers and improving access to essential services. For instance, in sub-Saharan Africa, task shifting from physicians to nurses and community health workers addresses the scarcity of skilled providers [ 17 , 18 ]. Community health workers, trained to deliver basic healthcare services and education, play crucial roles in preventive and promotive interventions, particularly in rural and underserved areas [ 18 ]. Nurse-led clinics have also proven successful in delivering comprehensive primary care services, such as antenatal care and family planning, alleviating pressure on strained healthcare systems [ 19 , 20 , 21 ].

Allied health professionals, encompassing disciplines such as speech pathology, pharmacy, dietetics, physiotherapy, occupational therapy, radiography, sonography, psychology, and social work, are increasingly vital in diverse healthcare settings. Supported by mounting evidence of their effectiveness, their role continues to expand [ 10 , 22 ]. Despite substantial growth, particularly notable in Australia where they rank as the second-largest healthcare group, [ 23 ] the implementation of professional role substitution within allied health is relatively new compared to fields like nursing and physician assistants [ 24 ].

The successful implementation of all professional role substitution models including allied health is complex and contingent on various factors which are not currently well understood or defined [ 10 , 22 , 25 ]. To ensure success, it is essential to consider the impact on patients, healthcare professionals, and the healthcare system [ 25 ]. This must be approached from a multi-stakeholder perspective, involving experts in the field, key opinion leaders, healthcare leaders, decision makers, policy makers, recipients, and frontline implementers.

Research into the expanded scope of practice within allied health disciplines, including implementation and performance evaluation, is crucial [ 6 , 23 , 25 , 26 ]. Previous studies have highlighted patients' perceptions and experiences of healthcare quality in role substitution models [ 27 , 28 ]. While clinicians express support for performance evaluation, there's a gap between support and effective implementation [ 29 ]. There's also a lack of agreed-upon approaches for measuring performance [ 25 , 29 ]. Collaborative efforts involving multiple stakeholders are essential for understanding robust evaluation methods and optimising alternative models of care for healthcare transformation and sustainability [ 25 ].

To address this gap in knowledge and practice, this study aimed to describe the individual and contextual factors that influence the implementation and performance evaluation of allied health professional role substitution models from a multi-stakeholder perspective. Furthermore, the study aimed to identify outcome measures that can demonstrate the successful implementation and impact of these models of care.

Study approach and design

An exploratory qualitative approach was used to explore expectations, perceptions, and experiences of stakeholders involved in the implementation and performance evaluation of professional role substitution models of care. Semi-structured interviews were chosen as the primary method of data collection to allow for flexible exploration of specific topics and issues, maximising the richness of the data [ 30 ]. The study adhered to the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines [ 31 ]. Please see Additional file 1

Study setting

This study was conducted within the public healthcare system of the State of Queensland, Australia. This comprises 16 hospital and health services and approximately 35,000 allied health professionals [ 32 ]. Queensland initiated an allied health strategy in 2014 to expand professionals' scope of practice, resulting in the establishment of 133 distinct models of care by 2019 [ 24 , 33 , 34 ]. Examining this system offers valuable insights into implementing and evaluating professional role substitution models, providing practical understanding within a specific healthcare context.

Study participants and recruitment

A purposeful sampling strategy was employed to recruit key stakeholders at various levels of the healthcare system who were involved in some way in the implementation and performance evaluation of allied health professional role substitution models of care. Participants included experts in the field, key opinion leaders, decision makers, recipients, and frontline implementers, implementation facilitators and support teams. A sampling matrix was used to consider factors such as location, affiliation, organisational role, tenure, and profession ensuring diversity and representation across the different dimension of the healthcare system. While a specific target number of participants was not predetermined, our aim was to achieve saturation in the sample, ensuring comprehensive coverage of perspectives and experiences relevant to our research objectives. Email invitations were sent to potential participants/participant groups, along with study information and consent forms. Those who agreed to participate contacted the principal investigator to arrange a suitable interview time.

Positionality of researchers

The research team comprised individuals with diverse backgrounds and roles, including experts in professional role substitution, health services research, economics, qualitative study methodology, and healthcare management.

This study was performed in line with the principles of the Declaration of Helsinki with approval granted by Gold Coast Hospital and Health Service (HREC/2020/QGC/62104) and Griffith University (GU Ref No: 2020/876). All participants provided written informed consent.

Data collection

An interview guide was developed by the research team to ensure coverage of the study aims and objectives (Additional file 2). The guide was pilot tested with three eligible participants, resulting in minor wording adjustments for clarity. Interviews were conducted either face-to-face or via video conferencing with only interviewer and participant present. The semi-structured interviews were designed to elicit open-ended responses from participants, with the interviewer using prompts and probing techniques as needed. Data collection continued until data saturation was reached, indicating that no new themes were emerging [ 35 ]. All interviews were audio-recorded, transcribed, and supplemented with field notes for additional context and consistency. Each participant was allocated an anonymous identifier, comprising their participant number along with a descriptor of their role or professional background. (e.g., P34, Workforce and Education). Participants were offered the opportunity to check their transcript.

Data analysis and interpretation

Descriptive statistics were used to analyse demographic data, such as participants' time in their current role, age, gender, and education level. Exploration of contextual influences on implementation and performance evaluation was guided by the Consolidated Framework for Implementation Research (CFIR) [ 36 ]. The CFIR is a comprehensive framework that focuses on understanding and improving the implementation and evaluation of health innovations. Its adaptability enables integration into various contexts, fostering analysis and facilitating cross-study comparisons. This versatility supports a systematic approach to evaluating implementation processes and outcomes, thereby enriching our understanding of innovation dynamics across diverse settings [ 36 ].

It consists of six domains: 1. Innovation domain (the model of care being implemented), 2. Outer setting (the healthcare system in which the inner setting exists) 3. Inner setting (the site in which the model of care is implemented e.g., hospital) 4. Individuals (the roles and characteristics of individuals involved in the implementation process), 5. Implementation process (the activities and strategies used to implement the model of care), 6. Implementation outcomes (perceptions and measures of implementation success or failure), and Innovation outcomes (outcomes that capture success or failure of model of care) [ 36 , 37 , 38 ].

A reflexive thematic approach was taken for qualitative analysis [ 39 ]. The analysis began deductively with codes derived from the CFIR, followed by inductive coding to identify additional categories. These codes were assigned using CFIR definitions, inclusion/exclusion criteria, and appropriate quote examples. NVivo V10 software (QSR International Ltd.) was used to facilitate data management.

Investigator triangulation was employed, with the principal researcher (R.N.M) coding all interviews and 20% of the interviews coded by a second researcher (R.L.A) to enhance reliability and provide different perspectives [ 40 ]. All authors participated in summarising codes prioritised for analysis and interpreting the results. A matrix was created to compare the ratings of each CFIR construct, focusing on any differences among stakeholders. Data extracts were selected to illustrate themes and subthemes, incorporating multiple perspectives for interpretation.

Study population

A total of 39 stakeholders from various hospital and health services across Queensland were interviewed. The stakeholders represented a broad spectrum of positions and roles within the healthcare system, categorised into eight groups: allied health clinicians, medical practitioners/general practitioners, nursing staff, allied health leadership, hospital and health services/statewide leadership, recipients, implementation support personnel, workforce and education. Table 1 provides demographic details of the participants.

Participants had been in their roles on average 11 years, (range 1-27 years). Interviews had an average duration of 32 minutes (range 15-59 minutes). Five interviews were conducted face to face with the remainder ( n =34) conducted through video conferencing.

Six themes were identified which aligned with the five CFIR domains and the outcomes domain. Twenty-seven underlying constructs and subconstructs of the CFIR were identified as factors influencing implementation of professional role substitution in our analysis. Ten constructs were identified in the implementation and innovation outcome categories. Main domains and constructs are illustrated in Fig. 1 .

Key implementation and evaluation constructs for professional role substitution models of care

Innovation domain

Examining the dynamics of innovation catalysts, evidence, advantages, and disadvantages in allied health professional role substitution models of care.

The following section delineates the three primary constructs aligning with CFIR domains and an additional domain, namely "relative disadvantage." These constructs were identified from the perspectives of participants regarding professional role substitution models of care as an innovative approach.

Innovation source

Participants recognised healthcare system strain due to workforce shortages, rising costs, and increased needs with policies now prioritising workforce reform as a key healthcare strategy. The 2006 Australian Productivity Commission review, focusing on optimising scope, competencies, and job redesign was frequently cited by interviewees as the catalyst for change. The Ministerial Taskforce on Health Practitioner Expanded Scope of Practice (Queensland), alongside similar taskforces nationwide, played a vital role in implementing allied-health professional role substitution models of care. “There were a broad range of stakeholders involved in the task force across Queensland Health and external to Queensland in 2014. There was a number of recommendations in the report with overall endorsement from the Minister.” (P34, Workforce and Education)

Furthermore, the Allied Health Professions' Office supported these efforts by funding care models, addressing legislative barriers, developing training, supporting research, monitoring progress, and sharing achievements. A participant explained, ‘The office was charged with implementing the recommendations and to test these models of care. Particularly things like requesting and interpreting forms for diagnostic imaging and requesting pathology.” ( P34, Workforce and Education)

Evidence base

Participants expressed varying perspectives on the evidence base for professional role substitution models of care. Some noted a reliance on grey literature or information from pilots, highlighting the limited evidence supporting certain models. Conversely, others believed the evidence base was robust and questioned the need for further piloting. “It should be business as usual and that’s something we’ve tried to promote where we’ve got evidence from other jurisdictions and internationally. There should then be efforts to implement and try and replicate those results and take it to scale.” (P34, Workforce and Education).

Established services in other countries and professions influenced the implementation in Australia. Clinician leads or facilitators with prior experience in allied health models were identified as key enablers of this process. One participant shared their experience stating, “I was involved with that over there in the UK and so I came with that mentality to Queensland. When I worked as a fellow, I was surprised that there wasn’t that model, and I advocated for it and was told we don’t do that here. So, we ended up running extra clinics as fellows to see the long-wait patients when I knew that back in the UK it would have been [allied health discipline ].” (P19, Medical Specialist)

Relative advantage

Participants, healthcare professional and patients alike identified several advantages of allied health professional role substitution models. These models improved access to care, particularly benefiting underserved areas, and boosted efficiency by “streamlining decision-making and minimising duplication” (P19, Medical specialist). A patient shared their positive experience, stating, “If anything, I thought I was really special. I got pushed ahead really. I didn’t have to wait so long, and I wasn’t made to feel silly for my symptoms and they were investigated. The whole experience was positive.” ( P39, Consumer/Recipient)

While considered cost-effective by those interviewed due to reduced reliance on specialists in resource-limited settings, many highlighted the need for further cost-effectiveness data. One participant mentioned,

“You can get comparable or sometimes a better service at a lower cost using alternate models of care.” (P17, Medical Specialist)

Participants indicated that these models enabled allied health professionals to provide comprehensive, patient-centred care, enhancing overall healthcare experiences and patient satisfaction. Another viewpoint shared was, “ It's about getting patients to clinicians with holistic skill sets rather than just the medical model. Traditionally, patients wait a long time to see a medical professional, only to be referred back to the same clinician “, ( P15, Implementation Support). Additionally, clinicians working in these roles noted , “So, they’re kind of getting that one stop assessment, where the speech pathologist looks at the functional component, as well as pathology or organic disease” (P13, Allied Health Clinician). Moreover, they promoted professional growth, job satisfaction, and workforce retention through expanded roles and skill development opportunities, fostering collaboration among healthcare professionals from various disciplines for improved patient outcomes. One individual expressed “That responsibility and that extra challenge for me is where I get the buzz. (P 12, Allied Health Clinician )

Relative disadvantage

In addition to the benefits of professional role substitution in healthcare, participants emphasised other key factors. Patient safety and care quality surfaced as paramount concerns. A participant with workforce and education background stated, “There was a lot of the discussion and particularly the negative media coverage around the model of care. I was quite driven to answer the questions, or the concerns raised by the health professionals around safety .” (P29, Workforce and Education)

The imperative of ensuring skill, competence, and appropriate clinical governance was strongly emphasised. In some settings, participants flagged the potential for resistance and conflicts with traditional providers and organisations, driven by apprehensions about expertise encroachment, de-skilling, and role ambiguity. A Medical Specialist (P19) highlighted this, “The risk is that if you promote therapists from being treating therapists to being screening and treating therapists, you’re on the risk of deskilling your (medical) workers.”

Building public and patient trust, especially in unfamiliar models, highlighted the importance of transparent communication and educational efforts, as noted by both consumers and healthcare professionals. A patient shared, “I really didn’t know what to expect because I hadn’t been to a clinic like that before and I didn’t know what they were going to do”. (P39, Consumer/recipient). A healthcare professional suggested, “ Another barrier is patient perception, especially if they are expecting to see a doctor” but went on to add , “In my experience this has often not been the case with patients often reassured once they have had a thorough assessment ” (P13, Allied Health Clinician) Initial challenges in interaction with General Practitioners (GPs), were also highlighted with one participant noting, “ See the problems at the beginning where the GPs would ring up and say, I wanted a specialist opinion, and I got a physiotherapist. But once they were educated, those complaints dropped off especially when the patient satisfaction scores were high” (P19, Medical Specialist).

The implementation of these models often demands additional investments in training and supervision, with a consideration of their economic and logistical impact on the healthcare system required. Lastly, “striking a delicate balance between expanded scopes and core responsibilities” (P 25, Allied Health Leadership) is essential. Another participant noted, “ It is also worth considering the amount of time it takes for this training and to set up these roles. It is also important to consider the cost. Once you have a model of care set up well, what’s my sustainability plan for this model in relation to, succession planning, leave management, etcetera ? “(P15, Implementation Support)

Outer setting

Navigating the complex landscape of external factors that influence implementation and evaluation of allied health professional role substitution models of care, partnerships and connections.

Collaborative care teams and strong referral networks emerged as crucial elements for successful role substitution practice. Participants emphasised the importance of interdisciplinary collaboration, where professionals from various disciplines worked together to provide holistic patient care. Furthermore, partnerships with specialists, hospitals, community resources, and primary care facilities were highlighted as essential for ensuring seamless transitions and continuity of care. This was articulated by one GP, (P7) “I think for me and my style of medicine, it’s helpful. I really enjoyed that sort of team, that real MDT and holistic approach to patient care.”

Policies and laws

Implementing professional role substitution and scope extension may require legal and regulatory adjustments, including redefining boundaries and establishing standards which participants noted as a challenge. Variations across jurisdictions, were highlighted emphasising the need for a national approach to align state and federal policies. An occupational therapist identified legislative barriers stating, “Legislation prevents us from ordering imaging, but we all have local agreements with our departments that enable us to order basic radiology. But we want to be able to order that radiology in our general role as well and potentially expand that into other forms of the imaging down the road. This role has expanded even further in the UK to some of those therapists prescribing and referring people for MRIs and CT scans.” (P11, Allied Health Clinician)

Both allied health clinicians and medical doctors expressed concerns about legal accountability in the event of adverse events or complications in professional role substitution models. Stakeholders, including allied health clinicians, medical doctors, and healthcare leaders, emphasised the importance of assurance of indemnity through health services. “We’re protected by public indemnity in this system. And ultimately the directors are responsible for all the patients, even the ones we don’t directly treat. So that model protected our junior doctors and subsequently protects the therapists as well ” (P19, Medical Specialist). Participants also acknowledged the need for regular training and re-assessment of knowledge and skills for medical professionals but were uncertain about the lack of similar scrutiny and regulation mechanisms for allied health clinicians in professional role substitution roles.

Participants had differing perspectives on funding for new models of care. Implementing professional role substitution models of care often relied on short-term funding and grants to pilot services. A participant with an allied health clinician background highlighted complexities in healthcare funding and incentives, pointing out “General practice won’t make money unless the patient sees the GP. They would need to look at some sort of MBS (Medicare Benefits Schedule) item number so that the practice or hospital and health service can generate money from those expanded roles.” (P10, Allied Health Clinician)

Suggestions were made to review Medicare and activity-based funding structures to provide support for professional role substitution models ensuring their viability. A participant who has supported implementation of a professional role substitution model noted, “That’s also based on the fact that with Activity-Based Funding framework, we have to demonstrate that the model can generate enough activity to be viable and valuable.” (P15, Implementation Support).

Additionally, participants emphasised the importance of funding models that prioritise outcomes rather than specific care delivery mechanisms. A healthcare executive highlighted, “W e don’t purchase models of care. I would like to think that we purchase outcomes, and we are quite agnostic in how health services go about achieving those outcomes. We’ve wanted to make sure that the funding model is enabled and that it’s not a barrier to people trying alternative ways using new and different models to achieve those outcomes that we’re interested in.” (P32, Hospital and Health Services/ State-wide Leadership).

Performance management pressure

Participants acknowledged the challenge of meeting patient waiting time targets set by federal and state governments. This was an enabling factor, with professional role substitution models of care implemented as strategies to reduce specialist outpatient waitlists and improve access to services, aligning with performance targets. “There was a wait list issue for the specialty area. There was a big project to see who else could help see patients and try and reduce the waitlists. They highlighted that the [allied health specialist area clinician] might be something that could help with that.” (P8, Allied Health Clinician).

Inner setting

Impact of internal structural, political, and cultural contexts on the implementation and performance evaluation of allied health professional role substitution models in healthcare, work infrastructure.

Implementing role substitution models had workforce implications, including assessing skills availability and workload management. Sustainability relied on individual commitment, posing threats to the longevity of these models of care. An allied health leader, (P22) noted, “Often the first people you get in are personally passionate about it. It’s hard to find those people all the time, but a succession plan is important for sustainability of extended scope roles.“ These sentiments were echoed by a physician who mentioned, “Workforce and sourcing the right resources and clinicians is something that is a bit of a challenge for the health services moving forward.” (P17, Medical specialist)

Relational connections

Participants identified strong relationships and networks as vital for implementing and sustaining professional role substitution models. Trust between medical doctors and allied health clinicians was essential. As one participant noted: “Most of the time when these models fall down, it’s because the relationships between the allied health and the multidisciplinary team, including the doctors, have broken down. The doctor’s left or there’s been an issue that they couldn’t resolve and then everything falls to pieces.” (P23, Allied Health Leadership) Key roles of advocates and clinical leads were emphasised, but overreliance on individuals was a concern. Building resilience in these models across all levels of leadership was an important consideration as are clear governance structures which include supervision and escalation pathways.

Communications

Effective communication was necessary for high-quality care, patient safety, and collaborative relationships in both implementing and sustaining professional role substitution models. Iterative modifications and a willingness to learn were recognised as important. Collaboration involved shared decision-making, regular communication, and joint management of complex cases. Specialist doctors provided guidance and medical expertise, while allied health professionals contributed their specialised skills including ability to provide holistic care. “We still needed to iron out all of the kinks, so each side still needs to continue to learn from each other. So, I would say it probably took a good 12 to 18 months before we felt like we had a system that was working well for both sides and streamlining the process.” (P11, Allied Health Clinician). Additionally, some participants emphasised transitioning from “substitution-focused to team-based approaches” (P27, General Practitioner and Healthcare Executive), promoting interdisciplinary and transdisciplinary care.

For some participants, professional role substitution raised concerns about autonomy with potential for conflicts among healthcare professionals. Cultivating a collaborative culture, renegotiating traditional hierarchies, and addressing professional dynamics were identified as strategies to enable interprofessional collaboration, promoting innovation and excellence in patient care. However, despite the progress made, some participants expressed reservations about barriers that still exist, even in allied health practitioners performing tasks that were within their scope of practice. One executive leader expressed frustration at the slow pace of change stating, “ It’s an imperative at the moment that we actively promote full scope of practice and give more support for our allied health staff to do extended scope of practice qualifications. So, we have a role to ensure that we have a culture that encourages the new models of care, because just to have the old models of care, it’s not simply sustainable, it’s not sustainable, at all. “ (P36, Hospital and Health Services/ State-wide Leadership)

Another participant, an allied health leader (P22), highlighted the positive impact of professional role substitution on organisational culture and the morale of younger professionals, stating, “It’s good for our culture and gives some sort of energy to the younger professionals. It also flows through to junior doctors particularly working alongside a consultant that already holds these clinicians and models of care in high standard.”

Mission alignment and tension for change

In many organisations, clinical demand drove professional role substitution adoption, facilitated by change management teams and frameworks. Professional role substitution models aligned with healthcare organisational goals and objectives, promoting innovation, equity, and sustainable use of resources. As articulated by a Medical Specialist (P19) “We have a limited number of specialists, and training for medical students and junior doctors hasn't significantly increased to meet demand. With advanced technology and reduced working hours, we need to expand services. Having other clinicians who can treat patients without surgery is invaluable.”

Additionally, participants acknowledged their role in healthcare delivery to underserved communities and advancing health equity in First Nations, rural, and regional areas, “ improving access and preventing, fragmented care ,” (P33, Nursing Health Professional). Furthermore, participants discussed the impact of population growth on surgical waitlists, revealing the pressing need for effective solutions to address increasing demand. An allied health clinician (P11) highlighted the challenges posed by population growth, stating, “ There's been a significant increase in people moving to Queensland now for many years and our surgical wait lists were continuing to grow. So, when I started in this role… the waitlist was almost four years long.”

Participants stressed the strategic importance of expanded scope in advancing organisational objectives. An Allied health leadership participant (P25), emphasised the multifaceted benefits of expanded scope, highlighting its alignment with strategic goals and the need to reassess care delivery models: “Expanded scope hits all the strategic goals really. We need to disinvest in some of the low value care because we know that we’re not getting any outcomes. We also need to look at the impact of these models of care.”

Available resources

Funding for professional role substitution models varied, with some implemented without dedicated funding which posed challenges in attracting skilled clinicians. Stakeholders recognised the benefits of co-locating allied health clinicians and medical doctors for interdisciplinary case discussions but sometimes faced challenges due to high demand for limited space. An allied health clinician (P8) highlighted the impact of dedicated funding on the feasibility and efficiency of implementing such models “They had a certain amount of funding for this project to set it all up. And I think that really made it feasible. So, then we got the right equipment, the right time to set it up. It was a very set process with money attached to it that got it off the ground quicker.”

Access to knowledge and information

Clinicians in extended scope roles actively sought professional development opportunities to expand their skills. Local credentialing and on-the-job training were the norm. A workforce development officer highlighted the rigorous process of credentialing for clinicians in such roles. “Our credentialling package is fairly intense. It takes months and months and months to become credentialed in a first point of contact clinic like this and needs [Health Service] approval before a clinician can work in a space like this.” (P29, Workforce and Education). In contrast to nurse practitioner programs offered by universities and specialised training institutions, formal education programs for allied health professionals were scarce. Many participants recommended development of formalised training and credentialing programs to ensure high quality and safe care. “We’re now in the process of developing our own course here in Australia in collaboration with the university in New South Wales so that we can provide that level of education that we need in these advanced scope roles ” (P11, Allied Health Clinician).

Individuals domain

The roles and contributions of individuals in the implementation of allied health professional role substitution models of care.

The implementation of allied health professional role substitution models of care heavily relies on the engagement of various individuals who play pivotal roles in the process. Through our interviews, participants identified nine key roles integral to the implementation and evaluation of these alternative healthcare delivery models. These roles, aligned with those in the individuals’ domain of the CFIR, encompassed high-level leaders, mid-level leaders, opinion leaders, implementation facilitators, implementation leads, implementation team members, other implementation support, innovation deliverers, and innovation recipients. Our analysis revealed representation across these roles within our study population, demonstrating the diverse range of contributions.

Participants described the characteristics of these individuals, which we analysed based on the Capability, Opportunity, Motivation-Behaviour (COM-B) theoretical behaviour change model integrated into the CFIR framework. This system evaluates individuals' influence on the implementation process across four constructs: Need, Capability, Opportunity, and Motivation. These constructs assess individuals' deficits addressed by the models of care, their interpersonal competence, availability and power, and commitment and motivation in fulfilling their roles respectively.

Participants emphasised the critical role of medical and executive buy-in for the success of these models. Without their support and commitment, implementation efforts often faced significant hurdles. As one participant stated, " Medical and executive buy-in, if they are not supportive, it doesn’t happen " (P34, Workforce and Education). Furthermore, participants highlighted the importance of strong endorsement from medical professionals and the need for active engagement from allied health clinicians and managers to ensure the sustainability of these models. As articulated by another participant, " Allied health clinicians and even up into the level of our managers, there's certain spheres of influence that we have, but to make something like this come together and to be able to make it sustainable, you really need strong medical endorsement and that real commitment to push it " (P12, Allied Health Clinician).

Moreover, participants identified the Allied Health Office as having a crucial role in facilitating implementation. However, they also expressed the need for greater visibility and recognition of successful implementation efforts. As one participant suggested, " The Allied Health Office has a role to play in that. I think we should certainly see more things up in lights, you know, presentations, success stories et cetera and opportunities for these models to be shared and celebrated more widely across the state " (P15, Implementation Support).

A matrix analysis (Table 2 ) provides detailed insights into the roles and characteristics of individuals within different groups/roles. This elucidates their contributions to the successful implementation of professional role substitution models of care, as perceived by the study participants.

Implementation process

Essential phases and strategies for effective implementation of allied professional role substitution healthcare models.

Participants in our study provided insights into key stages necessary for implementing allied health professional role substitution models of care. We analysed their responses and mapped them to constructs in the implementation process domain of the CFIR, finding alignment with five out of the nine constructs. In the planning phase, participants emphasised the importance of conducting needs assessments and developing comprehensive implementation plans to identify gaps, set objectives, and consider resources and stakeholders' roles. One participant described, "At the start-up of our model of care, we had a series of meetings involving all stakeholders... to develop very clear guidance and pathways for how patients would move through these services " (P13, Allied Health Clinician).

Engaging was highlighted as crucial focused on involving diverse stakeholders, from healthcare providers to patients, forming multi-stakeholder teams to ensure a variety of perspectives and support for long-term sustainability. " There were a broad range of stakeholders involved in the task force across Queensland Health and external to Queensland ." (P34, Workforce and Education) In the doing phase models of care often started as pilot projects, with services developing iteratively.

Reflecting and evaluation Participants stressed the importance of building evaluation into the model of care to ensure sustainability and strategic outcomes. However, challenges such as limited time and funding were acknowledged, as one participant stated, "We don't get the time or the funding in my experienc e" (P1, Allied Health Clinician). Lastly , in adapting , participants recognised the need for continuous learning and tailored strategies to the local context, acknowledging the necessity for flexibility in response to evolving healthcare systems. Moreover, strategies to enhance evaluation included dedicated funding, external evaluation to reduce bias, development of performance frameworks, and tailored technology and digital systems allowing data collection and analysis at the point of care. Collaboration with universities and the use of research frameworks and grants were also seen by participants as facilitators to enhance performance measurement.

Implementation and innovation outcomes

The assessment of outcomes derived from allied health professional role substitution models.

In our study, participants highlighted the importance of evaluating healthcare models' success and failure, focusing on both implementation process and innovation outcomes. They identified eight key domains, including implementation aspects such as adoptability, implementability, and sustainability, as well as innovation delivery outcomes like effectiveness, safety, patient-centeredness, healthcare provider experience, access, activity, and economic evaluation. One participant stressed the need for thoughtful measurement, stating , “You do need to think about what you need to measure to prove the value of your service .” (P35, Allied Health Leader). Figure 2 summarises these outcomes and provides examples of measures discussed by participants.

Recommended outcomes and examples to measure the impact of professional role substitution models of care

Implementation outcomes

Participants shared diverse perspectives on implementation success for allied health professional role substitution models, with factors like regulatory environment, financing, medical acceptance, stakeholder engagement, and individual characteristics playing key roles. Sustainability was particularly highlighted, as expressed by a participant, “ You need to know that a service that has been implemented is still running after several years ” (Participant 1, Allied Health Clinician).

Innovation outcomes

Participants emphasised specific outcomes in evaluating the impact of allied health professional role substitution models of care. One participant stressed the importance of measuring performance and demonstrating improved access and cost saving, saying, " I think it is important to measure performance and to show that there is improved access and economic benefits. You know, to show that the service is doing what it was intended to do" (P5, Allied Health Clinician). They also highlighted the need to track activity data, with another participant mentioning, "We basically keep data on all of the occasions of service, how many patients are seen within the service, and how many patients are discharged." (Participant 8, Allied Health Clinician).

Healthcare provider experience, including clinician and patient satisfaction, emerged as essential, with one participant suggesting : "Surveying the general practitioners would be a good way of doing it as well, asking if they are happy with the service" (P36, Executive Leader). Continuity of care and patient centredness were also emphasised. "It's actually quite heartening hearing what our patients value and to see if our services line up with that " (P29, Allied Health Clinician). Participants in the study also stressed the importance of safety as a crucial outcome measure in evaluating allied health professional role substitution models of care. One general practitioner (P7) highlighted this by stating, "We need to know we are providing great healthcare to patients. You know that we are reducing harm, not causing harm, and hopefully not missing diagnoses”.

Finally, participants perceived effectiveness as paramount for assessing the success and impact of the model on patient care experiences and health outcomes. One executive leader (P36) expressed: "Forgetting about everything else, the patients' view of whether or not they've been treated adequately to me is the most important. If there are no outcomes with the model of care, the patients won't be satisfied, and they will say so."

This study investigated factors influencing the implementation and performance evaluation of allied health professional role substitution models of care using the CFIR framework. We identified six overarching themes aligned with CFIR domains and outcomes. These themes covered dynamics such as innovation catalysts, evidence, advantages, and disadvantages; external factors affecting implementation and evaluation; internal structural, political, and cultural contexts; roles and contributions of individuals; essential implementation phases and strategies; and assessment of model outcomes. Our analysis identified twenty-seven underlying constructs and subconstructs within the CFIR framework that influence professional role substitution implementation. Additionally, we identified ten key constructs across implementation and innovation outcome categories: adoptability, sustainability, implementability, effectiveness, safety, patient-centeredness, accessibility, healthcare provider experiences, service delivery metrics, and economic evaluations. These findings addressed critical questions regarding factors influencing implementation and methods for assessing the impact of care models. Overall, this study provides a robust framework for implementing and evaluating allied health professional role substitution models, effectively addressing gaps in literature and practice.

Priority areas of focus

While prior studies have demonstrated the potential benefits of these models in terms of providing safe, effective, and cost-efficient care, [ 10 , 11 ] the current research goes further by exploring stakeholders' perceptions and experiences in depth. Grounded in the Consolidated Framework for Implementation Research (CFIR), [ 36 , 37 , 38 ] it explores the multifaceted factors influencing the adoption and integration of these models within healthcare systems.

Healthcare organisations play a significant role in either facilitating or impeding the implementation of professional role substitution models of care [ 41 ]. Along with previous research , this work underscores the significance of supportive organisational cultures, adequate resources, leadership commitment, and medical endorsement as critical factors for the successful adoption of such models [ 42 ]. Conversely, factors like resistance to change, resource limitations, and insufficient infrastructure can pose significant barriers that must be addressed to ensure successful implementation.

Traditional healthcare structures were once considered conducive to advancing medical sciences [ 43 ]. However, recent reviews have highlighted how entrenched organisational cultures and long-held traditions within healthcare settings may now act as barriers to alternative models of practice and hinder improvements in healthcare access for the community [ 41 ]. Consequently, healthcare organisations must proactively assess their readiness for new models and develop strategies to overcome these barriers. Leveraging the constructs and principles identified in the inner setting domain of this study is essential for cultivating a culture that fosters role substitution and innovation in healthcare delivery.

Stakeholders' perceptions and attitudes play a significant role in shaping the success of professional role substitution models of care, influenced by factors like medical buy-in, leadership support, and engagement strategies [ 41 , 44 , 45 ]. Effective stakeholder engagement strategies, alongside tailored training, communication programs and ongoing support mechanisms, emerge as crucial tools for addressing individual concerns and fostering buy-in from all involved parties. These findings align with similar studies in physiotherapy and nursing, emphasising the universal importance of considering individual perspectives in healthcare implementation efforts [ 41 , 44 , 45 , 46 ].

This research emphasises the importance of incorporating perspectives from patients and innovation recipients to enhance the success of healthcare interactions. Integrating these viewpoints strengthens the potential for sustainable adoption of evidence-based innovations, promoting patient-centred care [ 47 , 48 , 49 ]. Patient involvement in co-designing and evaluating alternative healthcare models improves trust and acceptance, highlighting the significance of collaboration and patient engagement strategies for optimising implementation and evaluation processes [ 27 , 28 , 47 ].

Performance evaluation plays a pivotal role in assessing the implementation of professional role substitution models of care [ 25 ]. Monitoring various factors, including outcomes, patient satisfaction, quality of care, safety, healthcare professionals’ performance, healthcare system efficiency, and cost-effectiveness, can provide valuable insights for ongoing improvement, optimisation, and sustainability of models of care [ 25 , 50 ]. We address gaps highlighted in previous research, particularly concerning the lack of comprehensive evaluations and guidance on outcome measures [ 10 , 22 , 25 ]. Many current frameworks lack specificity in identifying key metrics relevant to professional role substitution models [ 25 , 26 ]. However, this study delineating eight key outcome measures emphasises a data-driven approach to decision-making. This represents an advancement in the field, providing a structured framework for assessing the impact and value of these models.

Implications for policy, practice and future research

In combination with existing literature in various alternative healthcare delivery models, this study highlights the shared challenges and opportunities across healthcare professions and settings [ 41 , 45 , 46 ]. Our analysis of implementation considerations, stakeholder perspectives, and outcome measures, advances theoretical understanding and also provides practical guidance for real-world implementation and evaluation. These insights can be extended beyond Australia's healthcare system, with implications for policy development, collaboration, knowledge exchange, and healthcare delivery practices in other regions.

In practice, maximising the effectiveness and sustainability of professional role substitution models necessitates comprehensive training and education initiatives for health professionals. Collaborating with professional bodies and universities can standardise training, provide continuous professional development opportunities, and address individual factors that impact implementation readiness for alternative healthcare delivery models.

Adapting regulatory frameworks to the evolving healthcare landscape is paramount, necessitating clear guidelines and legal frameworks to delineate practice boundaries and facilitate the seamless implementation of expanded roles. Adequate funding is critical to support various aspects, including staffing, infrastructure development, establishment of incentivising reimbursement models, research, evaluation, and ensuring ongoing sustainability. Prioritising evidence-based policy development, informed by comprehensive evaluation of care models, is essential to ensure alignment with best practices and standards of care. Integrating standard outcome measures into evaluation frameworks is crucial for accurately assessing the impact and effectiveness of care models, thereby enabling informed decision-making based on evidence. The research we have conducted supports these assertions, emphasising the importance of these factors for the successful implementation and sustainability of alternative healthcare delivery models.

Our findings may therefore serve as a catalyst for discussion and debate on allied health professional role substitution and other alternative healthcare delivery models, guiding future research endeavours. Exploring longitudinal studies to gauge sustainability and long-term impact, conducting comparative analyses across diverse settings and patient populations, and conducting qualitative inquiries to identify implementation and evaluation facilitators and barriers are critical. Additionally, research in health economics, health information technology, policy analysis, and interprofessional collaboration can provide valuable insights to optimise implementation practices and enhance the applicability of these models across different healthcare systems and cultural contexts.

Strengths and limitations

The strengths of this study lie in the diverse range of stakeholders involved, including key opinion leaders, decision makers, allied health clinicians, medical professionals, policymakers, healthcare administrators, university partners, professional bodies, advocates, and patients. The inclusion of participants with varied experiences enhances the robustness of the findings. Purposeful sampling with maximum variation further improves the transferability of the results.

The use of the COREQ-checklist and independent co-coding and discussions among the research team enhance the credibility, trustworthiness, and transparency of the study [ 31 ]. Another notable strength is the use of the Consolidated Framework for Implementation Research (CFIR) to guide the analysis, which helped identify and organise themes into multi-level intervention principles that influence implementation effectiveness. It should be noted that the CFIR was not used to guide data collection, as is often practiced, [ 38 ] as this may have limited the exploration of qualitative themes relevant to the research question but not explicitly aligned with CFIR domains and constructs.

As the study was conducted in Australia, the generalisability of the findings to other stakeholders or healthcare contexts in different countries and settings may be limited. Additionally, as with any research involving human subjects, the possibility of self-selection bias influencing the results cannot be excluded, and the findings should be interpreted with this in mind. Insights gained from this study may also have broader implications for other countries facing similar challenges in healthcare delivery. By examining similarities and differences in healthcare systems and regulatory environments, countries can however learn from Queensland’s experiences adopting strategies to support the implementation of role substitution models.

In conclusion, this study provides a systematic examination of the key elements and principles influencing the implementation and performance evaluation of professional role substitution models of care. By understanding the multifaceted nature of these factors and addressing the challenges and opportunities associated with expanded healthcare roles, healthcare systems can navigate complexities and capitalise on opportunities. This holistic approach, involving collaboration among stakeholders and considering patient safety, quality of care, and optimal healthcare outcomes will contribute to the development of more efficient, equitable, sustainable, and patient-centred models of care and healthcare systems.

Availability of data and materials

Data is available from corresponding author on reasonable request.

Abbreviations

Consolidated Framework for Implementation Research

Consolidated Criteria for Reporting Qualitative Research

Physician Assistants

Duckett SJ. Health workforce design for the 21st century. Aust Health Rev. 2005;29(2):201–10.

Article   PubMed   Google Scholar  

Segal L, Bolton T. Issues facing the future health care workforce: the importance of demand modelling. Aust New Zealand Health Policy. 2009;6(1):12.

Article   PubMed   PubMed Central   Google Scholar  

Commission AGP. Productity Commision Research Report https://www.pc.gov.au/inquiries/completed/health-workforce/report/healthworkforce.pdf2006 [Available from: https://www.pc.gov.au/inquiries/completed/health-workforce/report/healthworkforce.pdf .

Dower C, Moore J, Langelier M. It is time to restructure health professions scope-of-practice regulations to remove barriers to care. Health Affairs. 2013;32(11):1971–6.

Porter ME, Lee TH. The strategy that will fix health care. Harvard business review. 2013;91(10):1–19.

Google Scholar  

Baker A. Crossing the quality chasm: a new health system for the 21st century. Br Med J Publishing Group. 2001;323(7322):1192.

Owens DK, Qaseem A, Chou R, Shekelle P. High-value, cost-conscious health care: concepts for clinicians to evaluate the benefits, harms, and costs of medical interventions. Ann Intern Med. 2011;154(3):174–80.

Roberts N, Carrigan A, Clay-Williams R, Hibbert PD, Mahmoud Z, Pomare C, et al. Innovative models of healthcare delivery: an umbrella review of reviews. BMJ open. 2023;13(2):e066270.

Wang H, English M, Chakma S, Namedre M, Hill E, Nagraj S. The roles of physician associates and advanced nurse practitioners in the National Health Service in the UK: a scoping review and narrative synthesis. Human Resour Health. 2022;20(1):69.

Article   CAS   Google Scholar  

McPherson K, Kersten P, George S, Lattimer V, Breton A, Ellis B, et al. A systematic review of evidence about extended roles for allied health professionals. J Health Serv Res Policy. 2006;11(4):240–7.

Martin-Misener R, Kilpatrick K, Donald F, Bryant-Lukosius D, Rayner J, Valaitis R, et al. Nurse practitioner caseload in primary health care: scoping review. Int J Nurs Stud. 2016;62:170–82.

Barnett M, Balkissoon C, Sandhu J. The level of quality care nurse practitioners provide compared with their physician colleagues in the primary care setting: A systematic review. J Am Assoc Nurse Pract. 2022;34(3):457–64.

Middleton S, Gardner A, Della PR, Lam L, Allnutt N, Gardner G. How has the profile of Australian nurse practitioners changed over time? Collegian. 2016;23(1):69–77.

Freund T, Everett C, Griffiths P, Hudon C, Naccarella L, Laurant M. Skill mix, roles and remuneration in the primary care workforce: who are the healthcare professionals in the primary care teams across the world? Int J Nurs Stud. 2015;52(3):727–43.

Halter M, Drennan V, Chattopadhyay K, Carneiro W, Yiallouros J, de Lusignan S, et al. The contribution of physician assistants in primary care: a systematic review. BMC Health Serv Res. 2013;13:1–13.

Article   Google Scholar  

Henry LR, Hooker RS, Yates KL. The role of physician assistants in rural health care: a systematic review of the literature. J Rural Health. 2011;27(2):220–9.

Terry B, Bisanzo M, McNamara M, Dreifuss B, Chamberlain S, Nelson SW, et al. Task shifting: meeting the human resources needs for acute and emergency care in Africa. Afr J Emerg Med. 2012;2(4):182–7.

Mijovic H, McKnight J, English M. What does the literature tell us about health workers’ experiences of task-shifting projects in sub-Saharan Africa? A systematic, qualitative review. J Clin Nurs. 2016;25(15–16):2083–100.

Michel JE, D Tediosi F, D Egger MB, T McIntyre DR. Lest we forget, primary health care in Sub-Saharan Africa is nurse led. Is this reflected in the current health systems strengthening undertakings and initiatives? J Global Health Rep. 2018;2:e2018009.

Kengne AP, Sobngwi E, Fezeu LL, Awah PK, Dongmo S, Mbanya JC, et al. Nurse-led care for asthma at primary level in rural sub-Saharan Africa: the experience of Bafut in Cameroon. J Asthma. 2008;45(6):437–43.

Eberly LA, Rusingiza E, Park PH, Ngoga G, Dusabeyezu S, Mutabazi F, et al. 10-year heart failure outcomes from nurse-driven clinics in rural sub-Saharan Africa. J Am Coll Cardiol. 2019;73(8):977–80.

Saxon RL, Gray MA, Oprescu FI. Extended roles for allied health professionals: an updated systematic review of the evidence. J Multidisciplinary Healthcare. 2014;7:479–88.

Welfare AIoHa. Australia's health performance framework 2017 [Available from: https://www.aihw.gov.au/reports-data/indicators/australias-health-performance-framework .

Queensland Health Clinical Excellence Division. Allied Health Expanded Scope Strategy 2016–2021. 2016. Available from https://www.health.qld.gov.au/__data/assets/pdf_file/0024/449520/expanded-strategy.pdf . Accessed 09 July 2024.

Mutsekwa RN, Wright C, Byrnes JM, Canavan R, Angus RL, Spencer A, et al. Measuring performance of professional role substitution models of care against traditional medical care in healthcare—A systematic review. J Eval Clin Pract. 2022;28(2):208–17.

Levesque J-F, Sutherland K. Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework. BMC Health Serv Res. 2020;20(1):1–14.

Mutsekwa RN, Byrnes JM, Larkins V, Canavan R, Angus RL, Campbell KL. Role substitution of specialist medical doctors with allied-health professionals: A qualitative exploration of patients’ experiences and perceptions of healthcare quality. J Eval Clin Pract. 2022;28(6):1096–105.

Mutsekwa RN, Campbell KL, Canavan R, Mulhern B, Angus RL, Byrnes JM. Patient preferences for attributes that characterise alternative models of care in gastroenterology: a discrete choice experiment. Patient. 2023;16(2):165–77.

Mutsekwa RN, Campbell KL, Canavan R, Angus RL, McBride L-J, Byrnes JM. Measuring performance in allied health professional role substitution models of care: a clinician survey. BMC Health Serv Res. 2024;24(1):79.

Galletta A. Mastering the semi-structured interview and beyond: From research design to analysis and publication, vol. 18. NYU press; 2013.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Government Q. Fact Sheet: Allied Health and Oral Health 2019 [Available from: https://www.health.qld.gov.au/__data/assets/pdf_file/0027/1067571/Allied-Health-and-Oral-Health-2019.pdf .

Queensland Health Allied Health Professions’ Office of Queensland. Ministerial Taskforce on health practitioner expanded scope of practice: final report. 2014. Available from: Ministerial taskforce final report ( health.qld.gov.au ). Accessed 09 July 2024.

Queensland Government. Hospital care, surgical procedures and waiting lists 2015 [Available from: https://www.qld.gov.au/health/services/hospital-care/waiting-lists .

Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–907.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):1–16.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):1–15.

Damschroder LJ, Reardon CM, Opra Widerquist MA, Lowery J. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):1–10.

Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qualitat Res Sport Exerc Health. 2019;11(4):589–97.

Carter N. The use of triangulation in qualitative research. Number 5/September 2014. 1969;41(5):545–7.

Wiggins D, Downie A, Engel RM, Brown BT. Factors that influence scope of practice of the five largest health care professions in Australia: a scoping review. Human Resour Health. 2022;20(1):1–13.

Morris J, Grimmer K, Gilmore L, Perera C, Waddington G, Kyle G, et al. Principles to guide sustainable implementation of extended-scope-of-practice physiotherapy workforce redesign initiatives in Australia: stakeholder perspectives, barriers, supports, and incentives. J Multidiscip Healthc. 2014;7:249–58.

Fiorio CV, Gorli M, Verzillo S. Evaluating organizational change in health care: the patient-centered hospital model. BMC Health Serv Res. 2018;18:1–15.

Wiles L, Milanese S. Stakeholder perspectives of the Extended Scope Physiotherapy Practitioner (ESPP) role in Australia–a qualitative study. Physical Therapy Rev. 2016;21(3–6):228–35.

Parker RM, Desborough JL, Forrest LE. Stakeholder perceptions of a nurse led walk-in centre. BMC Health Serv Res. 2012;12:1–7.

Bryant-Lukosius D, DiCenso A. A framework for the introduction and evaluation of advanced practice nursing roles. J Adv Nurs. 2004;48(5):530–40.

Harding P, Prescott J, Block L, O’Flynn AM, Burge AT. Patient experience of expanded-scope-of-practice musculoskeletal physiotherapy in the emergency department: a qualitative study. Australian Health Rev. 2015;39(3):283–9.

Browne K, Roseman D, Shaller D, Edgman-Levitan S. Measuring patient experience as a strategy for improving primary care. Health Affairs. 2010;29(5):921–5.

Reeve S, May S. Exploration of patients’ perspectives of quality within an extended scope physiotherapists’ spinal screening service. Physiother Theory Pract. 2009;25(8):533–43.

Moretto N, Stute M, Sam S, Bhagwat M, Raymer M, Buttrum P, et al. A uniform data set for determining outcomes in allied health primary contact services in Australia. Aus J Prim Health. 2020;26(1):58–69.

Download references

Acknowledgements

A special thanks to the participants who gave up their time to share their experiences and perceptions on implementation and performance evaluation of professional role substitution models of care.

This work was supported by the Gold Coast Hospital and Health Service Collaborative Research Grant [grant number RGS20190041].

Author information

Authors and affiliations.

Gold Coast Hospital and Health Service, Nutrition and Food Services, 1 Hospital Boulevard, Southport, Queensland, 4215, Australia

Rumbidzai N. Mutsekwa & Rebecca L. Angus

Gold Coast Hospital and Health Service, Allied Health Research Team, 1 Hospital Boulevard, Southport, Queensland, 4215, Australia

Rumbidzai N. Mutsekwa

Centre for Applied Health Economics, School of Medicine, Griffith University, Sir Samuel Griffith Centre, 1 Parklands Drive, Nathan, Queensland, 4111, Australia

Rumbidzai N. Mutsekwa, Katrina L. Campbell & Joshua M. Byrnes

Menzies Health Institute Queensland, Griffith University, Gold Coast Campus, 1 Parklands Drive, Southport, Queensland, 4215, Australia

Katrina L. Campbell & Joshua M. Byrnes

Healthcare Excellence and Innovation, Metro North Hospital and Health Service, 153 Campbell Street, Bowen Hills, Queensland, 4029, Australia

Katrina L. Campbell

Gastroenterology Department, Gold Coast Hospital and Health Service, 1 Hospital Boulevard, Southport, Queensland, 4215, Australia

Russell Canavan

School of Health Sciences and Social Work, Griffith University, Gold Coast Campus, 1 Parklands Drive, Southport, Queensland, 4215, Australia

Rebecca L. Angus

Department of Health, Clinical Excellence, 15 Butterfield Street, Herston, Queensland, 4006, Australia

Liza-Jane McBride

You can also search for this author in PubMed   Google Scholar

Contributions

RNM, RC, KC, RLA, LJM, JB contributed to conception and design of the study. RM led the data collection and analysis and wrote the initial draft of the manuscript. RNM, KC, RC, RLA, LJM and JB interpreted results, critically revised the manuscript, and approved the final version.

Corresponding author

Correspondence to Rumbidzai N. Mutsekwa .

Ethics declarations

Ethics approval and consent to participate.

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Gold Coast Hospital and Health Service (HREC/2020/QGC/62104) and Griffith University (GU Ref No: 2020/876) Human Research Ethics Committees

Participants provided informant consent and completed participant consent and information statements.

Consent for publication

All authors consented for the manuscript to be submitted for publication.

Competing interests

None declared. Participants provided informant consent and completed participant consent and information statements.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mutsekwa, R.N., Campbell, K.L., Canavan, R. et al. Unlocking potential: a qualitative exploration guiding the implementation and evaluation of professional role substitution models in healthcare. Implement Sci Commun 5 , 73 (2024). https://doi.org/10.1186/s43058-024-00611-x

Download citation

Received : 22 December 2023

Accepted : 28 June 2024

Published : 12 July 2024

DOI : https://doi.org/10.1186/s43058-024-00611-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Professional role substitution
• Scope of practice
• Implementation
• Measuring performance

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Systematic review
• Open access
• Published: 15 July 2024

Teamwork and implementation of innovations in healthcare and human service settings: a systematic review

• Elizabeth A. McGuier   ORCID: orcid.org/0000-0002-6219-6358 1 ,
• David J. Kolko 1 ,
• Gregory A. Aarons 2 , 3 , 4 ,
• Allison Schachter 5 , 6 ,
• Mary Lou Klem 7 ,
• Matthew A. Diabes 8 ,
• Laurie R. Weingart 8 ,
• Eduardo Salas 9 &
• Courtney Benjamin Wolk 5 , 6  

Implementation Science volume  19 , Article number:  49 ( 2024 ) Cite this article

610 Accesses

10 Altmetric

Metrics details

Implementation of new practices in team-based settings requires teams to work together to respond to new demands and changing expectations. However, team constructs and team-based implementation approaches have received little attention in the implementation science literature. This systematic review summarizes empirical research examining associations between teamwork and implementation outcomes when evidence-based practices and other innovations are implemented in healthcare and human service settings.

We searched MEDLINE, CINAHL, APA PsycINFO and ERIC for peer-reviewed empirical articles published from January 2000 to March 2022. Additional articles were identified by searches of reference lists and a cited reference search for included articles (completed in February 2023). We selected studies using quantitative, qualitative, or mixed methods to examine associations between team constructs and implementation outcomes in healthcare and human service settings. We used the Mixed Methods Appraisal Tool to assess methodological quality/risk of bias and conducted a narrative synthesis of included studies. GRADE and GRADE-CERQual were used to assess the strength of the body of evidence.

Searches identified 10,489 results. After review, 58 articles representing 55 studies were included. Relevant studies increased over time; 71% of articles were published after 2016. We were unable to generate estimates of effects for any quantitative associations because of very limited overlap in the reported associations between team variables and implementation outcomes. Qualitative findings with high confidence were: 1) Staffing shortages and turnover hinder implementation; 2) Adaptive team functioning (i.e., positive affective states, effective behavior processes, shared cognitive states) facilitates implementation and is associated with better implementation outcomes; Problems in team functioning (i.e., negative affective states, problematic behavioral processes, lack of shared cognitive states) act as barriers to implementation and are associated with poor implementation outcomes; and 3) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier.

Conclusions

Teamwork matters for implementation. However, both team constructs and implementation outcomes were often poorly specified, and there was little overlap of team constructs and implementation outcomes studied in quantitative studies. Greater specificity and rigor are needed to understand how teamwork influences implementation processes and outcomes. We provide recommendations for improving the conceptualization, description, assessment, analysis, and interpretation of research on teams implementing innovations.

Trial registration

This systematic review was registered in PROSPERO, the international prospective register of systematic reviews. Registration number: CRD42020220168.

Peer Review reports

Contributions to the Literature:

This paper reviews more than 20 years of research on teams and implementation of new practices in healthcare and human service settings.

We concluded with high confidence that adaptive team functioning is associated with better implementation outcomes and problems in team functioning are associated with poorer implementation outcomes. While not surprising, the implementation science literature has lacked clear empirical evidence for this finding.

Use of the provided recommendations will improve the quality of future research on teams and implementation of evidence-based practices.

Healthcare and human service providers (e.g., clinicians, case managers) often work in team-based settings where professionals work collaboratively with one another and service recipients toward shared goals [ 1 , 2 ]. Team-based care is intended to include multiple professionals with varying skills and expertise [ 1 , 3 ]. It requires shared responsibility for outcomes and increases team members’ dependence on one another to complete work [ 1 , 3 , 4 ]. Effective team-based care and higher quality teamwork are associated with improvements in care access and quality, patient safety, patient satisfaction, clinical outcomes, and costs [ 2 , 4 , 5 , 6 , 7 , 8 , 9 ].

We use the term ‘teamwork’ to refer to an array of team constructs using the input-mediator-outcome-input (IMOI) framework (Fig.  1 ) [ 10 , 11 , 12 ]. The IMOI framework recognizes that team interactions are dynamic and complex, with processes unfolding over time and feedback loops between processes, outcomes, and inputs [ 10 ]. Team inputs include team structure and composition, task demands, and contextual features [ 13 ]. Mediators are aspects of team functioning (i.e., what team members think, feel, and do [ 12 ]) through which inputs influence outcomes. These processes and emergent states may be cognitive, affective, or behavioral [ 5 , 14 , 15 , 16 ]. Team effectiveness outcomes are multidimensional and include team performance as well as team viability and the impact of the team on members’ development [ 12 , 17 , 18 , 19 ].

Conceptual model of team effectiveness and key terminology. Figure adapted from “Advancing research on teams and team effectiveness in implementation science: An application of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework” by E.A. McGuier, D.J. Kolko, N.A. Stadnick, L. Brookman-Frazee, C.B. Wolk, C.T. Yuan, C.S. Burke, & G.A. Aarons, 2023, Implementation Research and Practice , 4 , 26334895231190855. [CC BY-NC]

Implementation of new practices in team-based service settings requires team members to work together to respond to changing demands and expectations. Extensive research has identified barriers and facilitators to implementation of new practices at the individual provider, organization, and system levels; however, the team level has received little empirical attention [ 20 , 21 ]. This is a problem because implementation efforts increasingly rely on teams, and responses to a new practice are likely to be influenced by team characteristics and processes. See McGuier and colleagues [ 20 ] for an overview of team constructs in the context of implementation science and the Exploration, Preparation, Implementation, Sustainment (EPIS) framework [ 22 , 23 ]. Given increasing use of team-based care and interest in implementation strategies targeting teams, examining how teamwork is associated with implementation processes and outcomes is critical. This systematic review identified and summarized empirical research examining associations between teamwork and implementation outcomes when evidence-based practices (EBPs) and other innovations were implemented in healthcare and human service settings.

This systematic review was registered (PROSPERO; registration number: CRD42020220168) and conducted following the published protocol [ 24 ]. The review was conducted in accordance with PRISMA and SWiM guidance [ 25 , 26 ]; relevant checklists are in Additional File 1.

Information sources and search strategy

We searched the following databases: MEDLINE (Ovid), CINAHL (Ebsco), APA PsycINFO (Ovid), and ERIC (Ebsco). Database searches were run on August 7, 2020, and again on March 8, 2022. For all searches, a publication date from 2000 to current was applied; there were no language restrictions (see [ 24 ]). An experienced health sciences librarian (MLK) designed the Ovid MEDLINE search and translated that search for use in the other databases (see additional file in [ 24 ]). The search strings consisted of controlled vocabulary (when available) and natural language terms representing concepts of teamwork and implementation science or innovation or evidence-based practice. Results were downloaded to an EndNote (version X9.3.3) library and duplicate records removed [ 27 ]. Additional relevant articles were identified by hand searches of reference lists of included articles, a cited reference search for included articles in the Web of Science (Clarivate) bibliographic database (completed in February 2023), and requests sent to implementation science listservs and centers for suggestions of relevant articles.

Eligibility criteria

We included empirical journal articles describing studies using quantitative, qualitative, or mixed methods. Study protocols, reviews, and commentaries were excluded. All studies were conducted in healthcare or human service settings (e.g., hospitals, clinics, child welfare) and described the implementation of a practice to improve patient care. Studies of interventions to improve teamwork (e.g., team building interventions) and studies of teams created to implement the innovation (e.g., quality improvement teams, implementation support teams) were excluded. Eligible studies assessed at least one team construct and described its influence on implementation processes and outcomes.

Changes from protocol

Several changes were made from our systematic review protocol (PROSPERO CRD42020220168; [ 24 ]). Specifically, during the full-text review stage, we broadened the scope from team functioning (i.e., processes and states) to include team structure and performance because of the small number of studies that assessed and reported specific processes or states. This change increased the number of included studies. Similarly, because implementation outcomes were often inconsistently defined and poorly reported [ 28 , 29 , 30 ], we broadened our scope to include studies that identified team constructs as implementation determinants (i.e., barriers/facilitators) without explicitly defining and measuring an implementation outcome. Because of changes in university access to bibliographic databases, the cited reference search was performed in the Web of Science only instead of the Web of Science and Scopus. This bibliographic database indexes more than 21,000 scientific journals [ 31 ]. Lastly, because of time and resource constraints, we did not search conference abstracts or contact authors for unreported data.

Selection process and data extraction

Title/abstract screening and review of full-text articles were conducted by pairs of trained independent reviewers in DistillerSR. Conflicts were resolved through re-review, discussion between reviewers, and when needed, discussion with a senior team member (EAM). A final review of all included articles was conducted by EAM. Relevant data from each article was extracted into an Excel spreadsheet by one reviewer (AS). A second reviewer (EAM) conducted a line-by-line review and verification. Our data extraction form was informed by existing forms and guides (e.g., [ 32 , 33 ]). For each included study, we extracted information on measures of teamwork and implementation-relevant outcomes, characteristics of the setting, teams, and participants, analysis methods, and results. For quantitative studies, we recorded correlation coefficients and/or regression coefficients as standardized metrics of association. For qualitative studies, we recorded themes [ 33 ].

Quality and risk of bias assessment

The Mixed Methods Appraisal Tool (MMAT) [ 34 ] was used to evaluate quality and risk of bias for each included study. Multiple publications from the same study were evaluated separately because they reported different outcomes. Consistent with Powell and colleagues [ 35 ], quality evaluations were only made for the components of the study relevant to our question. Quality evaluations were conducted by two independent reviewers (EAM, MAD) with discrepancies resolved through consensus discussion. After completing the MMAT, the reviewers jointly categorized each article as high, moderate, or low quality. High quality studies were those with affirmative responses to all MMAT questions. Moderate quality studies had at least one minor methodological problem, and low-quality studies had serious flaws (e.g., qualitative studies with poor coherence between data, analysis, and conclusions; quantitative studies with biased samples and/or inappropriate statistical analyses).

We rated the relevance of each publication to our research question as high, moderate, or low. Highly relevant studies reported implementation of a well-defined innovation, thoroughly described team constructs and implementation outcomes, and clearly linked team constructs to implementation outcomes. Most studies rated as low relevance provided very limited information about teamwork and/or implementation outcomes. Studies that only described barriers/facilitators were rated as low or moderate relevance. Ratings were conducted by two independent reviewers (EAM, CBW) with discrepancies resolved through consensus discussion.

Data synthesis

We conducted a narrative synthesis of included studies following guidelines for synthesis without meta-analysis (SWiM) [ 36 ]. We prioritized reporting of high quality, highly relevant studies. Studies categorized as low quality and/or low relevance were not included in the synthesis but are included in the description of study characteristics to convey the breadth of the literature. We organized studies based on the IMOI framework (i.e., team inputs, processes/states, and outputs) and organized studies of processes/states by affective, behavioral, and cognitive constructs when possible. Because of the heterogeneity in team constructs and implementation outcomes, we were unable to quantitatively synthesize results using meta-analysis or formally investigate heterogeneity; this challenge is common in implementation science systematic reviews [ 30 ]. We assessed the strength of the overall body of evidence with GRADE for quantitative studies [ 37 ] and GRADE-CERQual for qualitative studies [ 38 , 39 ]. GRADE results in ratings of high, moderate, low, or very low quality of evidence for each outcome of interest. GRADE-CERQual results in ratings of high, moderate, low, or very low confidence in each review finding. GRADE ratings were made independently with discrepancies resolved through consensus discussion; GRADE-CERQual ratings were made through iterative discussions as recommended [ 39 ]. All ratings and decisions were made by the first and senior authors.

Search results

Our initial search, after removal of duplicates, yielded 7181 results. The second search (August 2020-March 2022) captured an additional 1341 results. The cited reference search yielded 1961 results. A total of 10,489 results were included in title/abstract review. Figure  2 provides a PRISMA flow diagram for included studies. After full-text review, 58 articles from 55 studies were included in analyses [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 ].

PRISMA flow diagram of included articles. From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. https://doi.org/10.1136/bmj.n71 . For more information, visit: http://www.prisma-statement.org/

As shown in Fig.  3 , publications on teamwork and implementation have increased substantially since 2000. Three articles on this topic (5%) were published between 2000 and 2007, 14 (24%) between 2008 and 2015, and 41 (71%) between 2016 and early 2023.

Included articles by year of publication

Study characteristics

Interrater agreement was good for assessment of study quality (81% agreement on MMAT questions) and ratings of relevance (88% agreement). There were 20 high quality articles, 23 moderate quality articles, and 15 low quality articles. Fourteen articles were rated as high relevance, 22 as moderate, and 22 as low relevance. Only 4 were rated as both high quality and high relevance. We report study characteristics for all 58 eligible articles. Our narrative synthesis includes 32 articles categorized as moderate/high quality and moderate/high relevance; it excludes 26 articles categorized as low quality and/or relevance.

Studies were conducted in inpatient healthcare ( n  = 22), outpatient/ambulatory healthcare ( n  = 21), mental health settings ( n  = 9), and other settings (e.g., residential facilities, multiple settings; n  = 6). There were 33 qualitative, 15 quantitative, and 10 mixed methods studies. All quantitative studies were descriptive observational studies.

Most studies examined team processes/states ( n  = 53); fewer examined team inputs ( n  = 27). Only two studies examined a team effectiveness outcome. The most common implementation outcomes were fidelity ( n  = 16) and other specified implementation outcomes (e.g., “extent of use,” “implementation success”) ( n  = 15). Less frequently identified implementation outcomes included adoption ( n  = 5), sustainment ( n  = 4), reach ( n  = 4), and perceptions of the innovation (e.g., acceptability, appropriateness, feasibility; n  = 3). Approximately one-third of studies ( n  = 21) did not report specific implementation outcomes but described implementation determinants (i.e., barriers and facilitators).

Synthesis: team inputs & implementation outcomes

Team inputs examined in studies included team stability/instability and staffing shortages, aspects of team structure and composition, interdependence, and hierarchy and professional roles. Quantitative findings are presented in Table  1 . A CERQual Summary of Qualitative Findings related to team inputs is shown in Table  2 . A CERQual Evidence Profile is provided in Additional File 2 (Table A1).

Team stability/instability and staffing shortages

Team stability/instability (i.e., consistency in membership over time) was examined in one mixed methods study [ 48 , 49 ] and three qualitative studies [ 70 , 81 , 94 ]. A study of surgical teams found variations in membership stability but no association between stability and “implementation success” (i.e., composite measure based on number of uses of new technique, proportion of uses, and changes in use) [ 48 , 49 ]. The authors suggested that stability facilitates the development of team coordination but that selecting small and exclusive teams may limit the spread of innovations within the organization. Another study found that a dedicated and stable team in which members were selected and trained together in the use of a new surgical technique led to quicker uptake and better integration into practice, theorizing that dedicated and stable teams increased trust, motivation, and collaborative problem-solving [ 81 ]. However, dedicated teams were difficult to sustain, and some sites instead used rotating team members from a larger pool of trained staff. In rural primary care, stability of team members facilitated sustainment of memory care clinics [ 70 ]. Lastly, another study in primary care found mixed perceptions of stable vs. rotating staff when adding a new team role (i.e., health coach); some team members liked rotating through different roles while others wanted more stability [ 94 ]. Across studies, we found that dedicated and stable team members facilitate implementation while instability in team membership is a barrier to implementation (moderate confidence).

Qualitative studies identified staffing shortages and turnover on teams as barriers to implementation [ 50 , 67 , 75 , 78 , 92 ]. In Veterans Health Administration (VA) clinics, “inadequate staffing posed an insurmountable barrier,” hindering communication and delivery of optimal care during the implementation of the patient-centered medical home (PCMH) model [ 92 ]. Similarly, staff shortages, turnover, and high workloads hindered guideline implementation in Kenyan hospitals [ 75 ]. Two studies found negative impacts of staffing shortages and turnover on sustainment. Staff turnover contributed to discontinuity in Dialectical Behavior Therapy (DBT) team members [ 78 ], and appropriate staffing (i.e., ensuring manageable workloads) and blocking time for team members were identified as critical to sustainment of a team-based model in the VA [ 67 ]. We found that staffing shortages and turnover hinder implementation (high confidence).

Team structure/composition

Studies examined multiple aspects of team structure and composition, specifically team size, workload, longevity (i.e., how long team members had worked together), history of change, and team member characteristics. Team size was examined in two studies of DBT. In a mixed methods study, team size was positively correlated with fidelity, and qualitative data suggested that team size may increase as a result of successful implementation [ 47 ]. In contrast, another study found that DBT team size was not associated with the number of DBT components adopted and was negatively associated with reach, suggesting reach may reflect high workloads [ 72 ]. In VA mental health clinics, team workload (i.e., number of patients seen) was negatively associated with sustainment of trauma-focused therapies [ 68 ]. In these studies, team longevity and history of change were not associated with implementation outcomes [ 47 , 68 ]. Team member characteristics, specifically team member competency/expertise, experience, and commitment/engagement, were identified as facilitators of implementation in some qualitative studies [ 40 , 70 , 81 , 84 , 95 ].

Overall, few findings could be made from quantitative studies examining team structure and composition. Two studies of team size found mixed results, and workload, longevity, and history of change were examined in only one study each. Across qualitative studies, we found team member competency/expertise, experience, and commitment/engagement facilitate implementation (moderate confidence).

Team interdependence

One quantitative study examined team interdependence [ 65 ]. In multidisciplinary child abuse teams implementing a mental health screening/referral protocol, task interdependence (i.e., reliance on team members to share resources and coordinate workflows) was positively associated with reach but not time to adoption. Outcome interdependence (i.e., extent to which outcomes are evaluated at the team vs. individual level) was significantly negatively correlated with time to adoption but not reach. Neither task nor outcome interdependence were associated with team members’ perceptions of acceptability, appropriateness, or feasibility of the innovation [ 65 ]. Because only one study examined interdependence, no review findings were made.

Hierarchy & professional roles

Hierarchy, power distributions, and rigid roles were identified as barriers to implementation in several qualitative studies [ 50 , 53 , 74 , 97 ]. Flatter hierarchies (i.e., more equal distribution of power and authority) supported guideline implementation in pediatric primary care; practices with low compliance to guidelines had more hierarchical relationships while practices with high compliance had more shared decision-making [ 97 ]. In a setting with hierarchy and rigid division of roles, nurses trained in an innovation reported concern that their decisions would be questioned by physicians without expertise in the innovation but more authority [ 74 ]. Similarly, in surgical teams, rigid professional roles and a hierarchical team culture constrained open discussion and created contention over how and when a “time-out” should be completed, resulting in inconsistent use and poor fidelity [ 50 , 53 ]. Across studies, we found that in multidisciplinary settings, rigid professional roles, hierarchical relationships, and power differentials are barriers to implementation (moderate confidence).

Summary of team inputs & implementation outcomes

There was no overlap among team input variables and implementation outcomes examined in quantitative studies (Table  1 ). Accordingly, we were unable to generate estimates of effects or ratings of evidence quality. Qualitative review findings are shown in Table  2 . We found: 1) Dedicated and stable team members facilitate implementation while instability in team membership is a barrier to implementation (moderate confidence); 2) Staffing shortages and turnover hinder implementation (high confidence); 3) Team member competency/expertise, experience, and commitment/engagement facilitate implementation (moderate confidence); and 4) In multidisciplinary settings, rigid professional roles, hierarchical relationships, and power differentials are barriers to implementation (moderate confidence).

Synthesis: team processes/states & implementation outcomes

Studies examined overall team functioning as well as specific affective states, behavioral processes, and cognitive states. Quantitative findings are presented in Table  3 , and a GRADE Evidence Profile is provided in Additional File 2 (Table A2). A CERQual Summary of Qualitative Findings related to team processes and states is shown in Table  4 . A CERQual Evidence Profile is provided in Additional File 2 (Table A3).

Overall team functioning

Nine studies examined quantitative associations between overall team functioning and implementation outcomes. Team functioning was positively associated with intervention fidelity in 2 of 3 studies. One study examined implementation of transition programs for adolescents with chronic health conditions in 29 teams. More positive team climate, measured by the Team Climate Inventory (i.e., shared vision, participative safety, task orientation, support for innovation), at study start was associated with greater improvements in quality of chronic care delivery one year later [ 45 ]. Additionally, improvements in team climate were associated with greater improvement in care delivery [ 45 ]. These findings were consistent across teams working with different patient populations, suggesting the influence of team climate generalizes across teams and settings [ 45 ]. Greater team climate for innovation was also associated with greater fidelity (i.e., implementation of more program elements) among DBT teams [ 47 ]. In contrast, no significant associations were found between team climate and fidelity to a multifaceted cardiovascular disease management intervention, with qualitative data suggesting variation in the influence of teamwork across practices [ 77 ]. There was no overlap in the metrics of association reported in these studies; therefore, we were unable to generate an estimate of the effect of team functioning on fidelity. The quality of the evidence for fidelity was rated very low because of serious methodological limitations, serious inconsistency, and very serious imprecision due to the small number of studies.

Three studies examined associations between teamwork and adoption, with no significant associations found. The first study found that teamwork climate (i.e., perceived quality of collaboration between personnel) was not significantly associated with adoption of a comprehensive safety program in intensive care units, although there were associations between adoption and organizational constructs (e.g., lower safety climate, more management support) [ 59 ]. In a study of DBT teams, neither positive nor negative team functioning was associated with the number of DBT modes adopted [ 72 ]. The third study assessed relational coordination (i.e., shared goals, communication, respect) in primary care practices implementing patient engagement strategies. Relational coordination was high across practices initially and did not differ for practices with high vs. low adoption, although it increased over time in practices with high adoption [ 83 ]. There was no overlap in the metrics of association reported in these studies; therefore, we were unable to generate an estimate of the effect of team functioning on adoption. The quality of the evidence was rated very low because of serious methodological limitations and very serious imprecision due to the small number of studies.

Reach and sustainment were each examined in one quantitative study. DBT teams with more negative functioning had greater reach, suggesting that reach may reflect high workloads; positive functioning was not associated with reach [ 72 ]. In VA mental health clinics, team functioning was positively correlated with sustainment of evidence-based trauma-focused psychotherapies and significantly associated with sustainment after controlling for covariates [ 68 ]. Two studies examined other implementation outcomes. One found that better team functioning was associated with greater implementation of changes to improve access to care in VA clinics [ 62 ]. In the other, primary care practices reporting better teamwork were more likely to be in later stages of transformation to PCMHs than practices with poorer teamwork [ 88 ]. Because of the small number of studies examining reach, sustainment, and other implementation outcomes, we were unable to generate estimates of effects or ratings of evidence quality for these outcomes.

Our qualitative review findings are based on 12 studies describing how team functioning influenced implementation processes and outcomes. There was considerable variation across studies in how team functioning was defined and what implementation outcomes were examined. Most findings were based on relatively thin and superficial data. Studies occurred in a variety of healthcare settings with varying resources and implemented diverse interventions. We found with high confidence that 1) Adaptive team functioning, characterized by positive affective states (e.g., trust, mutual respect, belonging), effective behavior processes (e.g., frequent communication and coordination), and shared cognitive states (e.g., clear roles, shared mental models of how to provide care), facilitates implementation and is associated with better implementation outcomes; and 2) Problems in team functioning, including negative affective states (e.g., tension, lack of trust), problematic behavioral processes (e.g., conflict, competition, poor communication), and a lack of shared cognitive states (e.g., unclear roles, lack of shared awareness, competing goals), act as barriers to implementation and are associated with poor implementation outcomes.

Affective states

Specific affective states were examined in one quantitative study, three mixed methods studies, and one qualitative study. There was no overlap in the associations between affective states and implementation outcomes reported in quantitative studies (Table  3 ). In a study of multidisciplinary teams responding to child abuse, affective integration (i.e., liking, trust, respect) was positively associated with acceptability, appropriateness, and feasibility but not time to adoption or reach [ 65 ]. In DBT teams, cohesion was associated with fidelity, and qualitative data indicated that liking one’s team members and having a shared team identity were critical to effective implementation [ 47 ]. Another study of DBT teams described conflicts and lack of safety and trust within teams resulting in their dissolution [ 78 ].

Edmondson and colleagues found that psychological safety and ease of speaking up (i.e., interpersonal climate that allows members to share questions and concerns) were associated with implementation success [ 48 , 49 ]. In teams with low psychological safety, lower-status team members were hesitant to speak up, hindering change and proficiency in the new practice [ 49 ]. Psychological safety was closely related to learning behavior within the team, including speaking up with questions and concerns [ 48 , 49 ]. From the mixed methods and qualitative studies, we found that trust, cohesion, and psychological safety within teams facilitate implementation by contributing to team members’ willingness to speak up and share experiences and feedback. Negative affective states, fear of judgment, conflict, and lack of safety hinder implementation (moderate confidence).

Behavioral processes

Specific behavioral processes, including communication, learning behavior, and coordination, were examined in two quantitative studies, two mixed methods studies, and five qualitative studies. There was no overlap in the associations between behavioral processes and implementation outcomes reported in quantitative studies (Table  3 ).

Only one study reported quantitative findings for communication. Communication in DBT teams was positively associated with fidelity [ 47 ]. Qualitative studies frequently identified communication as a determinant of implementation (Table  4 ). From qualitative studies, we found that open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier (high confidence).

Quantitative associations between team learning behavior and implementation outcomes were reported in three studies. Team learning behavior in child abuse teams was positively correlated with acceptability and feasibility; it was not associated with appropriateness, time to adoption, or reach [ 65 ]. Learning behavior was positively associated with knowledge and use of innovations in nursing teams [ 91 ] and with implementation success in surgical teams [ 48 ]. Because each of these studies examined different implementation outcomes, we were unable to generate an estimate of the effect of learning behavior or rate evidence quality.

Inter-team communication, specifically speaking up and learning from other teams (i.e., boundary spanning), was identified as a critical part of team learning processes associated with successful implementation [ 48 ]. Communication beyond the team was also identified as a facilitator of implementation in two qualitative studies [ 47 , 75 ]. We found that communication beyond the team facilitates implementation by providing opportunities for team learning (low confidence).

Lastly, two qualitative studies examined coordination among healthcare teams [ 40 , 95 ]. Findings were somewhat ambiguous and based on thin data. We found with low confidence that poor coordination among healthcare professionals interferes with providing high-quality care and can be a barrier to implementation of new approaches (low confidence).

Cognitive states

Specific cognitive states were examined in two quantitative studies. There was no overlap in the associations between cognitive states and implementation outcomes reported (Table  3 ). The first study found no significant associations between shared goals and implementation outcomes [ 65 ]. The second study found that greater team knowledge and skills were associated with greater implementation of key changes to improve access to care; team problem recognition was not associated with implementation [ 62 ].

Two studies reported qualitative findings related to shared goals. In VA mental health teams, shared mission differentiated teams with sustained high reach of EBPs from those with low reach [ 84 ]. Commitment to a shared goal consistent with the EBP supported sustainment [ 84 ]. Similarly, shared goals and vision were identified as a facilitator of DBT programs [ 47 ]. We found that shared goals, mission, and vision within teams facilitate implementation and sustainment (low confidence).

Summary of team processes/states & implementation outcomes

There was very little overlap in the reported associations between team processes/states and implementation outcomes (Table  3 ). We were unable to generate estimates of effects for any associations. When there was sufficient overlap to rate evidence quality, the evidence was rated very low quality (Table A2, Additional File 2).

Qualitative review findings are shown in Table  4 . We found the following: 1) Adaptive team functioning, characterized by positive affective states (e.g., trust, mutual respect, belonging), effective behavior processes (e.g., frequent communication and coordination), and shared cognitive states (e.g., clear roles, shared mental models of how to provide care), facilitates implementation and is associated with better implementation outcomes (high confidence); 2) Problems in team functioning, including negative affective states (e.g., tension, lack of trust), problematic behavioral processes (e.g., conflict, competition, poor communication), and a lack of shared cognitive states (e.g., unclear roles, lack of shared awareness, competing goals), act as barriers to implementation and are associated with poor implementation outcomes (high confidence); 3) Trust, cohesion, and psychological safety within teams facilitate implementation by contributing to team members’ willingness to speak up and openly share experiences and feedback. Negative affective states, fear of judgment, conflict, and lack of safety hinder implementation (moderate confidence); 4) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier (high confidence); 5) Communication beyond the team facilitates implementation by providing opportunities for team learning (low confidence); 6) Poor coordination among healthcare professionals interferes with providing high-quality care and can be a barrier to implementation of new approaches (low confidence); and 7) Shared goals, mission, and vision within teams facilitate implementation and sustainment (low confidence).

Synthesis: team effectiveness outcomes & implementation outcomes

Team effectiveness outcomes are multidimensional and include performance (i.e., productivity, efficiency, and quality of the team’s work), team viability, and the impact of the team on members’ development [ 12 , 17 , 18 , 19 ]. Only two studies examined associations between team effectiveness and implementation outcomes. Quantitative findings are presented in Table  5 . One quantitative study found that team members’ ratings of team performance were associated with innovation acceptability, appropriateness, and feasibility; performance was not associated with time to adoption or reach [ 65 ]. One qualitative study found that positive outcomes for team members (e.g., increased comfort working together, greater knowledge) were associated with sustainment [ 70 ]. No studies examined associations of team viability and implementation outcomes.

Summary of team effectiveness outcomes & implementation outcomes

Only one quantitative study examined associations between a dimension of team effectiveness and implementation outcomes (Table  5 ). Accordingly, we were unable to generate ratings of evidence quality or estimates of any effects. Similarly, because there was only one qualitative study, we were unable to make a review finding.

This systematic review summarizes over 20 years of empirical literature on the associations between teamwork and implementation outcomes in the context of implementation of new practices in health and human services. Consistent with increased attention to teams and reliance on team-based models of care, as well as the growth of implementation science, studies increased substantially over time. We included studies that used quantitative, qualitative, or mixed methods, yielding a total of 58 articles representing 55 studies. Included studies spanned naturalistic implementation evaluations and planned implementation research.

Key findings with high confidence were: 1) Staffing shortages and turnover hinder implementation, 2) Adaptive team functioning, characterized by positive affective states, effective behavior processes, and shared cognitive states, facilitates implementation and is associated with better implementation outcomes. Problems in team functioning, including negative affective states, problematic behavioral processes, and a lack of shared cognitive states, act as barriers to implementation and are associated with poor implementation outcomes, and 3) Open, ongoing, and effective communication within teams facilitates implementation of new practices; poor communication is a barrier. Our results generally align with conventional wisdom and scientific research outside of healthcare, increasing confidence in the findings. Team effectiveness and change management research in other types of organizations and settings (e.g., military, aviation, space exploration) [ 98 , 99 , 100 , 101 , 102 , 103 ] is largely converging.

Overall, the literature was heterogeneous, and many studies lacked specificity regarding team composition and implementation activities and outcomes. Teamwork was defined and measured inconsistently and with limited precision across studies, which hindered our ability to draw conclusions about how teams influence implementation processes and outcomes. There was also poor measurement and reporting of implementation outcomes in many studies, consistent with a recent review of research on implementation outcomes [ 28 , 29 ]. Many studies used broad measures encompassing multiple dimensions of teamwork. Among studies that assessed specific team processes and states, there was very little overlap across constructs assessed. Qualitative studies identified a rich array of specific team processes and states; research to confirm the presence of these factors in other settings and determine their associations with implementation outcomes is needed.

In Table  6 , we summarize the limitations of existing research on teams and implementation and provide recommendations for future research. Notably, increased specificity and rigor in how teamwork is conceptualized and assessed is needed to advance our understanding of how teamwork affects implementation processes and outcomes. Limited inclusion of teams and team constructs in implementation theories, models, and frameworks has likely contributed to the neglect of teams in implementation science [ 20 ]. Updates to theories, models, and framework should consider integrating teams and team-level constructs [ 20 ]. In addition, there are well-established theories of team effectiveness that could inform hypotheses about how specific team constructs affect implementation [ 104 , 105 , 106 , 107 ].

There is considerable room for improvement in the definition and description of teams and analysis of data from teams. Describing the structure and purpose of teams, as well as interdependencies within the team, can help differentiate teams from groups of individuals who do not constitute a team, an important conceptual distinction that can be difficult to discern in study descriptions. Reporting of sampling and recruitment procedures for teams and team-level response rates is needed. For quantitative studies, use of standardized, validated measures of teamwork constructs is recommended. Researchers should be careful to base inferences about teams on team-level data. Lastly, future research should follow recommendations for improving measurement and reporting of implementation outcomes [ 29 , 108 ] and consider the multilevel context of teams in theory, measurement, analysis, and interpretation of results [ 109 ].

Limitations

As with all systematic reviews, it is possible that we failed to identify some relevant articles or data. We did not search gray literature or conference abstracts or contact authors for unreported data. Our organization of studies by the IMOI framework is likely imperfect given the broad array of team constructs included and poor reporting in many studies. We included diverse innovations intended to improve patient care, including specific EBPs, clinical practice guidelines, models of care, care bundles, procedural changes, and technological innovations. This diversity in objects of implementation reflects ongoing debates about the necessary strength of evidence for objects of implementation and varying thresholds in different contexts [ 110 ]. In this review, high quality studies tended to involve clinical interventions with strong research evidence (e.g., DBT) and clinics in structured and often team-based healthcare systems (e.g., VA). Diversity of innovations and settings provides greater external validity for our findings but may mask some findings specific to certain innovations or settings.

We only included studies of existing teams providing clinical services, however, many studies provided limited descriptions of teams, and in some cases the distinction between clinical teams and implementation/quality improvement teams was unclear. There is increasing attention to use of teams in implementation frameworks [ 20 , 111 ] and evidence that functioning of implementation teams matters [ 112 , 113 ]. Research on the composition and functioning of implementation teams is an important area for future research.

Our systematic review findings indicate that teamwork matters for implementation. However, greater specificity and rigor are needed to advance our understanding of how teamwork influences implementation processes and outcomes. We provide recommendations for improving the conceptualization, description, assessment, analysis, and interpretation of research on teams implementing new practices.

Availability of data and materials

All data cited in this review came from published papers and are therefore already available. The data created as part of the review process are included in this published article and its supplementary information files.

Mitchell P, Wynia M, Golden R, McNellis B, Okun S, Webb CE, et al. Core principles and values of effective team-based health care. NAM Perspectives. 2012. Available from: https://nam.edu/perspectives-2012-core-principles-values-of-effective-team-based-health-care/  Cited 2020 Sep 23

Reiss-Brennan B, Brunisholz KD, Dredge C, Briot P, Grazier K, Wilcox A, et al. Association of integrated team-based care with health care quality, utilization, and cost. JAMA. 2016;316(8):826–34.

Article   PubMed   Google Scholar  

Katkin JP, Kressly SJ, Edwards AR, Perrin JM, Kraft CA, Richerson JE, et al. Guiding principles for team-based pediatric care. Pediatrics. 2017;140(2):e20171489.

Lipton HL. Home is where the health is: Advancing team-based care in chronic disease management. Arch Intern Med. 2009;169(21):1945–8.

Dinh JV, Traylor AM, Kilcullen MP, Perez JA, Schweissing EJ, Venkatesh A, et al. Cross-disciplinary care: A systematic review on teamwork processes in health care. Small Group Research. 2020;51(1):125–66.

Article   Google Scholar  

Fiscella K, McDaniel SH. The complexity, diversity, and science of primary care teams. Am Psychol. 2018;73(4):451–67.

Herbert JL, Bromfield LM. A quasi-experimental study of the Multi-Agency Investigation & Support Team (MIST): A collaborative response to child sexual abuse. Child Abuse Negl. 2020;27:104827.

Google Scholar  

Reiss-Brennan B. Mental Health Integration: Normalizing Team Care. J Prim Care Community Health. 2014;5(1):55–60.

Schmutz J, Manser T. Do team processes really have an effect on clinical performance? A systematic literature review. Br J Anaesth. 2013;110(4):529–44.

Article   CAS   PubMed   Google Scholar  

Ilgen DR, Hollenbeck JR, Johnson M, Jundt D. Teams in organizations: From input-process-output models to IMOI models. Annu Rev Psychol. 2005;56:517–43.

Kozlowski SWJ, Bell BS. Work groups and teams in organization. In: Borman WC, Ilgen DR, Klimoski RJ, editors. Handbook of Psychology (Vol 12): Industrial and Organizational Psychology. New York: Wiley-Blackwell. 2003. p. 333–75. Available from: http://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/9781118133880.hop212017   Cited 2020 Apr 10

Mathieu JE, Maynard MT, Rapp T, Gilson L. Team effectiveness 1997–2007: A review of recent advancements and a glimpse into the future. J Manag. 2008;34(3):410–76.

Cordery JL, Tian AW. Team Design. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes. John Wiley & Sons, Ltd. 2017;p. 103–28. Available from: http://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/9781118909997.ch5  Cited 2021 Apr 2 

Kozlowski SWJ, Bell BS. Work groups and teams in organizations: Review update. In: Schmitt N, Highhouse S, editors. Handbook of Psychology (Vol 12): Industrial and Organizational Psychology. 2nd ed. Hoboken, NJ: Wiley; 2013. p. 111.

Kozlowski SWJ, Ilgen DR. Enhancing the effectiveness of work groups and teams. Psychol Sci Public Interest. 2006;7(3):77–124.

Marks MA, Mathieu JE, Zaccaro SJ. A temporally based framework and taxonomy of team processes. AMR. 2001;26(3):356–76.

Hackman JR. A normative model of work team effectiveness. Arlington, VA: Office of Naval Research; 1983. Available from: https://apps.dtic.mil/sti/citations/ADA136398  Cited 2022 May 16

Mathieu JE, Hollenbeck JR, van Knippenberg D, Ilgen DR. A century of work teams in the Journal of Applied Psychology. J Appl Psychol. 2017;102(3):452–67.

Rosen MA, Dietz AS. Team performance measurement. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes. John Wiley & Sons, Ltd; 2017. p. 479–502.

Chapter   Google Scholar  

McGuier EA, Kolko DJ, Stadnick NA, Brookman-Frazee L, Wolk CB, Yuan CT, et al. Advancing research on teams and team effectiveness in implementation science: An application of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Research and Practice. 2023;1(4):26334895231190856.

Williams NJ, Beidas RS. Annual research review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. J Child Psychol Psychiatry. 2019;60:430–50.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.

Article   PubMed   PubMed Central   Google Scholar  

McGuier EA, Kolko DJ, Klem ML, Feldman J, Kinkler G, Diabes MA, et al. Team functioning and implementation of innovations in healthcare and human service settings: a systematic review protocol. Syst Rev. 2021;10(1):189.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339. Available from: http://www.bmj.com/content/339/bmj.b2535  Cited 2020 Apr 1

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;29(372):n71.

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–3.

Lengnick-Hall R, Gerke DR, Proctor EK, Bunger AC, Phillips RJ, Martin JK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17(1):16.

Proctor EK, Bunger AC, Lengnick-Hall R, Gerke DR, Martin JK, Phillips RJ, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18(1):31.

Chapman A, Rankin NM, Jongebloed H, Yoong SL, White V, Livingston PM, et al. Overcoming challenges in conducting systematic reviews in implementation science: a methods commentary. Syst Rev. 2023;12(1):116.

Pranckutė R. Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World. Publications. 2021;9(1):12.

Effective Practice and Organisation of Care (EPOC). Data collection form. EPOC Resources for review authors. Norwegian Knowledge Centre for the Health Services. 2013. Available from:  http://epoc.cochrane.org/epoc-specific-resources-review-authors .

Noyes J, Lewin S. Extracting qualitative evidence. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, et al., editors. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group; 2011. Available from: http://cqrmg.cochrane.org/supplemental-handbook-guidance

Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. J Clin Epidemiol. 2019;1(111):49-59.e1.

Powell BJ, Patel SV, Haley AD, Haines ER, Knocke KE, Chandler S, Determinants of implementing evidence-based trauma-focused interventions for children and youth: A systematic review. Adm Policy Ment Health, et al. cited 2020 Feb 20. Available from: 2019. https://doi.org/10.1007/s10488-019-01003-3 .

Article   PubMed Central   Google Scholar  

Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;16(368):l6890.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology. 2011;64(4):383–94.

Lewin S, Bohren M, Rashidian A, Munthe-Kaas H, Glenton C, Colvin CJ, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implementation Sci. 2018;13(1):11–23.

Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Sci. 2018;13(1):1–10.

Adjognon OL, Shin MH, Steffen MJA, Moye J, Solimeo S, Sullivan JL. Factors affecting primary care implementation for older veterans with multimorbidity in Veterans Health Administration (VA). Health Serv Res. 2021;56(S1):1057–68.

Alidina S, Chatterjee P, Zanial N, Alreja SS, Balira R, Barash D, et al. Improving surgical quality in low-income and middle-income countries: why do some health facilities perform better than others? BMJ Qual Saf. 2021;30(12):937–49.

Boltey EM, Iwashyna TJ, Hyzy RC, Watson SR, Ross C, Costa DK. Ability to predict team members’ behaviors in ICU teams is associated with routine ABCDE implementation. J Crit Care. 2019;51:192–7.

Chenoweth L, Jeon YH, Stein-Parbury J, Forbes I, Fleming R, Cook J, et al. PerCEN trial participant perspectives on the implementation and outcomes of person-centered dementia care and environments. Int Psychogeriatr. 2015;27(12):2045–57.

Costa DK, Valley TS, Miller MA, Manojlovich M, Watson SR, McLellan P, et al. ICU team composition and its association with ABCDE implementation in a quality collaborative. J Crit Care. 2018;44:1–6.

Cramm JM, Strating MMH, Nieboer AP. The role of team climate in improving the quality of chronic care delivery: a longitudinal study among professionals working with chronically ill adolescents in transitional care programmes. BMJ Open. 2014;4(5):e005369.

DePuccio MJ, Gaughan AA, Sova LN, MacEwan SR, Walker DM, Gregory ME, et al. An examination of the barriers to and facilitators of implementing nurse-driven protocols to remove indwelling urinary catheters in acute care hospitals. The Joint Commission Journal on Quality and Patient Safety. 2020;46(12):691–8.

Ditty MS, Landes SJ, Doyle A, Beidas RS. It takes a village: A mixed method analysis of inner setting variables and Dialectical Behavior Therapy implementation. Adm Policy Ment Health. 2015;42(6):672–81.

Edmondson AC. Speaking up in the operating room: How team leaders promote learning in interdisciplinary action teams. J Manage Stud. 2003;40(6):1419–52.

Edmondson AC, Bohmer RM, Pisano GP. Disrupted routines: Team learning and new technology implementation in hospitals. Adm Sci Q. 2001;46(4):685–716.

Gillespie BM, Chaboyer W, Wallis M, Fenwick C. Why isn’t “time out” being implemented? An exploratory study. Qual Saf Health Care. 2010;19(2):103–6.

Gillespie BM, Harbeck E, Lavin J, Gardiner T, Withers TK, Marshall AP. Using normalisation process theory to evaluate the implementation of a complex intervention to embed the surgical safety checklist. BMC Health Serv Res. 2018;18(1):170.

Gillespie BM, Marshall AP, Gardiner T, Lavin J, Withers TK. Impact of workflow on the use of the Surgical Safety Checklist: a qualitative study: Surgical Safety Checklist and work flow. ANZ J Surg. 2016;86(11):864–7.

Gillespie BM, Withers TK, Lavin J, Gardiner T, Marshall AP. Factors that drive team participation in surgical safety checks: a prospective study. Patient Saf Surg. 2016;10(3):1–9.

Gosling AS. Clinical team functioning and IT innovation: A study of the diffusion of a point-of-care online evidence system. J Am Med Inform Assoc. 2003;10(3):244–51.

Graetz I, Huang J, Brand R, Shortell SM, Rundall TG, Bellows J, et al. The impact of electronic health records and teamwork on diabetes care quality. American Journal of Managed Care. 2015;21(12):878–84.

PubMed   Google Scholar  

Haider S, Ali RF, Ahmed M, Humayon AA, Sajjad M, Ahmad J. Barriers to implementation of emergency obstetric and neonatal care in rural Pakistan. Sawyer A, editor. PLoS ONE. 2019;14(11):e0224161.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Harvey C, Killaspy H, Martino S, White S, Priebe S, Wright C, et al. A comparison of the implementation of Assertive Community Treatment in Melbourne, Australia and London. England Epidemiol Psychiatr Sci. 2011;20(2):151–61.

Harvey C, Killaspy H, Martino S, Johnson S. Implementation of Assertive Community Treatment in Australia: Model fidelity, patient characteristics and staff experiences. Community Ment Health J. 2012;48(5):652–61.

Hsu YJ, Marsteller JA. Who applies an intervention to influence cultural attributes in a quality improvement collaborative? J Patient Saf. 2020;16(1):1–6.

Irimu GW, Greene A, Gathara D, Kihara H, Maina C, Mbori-Ngacha D, et al. Explaining the uptake of paediatric guidelines in a Kenyan tertiary hospital – mixed methods research. BMC Health Serv Res. 2014;14(1):119.

Lall D, Engel N, Devadasan N, Horstman K, Criel B. Team-based primary health care for non-communicable diseases: complexities in South India. Health Policy Plan. 2020;35(Supplement_2):ii22-34.

Lukas CV, Mohr D, Meterko M. Team effectiveness and organizational context in the implementation of a clinical innovation. Qual Manag Health Care. 2009;18(1):25–39.

Mariani E, Vernooij-Dassen M, Koopmans R, Engels Y, Chattat R. Shared decision-making in dementia care planning: barriers and facilitators in two European countries. Aging Ment Health. 2017;21(1):31–9.

Martinez RN, Hogan TP, Balbale S, Lones K, Goldstein B, Woo C, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemedicine and e-Health. 2017;23(7):567–76.

McGuier EA, Aarons GA, Byrne KA, Campbell KA, Keeshin B, Rothenberger SD, et al. Associations between teamwork and implementation outcomes in multidisciplinary cross-sector teams implementing a mental health screening and referral protocol. Implementation Science Communications. 2023;4(1):13.

Meloncelli N, Barnett A, de Jersey S. Staff resourcing, guideline implementation and models of care for gestational diabetes mellitus management. Aust N Z J Obstet Gynaecol. 2020;60(1):115–22.

Miller CJ, Kim B, Connolly SL, Spitzer EG, Brown M, Bailey HM, et al. Sustainability of the Collaborative Chronic Care Model in outpatient mental health teams three years post-Implementation: A qualitative analysis. Adm Policy Ment Health. 2023;50(1):151–9.

Mohr DC, Rosen CS, Schnurr PP, Orazem RJ, Noorbaloochi S, Clothier BA, et al. The influence of team functioning and workload on sustainability of trauma-focused evidence-based psychotherapies. PS. 2018;69(8):879–86.

Morgan D, Kosteniuk J, O’Connell ME, Kirk A, Stewart NJ, Seitz D, et al. Barriers and facilitators to development and implementation of a rural primary health care intervention for dementia: a process evaluation. BMC Health Serv Res. 2019;19(1):709.

Morgan D, Kosteniuk J, O’Connell ME, Seitz D, Elliot V, Bayly M, et al. Factors influencing sustainability and scale-up of rural primary healthcare memory clinics: perspectives of clinic team members. BMC Health Serv Res. 2022;22(1):148.

Munce SEP, Graham ID, Salbach NM, Jaglal SB, Richards CL, Eng JJ, et al. Perspectives of health care professionals on the facilitators and barriers to the implementation of a stroke rehabilitation guidelines cluster randomized controlled trial. BMC Health Serv Res. 2017;17(1):440.

Navarro-Haro MV, Harned MS, Korslund KE, DuBose A, Chen T, Ivanoff A, et al. Predictors of adoption and reach following Dialectical Behavior Therapy Intensive Training™. Community Ment Health J. 2019;55(1):100–11.

Newbould L, Ariss S, Mountain G, Hawley MS. Exploring factors that affect the uptake and sustainability of videoconferencing for healthcare provision for older adults in care homes: a realist evaluation. BMC Med Inform Decis Mak. 2021;21(1):13.

Nyondo-Mipando AL, Woo Kinshella ML, Bohne C, Suwedi-Kapesa LC, Salimu S, Banda M, et al. Barriers and enablers of implementing bubble Continuous Positive Airway Pressure (CPAP): Perspectives of health professionals in Malawi. Ameh CA, editor. PLoS ONE. 2020;15(2):e0228915.

Nzinga J, Mbindyo P, Mbaabu L, Warira A, English M. Documenting the experiences of health workers expected to implement guidelines during an intervention study in Kenyan hospitals. Implementation Sci. 2009;4(1):44.

Papoutsi C, Poots A, Clements J, Wyrko Z, Offord N, Reed JE. Improving patient safety for older people in acute admissions: implementation of the Frailsafe checklist in 12 hospitals across the UK. Age Ageing. 2018;47(2):311–7.

Patel B, Usherwood T, Harris M, Patel A, Panaretto K, Zwar N, et al. What drives adoption of a computerised, multifaceted quality improvement intervention for cardiovascular disease management in primary healthcare settings? A mixed methods analysis using normalisation process theory. Implementation Sci. 2018;13(1):140.

Popowich AD, Mushquash AR, Pearson E, Schmidt F, Mushquash CJ. Barriers and facilitators affecting the sustainability of dialectical behaviour therapy programmes: A qualitative study of clinician perspectives. Couns Psychother Res. 2020;20(1):68–80.

Prom MC, Canelos V, Fernandez PJ, Gergen Barnett K, Gordon CM, Pace CA, et al. Implementation of integrated behavioral health care in a large medical center: Benefits, challenges, and recommendations. J Behav Health Serv Res. 2021;48(3):346–62.

Provost S, Pineault R, Grimard D, Pérez J, Fournier M, Lévesque Y, et al. Implementation of an integrated primary care cardiometabolic risk prevention and management network in Montréal: does greater coordination of care with primary care physicians have an impact on health outcomes? Health Promot Chronic Dis Prev Can. 2017;37(4):105–13.

Randell R, Honey S, Alvarado N, Greenhalgh J, Hindmarsh J, Pearman A, et al. Factors supporting and constraining the implementation of robot-assisted surgery: a realist interview study. BMJ Open. 2019;9(6):e028635.

Robbins J, McAlearney AS. Encouraging employees to speak up to prevent infections: Opportunities to leverage quality improvement and care management processes. Am J Infect Control. 2016;44(11):1224–30.

Rodriguez HP, Poon BY, Wang E, Shortell SM. Linking practice adoption of patient engagement strategies and relational coordination to patient-reported outcomes in accountable care organizations. Milbank Q. 2019;97(3):692–735.

Sayer NA, Rosen CS, Bernardy NC, Cook JM, Orazem RJ, Chard KM, et al. Context matters: Team and organizational factors associated with reach of evidence-based psychotherapies for PTSD in the Veterans Health Administration. Adm Policy Ment Health. 2017;44(6):904–18.

Shin MH, Montano ARL, Adjognon OL, Harvey KLL, Solimeo SL, Sullivan JL. Identification of implementation strategies using the CFIR-ERIC matching tool to mitigate barriers in a primary care model for older veterans. Gerontologist. 2022;63(3):439–50.

Sopcak N, Aguilar C, O’Brien MA, Nykiforuk C, Aubrey-Bassler K, Cullen R, et al. Implementation of the BETTER 2 program: a qualitative study exploring barriers and facilitators of a novel way to improve chronic disease prevention and screening in primary care. Implementation Sci. 2016;11(1):158.

Steinmo SH, Michie S, Fuller C, Stanley S, Stapleton C, Stone SP. Bridging the gap between pragmatic intervention design and theory: using behavioural science tools to modify an existing quality improvement programme to implement “Sepsis Six.” Implementation Sci. 2016;11(1):14.

Stout S, Zallman L, Arsenault L, Sayah A, Hacker K. Developing high-functioning teams: Factors associated with operating as a “real team” and implications for patient-centered medical home development. Inquiry. 2017;12(54):0046958017707296.

Svensson B, Hansson L, Markström U, Lexén A. What matters when implementing Flexible Assertive Community Treatment in a Swedish healthcare context: A two-year implementation study. Int J Ment Health. 2017;46(4):284–98.

Tadyanemhandu C, van Aswegen H, Ntsiea V. Barriers and facilitators to implementation of early mobilisation of critically ill patients in Zimbabwean and South African public sector hospitals: a qualitative study. Disabil Rehabil. 2021;44(22):6699–709.

Timmermans O, Van Linge R, Van Petegem P, Van Rompaey B, Denekens J. A contingency perspective on team learning and innovation in nursing: Team learning and innovation in nursing teams. J Adv Nurs. 2013;69(2):363–73.

Tuepker A, Kansagara D, Skaperdas E, Nicolaidis C, Joos S, Alperin M, et al. “We’ve not gotten even close to what we want to do”: A qualitative study of early patient-centered medical home implementation. J GEN INTERN MED. 2014;29(S2):614–22.

van Bruggen S, Kasteleyn MJ, Rauh SP, Meijer JS, Busch KJG, Numans ME, et al. Experiences with tailoring of primary diabetes care in well-organised general practices: a mixed-methods study. BMC Health Serv Res. 2021;21(1):1218.

Van der Wees PJ, Friedberg MW, Guzman EA, Ayanian JZ, Rodriguez HP. Comparing the implementation of team approaches for improving diabetes care in community health centers. BMC Health Serv Res. 2014;14(1):608.

von Meyenfeldt EM, van Nassau F, de Betue CTI, Barberio L, Schreurs WH, Marres GMH, et al. Implementing an enhanced recovery after thoracic surgery programme in the Netherlands: a qualitative study investigating facilitators and barriers for implementation. BMJ Open. 2022;12(1):e051513.

Wakefield BJ, Lampman MA, Paez MB, Stewart GL. Care management and care coordination within a patient-centered medical home. J Nurs Admin. 2020;50(11):565–70.

Wiener-Ogilvie S, Huby G, Pinnock H, Gillies J, Sheikh A. Practice organisational characteristics can impact on compliance with the BTS/SIGN asthma guideline: Qualitative comparative case study in primary care. BMC Fam Pract. 2008;9(1):32.

Errida A, Lotfi B. The determinants of organizational change management success: Literature review and case study. International Journal of Engineering Business Management. 2021;1(13):18479790211016270.

Phillips J, Klein JD. Change management: From theory to practice. TechTrends. 2023;67(1):189–97.

Rousseau DM, ten Have S. Evidence-based change management. Organ Dyn. 2022;51(3):100899.

Goodwin GF, Blacksmith N, Coats MR. The science of teams in the military: Contributions from over 60 years of research. Am Psychol. 2018;73(4):322–33.

DiazGranados D, Woods JJ, Wilder A, Curtis MT. Team dynamics in the air: A review of team research relevant to aviation. Human factors in aviation and aerospace. 2023. p. 199–215.

Salas E, Tannenbaum SI, Kozlowski SWJ, Miller CA, Mathieu JE, Vessey WB. Teams in space exploration: A new frontier for the science of team effectiveness. Curr Dir Psychol Sci. 2015;24(3):200–7.

Hackman JR. From causes to conditions in group research. J Organ Behav. 2012;33(3):428–44.

Tannenbaum SI, Salas E. Teams that work: The seven drivers of team effectiveness. Illustrated. New York, NY: Oxford University Press; 2020. p. 272.

Book   Google Scholar  

Tuckman BW. Developmental sequence in small groups. Psychol Bull. 1965;63:384–99.

Salas E, Stagl KC, Burke CS. 25 years of team effectiveness in organizations: Research themes and emerging needs. In: International Review of Industrial and Organizational Psychology 2004. John Wiley & Sons, Ltd. 2004; p. 47–91. Available from: https://onlinelibrary.wiley.com/doi/abs/ https://doi.org/10.1002/0470013311.ch2    Cited 2022 May 16

Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Lengnick-Hall R, Williams NJ, Ehrhart MG, Willging CE, Bunger AC, Beidas RS, et al. Eight characteristics of rigorous multilevel implementation research: a step-by-step guide. Implement Sci. 2023;18(1):52.

Wensing M, Sales A, Aarons GA, Xu D (Roman), Wilson P. Evidence for objects of implementation in healthcare: considerations for Implementation Science and Implementation Science Communications. Implementation Sci. 2022;17(1):83.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu SY, Mendel P, et al. The role of perceived team effectiveness in improving chronic illness care. Med Care. 2004;42(11):1040–8.

Yakovchenko V, Morgan TR, Chinman MJ, Powell BJ, Gonzalez R, Park A, et al. Mapping the road to elimination: a 5-year evaluation of implementation strategies associated with hepatitis C treatment in the veterans health administration. BMC Health Serv Res. 2021;21(1):1348.

Download references

Acknowledgements

Thank you to Ikzzui Chu, Jamie Feldman, Grace Kinkler, Rachael Park, and Jaely Wright for their assistance with article screening.

This work was supported by the National Institute of Mental Health grants MH123729 (EAM), MH124914 (DJK), and MH126231 (GAA), the National Cancer Institute U01CA275118 (GAA), National Institute on Drug Abuse R01DA049891 (GAA), the Agency for Healthcare Research and Quality grant R18HS026862 (CBW), and the Collaboration and Conflict Research Lab at Carnegie Mellon University Tepper School of Business. The content of this manuscript does not represent the views of funding agencies and is solely the responsibility of the authors.

Author information

Authors and affiliations.

Department of Psychiatry, School of Medicine, University of Pittsburgh, 3811 O’Hara Street, Pittsburgh, PA, 15213, USA

Elizabeth A. McGuier & David J. Kolko

Department of Psychiatry, University of California San Diego, La Jolla, CA, USA

Gregory A. Aarons

ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA

Child and Adolescent Services Research Center, San Diego, CA, USA

Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

Allison Schachter & Courtney Benjamin Wolk

Penn Implementation Science Center at the Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, USA

Health Sciences Library System, University of Pittsburgh, Pittsburgh, PA, USA

Mary Lou Klem

Tepper School of Business, Carnegie Mellon University, Pittsburgh, PA, USA

Matthew A. Diabes & Laurie R. Weingart

Department of Psychological Sciences, Rice University, Houston, TX, USA

Eduardo Salas

You can also search for this author in PubMed   Google Scholar

Contributions

EAM conceptualized and developed the protocol for this review, participated in article screening, data extraction, quality appraisal, and analysis, and led manuscript development and writing. DJK, GAA, LRW, ES, and CBW provided guidance and input into the protocol and methods and interpretation of results. AS participated in article screening, data extraction, and preparation of supplementary files. MLK developed the search strategy and conducted database searches. MAD participated in quality appraisal. DJK, GAA, LRW, ES, and CBW reviewed multiple drafts of the manuscript and provided critical input and editing. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Elizabeth A. McGuier .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

GAA is a Co-Editor-in-Chief of Implementation Science and on the Editorial Board of Implementation Science Communications ; all decisions on this paper were made by another editor. The authors declare that they have no other competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

McGuier, E.A., Kolko, D.J., Aarons, G.A. et al. Teamwork and implementation of innovations in healthcare and human service settings: a systematic review. Implementation Sci 19 , 49 (2024). https://doi.org/10.1186/s13012-024-01381-9

Download citation

Received : 18 January 2024

Accepted : 07 July 2024

Published : 15 July 2024

DOI : https://doi.org/10.1186/s13012-024-01381-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation outcomes

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

How to Equitably Assign High-Profile Work

• Rachel M. Korn,
• Asma Ghani,
• Joan C. Williams

Working with researchers, four companies used a data-driven approach to balance who was doing the office housework and who was getting promotion-making assignments.

The gender and racial disparity in access to high-profile, career-enhancing work has been widely overlooked. Research across industries has shown that 81% to 85% of white men report fair access to desirable assignments, while that number falls to 50% for women of color. At the same time, women tend to be burdened with more non-promotable “office housework,” such as scheduling meetings, cleaning up, or being the peacemaker. The authors ran bias training workshops and fair distribution strategies in four companies, leading to noticeable improvements in the equitable allocation of both office housework and high-profile assignments.

Lack of access to high-profile work — the kind that sets you up for promotion and leadership opportunities — is a key reason people leave their organization. The costs of employee attrition are well-known and expensive — up to twice an employee’s annual salary.

• Rachel M. Korn is the research director at Equality Action Center, University of California College of the Law, San Francisco.
• Asma Ghani is the lead research scientist at Equality Action Center, University of California College of the Law, San Francisco. She is a social psychologist with research expertise in intersectionality — examining how multiple identities and systems of oppression overlap to create multilayered inequity and how best to mitigate that inequity.
• Joan C. Williams is a Sullivan Professor of Law at University of California College of the Law, San Francisco and the founding director of Equality Action Center. An expert on social inequality, she is the author of 12 books, including Bias Interrupted: Creating Inclusion for Real and for Good (Harvard Business Review Press, 2021) and White Working Class: Overcoming Class Cluelessness in America (Harvard Business Review Press, 2019). To learn about her evidence-based, metrics-driven approach to eradicating implicit bias in the workplace, visit www.biasinterrupters.org .

Partner Center

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Korean Med Sci
• v.37(16); 2022 Apr 25

A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles

Edward barroga.

1 Department of General Education, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan.

Glafera Janet Matanguihan

2 Department of Biological Sciences, Messiah University, Mechanicsburg, PA, USA.

The development of research questions and the subsequent hypotheses are prerequisites to defining the main research purpose and specific objectives of a study. Consequently, these objectives determine the study design and research outcome. The development of research questions is a process based on knowledge of current trends, cutting-edge studies, and technological advances in the research field. Excellent research questions are focused and require a comprehensive literature search and in-depth understanding of the problem being investigated. Initially, research questions may be written as descriptive questions which could be developed into inferential questions. These questions must be specific and concise to provide a clear foundation for developing hypotheses. Hypotheses are more formal predictions about the research outcomes. These specify the possible results that may or may not be expected regarding the relationship between groups. Thus, research questions and hypotheses clarify the main purpose and specific objectives of the study, which in turn dictate the design of the study, its direction, and outcome. Studies developed from good research questions and hypotheses will have trustworthy outcomes with wide-ranging social and health implications.

INTRODUCTION

Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses. 1 , 2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results. 3 , 4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the inception of novel studies and the ethical testing of ideas. 5 , 6

It is crucial to have knowledge of both quantitative and qualitative research 2 as both types of research involve writing research questions and hypotheses. 7 However, these crucial elements of research are sometimes overlooked; if not overlooked, then framed without the forethought and meticulous attention it needs. Planning and careful consideration are needed when developing quantitative or qualitative research, particularly when conceptualizing research questions and hypotheses. 4

There is a continuing need to support researchers in the creation of innovative research questions and hypotheses, as well as for journal articles that carefully review these elements. 1 When research questions and hypotheses are not carefully thought of, unethical studies and poor outcomes usually ensue. Carefully formulated research questions and hypotheses define well-founded objectives, which in turn determine the appropriate design, course, and outcome of the study. This article then aims to discuss in detail the various aspects of crafting research questions and hypotheses, with the goal of guiding researchers as they develop their own. Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points.

DEFINITIONS AND RELATIONSHIP OF RESEARCH QUESTIONS AND HYPOTHESES

A research question is what a study aims to answer after data analysis and interpretation. The answer is written in length in the discussion section of the paper. Thus, the research question gives a preview of the different parts and variables of the study meant to address the problem posed in the research question. 1 An excellent research question clarifies the research writing while facilitating understanding of the research topic, objective, scope, and limitations of the study. 5

On the other hand, a research hypothesis is an educated statement of an expected outcome. This statement is based on background research and current knowledge. 8 , 9 The research hypothesis makes a specific prediction about a new phenomenon 10 or a formal statement on the expected relationship between an independent variable and a dependent variable. 3 , 11 It provides a tentative answer to the research question to be tested or explored. 4

Hypotheses employ reasoning to predict a theory-based outcome. 10 These can also be developed from theories by focusing on components of theories that have not yet been observed. 10 The validity of hypotheses is often based on the testability of the prediction made in a reproducible experiment. 8

Conversely, hypotheses can also be rephrased as research questions. Several hypotheses based on existing theories and knowledge may be needed to answer a research question. Developing ethical research questions and hypotheses creates a research design that has logical relationships among variables. These relationships serve as a solid foundation for the conduct of the study. 4 , 11 Haphazardly constructed research questions can result in poorly formulated hypotheses and improper study designs, leading to unreliable results. Thus, the formulations of relevant research questions and verifiable hypotheses are crucial when beginning research. 12

CHARACTERISTICS OF GOOD RESEARCH QUESTIONS AND HYPOTHESES

Excellent research questions are specific and focused. These integrate collective data and observations to confirm or refute the subsequent hypotheses. Well-constructed hypotheses are based on previous reports and verify the research context. These are realistic, in-depth, sufficiently complex, and reproducible. More importantly, these hypotheses can be addressed and tested. 13

There are several characteristics of well-developed hypotheses. Good hypotheses are 1) empirically testable 7 , 10 , 11 , 13 ; 2) backed by preliminary evidence 9 ; 3) testable by ethical research 7 , 9 ; 4) based on original ideas 9 ; 5) have evidenced-based logical reasoning 10 ; and 6) can be predicted. 11 Good hypotheses can infer ethical and positive implications, indicating the presence of a relationship or effect relevant to the research theme. 7 , 11 These are initially developed from a general theory and branch into specific hypotheses by deductive reasoning. In the absence of a theory to base the hypotheses, inductive reasoning based on specific observations or findings form more general hypotheses. 10

TYPES OF RESEARCH QUESTIONS AND HYPOTHESES

Research questions and hypotheses are developed according to the type of research, which can be broadly classified into quantitative and qualitative research. We provide a summary of the types of research questions and hypotheses under quantitative and qualitative research categories in Table 1 .

Research questions in quantitative research

In quantitative research, research questions inquire about the relationships among variables being investigated and are usually framed at the start of the study. These are precise and typically linked to the subject population, dependent and independent variables, and research design. 1 Research questions may also attempt to describe the behavior of a population in relation to one or more variables, or describe the characteristics of variables to be measured ( descriptive research questions ). 1 , 5 , 14 These questions may also aim to discover differences between groups within the context of an outcome variable ( comparative research questions ), 1 , 5 , 14 or elucidate trends and interactions among variables ( relationship research questions ). 1 , 5 We provide examples of descriptive, comparative, and relationship research questions in quantitative research in Table 2 .

Hypotheses in quantitative research

In quantitative research, hypotheses predict the expected relationships among variables. 15 Relationships among variables that can be predicted include 1) between a single dependent variable and a single independent variable ( simple hypothesis ) or 2) between two or more independent and dependent variables ( complex hypothesis ). 4 , 11 Hypotheses may also specify the expected direction to be followed and imply an intellectual commitment to a particular outcome ( directional hypothesis ) 4 . On the other hand, hypotheses may not predict the exact direction and are used in the absence of a theory, or when findings contradict previous studies ( non-directional hypothesis ). 4 In addition, hypotheses can 1) define interdependency between variables ( associative hypothesis ), 4 2) propose an effect on the dependent variable from manipulation of the independent variable ( causal hypothesis ), 4 3) state a negative relationship between two variables ( null hypothesis ), 4 , 11 , 15 4) replace the working hypothesis if rejected ( alternative hypothesis ), 15 explain the relationship of phenomena to possibly generate a theory ( working hypothesis ), 11 5) involve quantifiable variables that can be tested statistically ( statistical hypothesis ), 11 6) or express a relationship whose interlinks can be verified logically ( logical hypothesis ). 11 We provide examples of simple, complex, directional, non-directional, associative, causal, null, alternative, working, statistical, and logical hypotheses in quantitative research, as well as the definition of quantitative hypothesis-testing research in Table 3 .

Research questions in qualitative research

Unlike research questions in quantitative research, research questions in qualitative research are usually continuously reviewed and reformulated. The central question and associated subquestions are stated more than the hypotheses. 15 The central question broadly explores a complex set of factors surrounding the central phenomenon, aiming to present the varied perspectives of participants. 15

There are varied goals for which qualitative research questions are developed. These questions can function in several ways, such as to 1) identify and describe existing conditions ( contextual research question s); 2) describe a phenomenon ( descriptive research questions ); 3) assess the effectiveness of existing methods, protocols, theories, or procedures ( evaluation research questions ); 4) examine a phenomenon or analyze the reasons or relationships between subjects or phenomena ( explanatory research questions ); or 5) focus on unknown aspects of a particular topic ( exploratory research questions ). 5 In addition, some qualitative research questions provide new ideas for the development of theories and actions ( generative research questions ) or advance specific ideologies of a position ( ideological research questions ). 1 Other qualitative research questions may build on a body of existing literature and become working guidelines ( ethnographic research questions ). Research questions may also be broadly stated without specific reference to the existing literature or a typology of questions ( phenomenological research questions ), may be directed towards generating a theory of some process ( grounded theory questions ), or may address a description of the case and the emerging themes ( qualitative case study questions ). 15 We provide examples of contextual, descriptive, evaluation, explanatory, exploratory, generative, ideological, ethnographic, phenomenological, grounded theory, and qualitative case study research questions in qualitative research in Table 4 , and the definition of qualitative hypothesis-generating research in Table 5 .

Qualitative studies usually pose at least one central research question and several subquestions starting with How or What . These research questions use exploratory verbs such as explore or describe . These also focus on one central phenomenon of interest, and may mention the participants and research site. 15

Hypotheses in qualitative research

Hypotheses in qualitative research are stated in the form of a clear statement concerning the problem to be investigated. Unlike in quantitative research where hypotheses are usually developed to be tested, qualitative research can lead to both hypothesis-testing and hypothesis-generating outcomes. 2 When studies require both quantitative and qualitative research questions, this suggests an integrative process between both research methods wherein a single mixed-methods research question can be developed. 1

FRAMEWORKS FOR DEVELOPING RESEARCH QUESTIONS AND HYPOTHESES

Research questions followed by hypotheses should be developed before the start of the study. 1 , 12 , 14 It is crucial to develop feasible research questions on a topic that is interesting to both the researcher and the scientific community. This can be achieved by a meticulous review of previous and current studies to establish a novel topic. Specific areas are subsequently focused on to generate ethical research questions. The relevance of the research questions is evaluated in terms of clarity of the resulting data, specificity of the methodology, objectivity of the outcome, depth of the research, and impact of the study. 1 , 5 These aspects constitute the FINER criteria (i.e., Feasible, Interesting, Novel, Ethical, and Relevant). 1 Clarity and effectiveness are achieved if research questions meet the FINER criteria. In addition to the FINER criteria, Ratan et al. described focus, complexity, novelty, feasibility, and measurability for evaluating the effectiveness of research questions. 14

The PICOT and PEO frameworks are also used when developing research questions. 1 The following elements are addressed in these frameworks, PICOT: P-population/patients/problem, I-intervention or indicator being studied, C-comparison group, O-outcome of interest, and T-timeframe of the study; PEO: P-population being studied, E-exposure to preexisting conditions, and O-outcome of interest. 1 Research questions are also considered good if these meet the “FINERMAPS” framework: Feasible, Interesting, Novel, Ethical, Relevant, Manageable, Appropriate, Potential value/publishable, and Systematic. 14

As we indicated earlier, research questions and hypotheses that are not carefully formulated result in unethical studies or poor outcomes. To illustrate this, we provide some examples of ambiguous research question and hypotheses that result in unclear and weak research objectives in quantitative research ( Table 6 ) 16 and qualitative research ( Table 7 ) 17 , and how to transform these ambiguous research question(s) and hypothesis(es) into clear and good statements.

a These statements were composed for comparison and illustrative purposes only.

b These statements are direct quotes from Higashihara and Horiuchi. 16

a This statement is a direct quote from Shimoda et al. 17

The other statements were composed for comparison and illustrative purposes only.

CONSTRUCTING RESEARCH QUESTIONS AND HYPOTHESES

To construct effective research questions and hypotheses, it is very important to 1) clarify the background and 2) identify the research problem at the outset of the research, within a specific timeframe. 9 Then, 3) review or conduct preliminary research to collect all available knowledge about the possible research questions by studying theories and previous studies. 18 Afterwards, 4) construct research questions to investigate the research problem. Identify variables to be accessed from the research questions 4 and make operational definitions of constructs from the research problem and questions. Thereafter, 5) construct specific deductive or inductive predictions in the form of hypotheses. 4 Finally, 6) state the study aims . This general flow for constructing effective research questions and hypotheses prior to conducting research is shown in Fig. 1 .

Research questions are used more frequently in qualitative research than objectives or hypotheses. 3 These questions seek to discover, understand, explore or describe experiences by asking “What” or “How.” The questions are open-ended to elicit a description rather than to relate variables or compare groups. The questions are continually reviewed, reformulated, and changed during the qualitative study. 3 Research questions are also used more frequently in survey projects than hypotheses in experiments in quantitative research to compare variables and their relationships.

Hypotheses are constructed based on the variables identified and as an if-then statement, following the template, ‘If a specific action is taken, then a certain outcome is expected.’ At this stage, some ideas regarding expectations from the research to be conducted must be drawn. 18 Then, the variables to be manipulated (independent) and influenced (dependent) are defined. 4 Thereafter, the hypothesis is stated and refined, and reproducible data tailored to the hypothesis are identified, collected, and analyzed. 4 The hypotheses must be testable and specific, 18 and should describe the variables and their relationships, the specific group being studied, and the predicted research outcome. 18 Hypotheses construction involves a testable proposition to be deduced from theory, and independent and dependent variables to be separated and measured separately. 3 Therefore, good hypotheses must be based on good research questions constructed at the start of a study or trial. 12

In summary, research questions are constructed after establishing the background of the study. Hypotheses are then developed based on the research questions. Thus, it is crucial to have excellent research questions to generate superior hypotheses. In turn, these would determine the research objectives and the design of the study, and ultimately, the outcome of the research. 12 Algorithms for building research questions and hypotheses are shown in Fig. 2 for quantitative research and in Fig. 3 for qualitative research.

EXAMPLES OF RESEARCH QUESTIONS FROM PUBLISHED ARTICLES

• EXAMPLE 1. Descriptive research question (quantitative research)
• - Presents research variables to be assessed (distinct phenotypes and subphenotypes)
• “BACKGROUND: Since COVID-19 was identified, its clinical and biological heterogeneity has been recognized. Identifying COVID-19 phenotypes might help guide basic, clinical, and translational research efforts.
• RESEARCH QUESTION: Does the clinical spectrum of patients with COVID-19 contain distinct phenotypes and subphenotypes? ” 19
• EXAMPLE 2. Relationship research question (quantitative research)
• - Shows interactions between dependent variable (static postural control) and independent variable (peripheral visual field loss)
• “Background: Integration of visual, vestibular, and proprioceptive sensations contributes to postural control. People with peripheral visual field loss have serious postural instability. However, the directional specificity of postural stability and sensory reweighting caused by gradual peripheral visual field loss remain unclear.
• Research question: What are the effects of peripheral visual field loss on static postural control ?” 20
• EXAMPLE 3. Comparative research question (quantitative research)
• - Clarifies the difference among groups with an outcome variable (patients enrolled in COMPERA with moderate PH or severe PH in COPD) and another group without the outcome variable (patients with idiopathic pulmonary arterial hypertension (IPAH))
• “BACKGROUND: Pulmonary hypertension (PH) in COPD is a poorly investigated clinical condition.
• RESEARCH QUESTION: Which factors determine the outcome of PH in COPD?
• STUDY DESIGN AND METHODS: We analyzed the characteristics and outcome of patients enrolled in the Comparative, Prospective Registry of Newly Initiated Therapies for Pulmonary Hypertension (COMPERA) with moderate or severe PH in COPD as defined during the 6th PH World Symposium who received medical therapy for PH and compared them with patients with idiopathic pulmonary arterial hypertension (IPAH) .” 21
• EXAMPLE 4. Exploratory research question (qualitative research)
• - Explores areas that have not been fully investigated (perspectives of families and children who receive care in clinic-based child obesity treatment) to have a deeper understanding of the research problem
• “Problem: Interventions for children with obesity lead to only modest improvements in BMI and long-term outcomes, and data are limited on the perspectives of families of children with obesity in clinic-based treatment. This scoping review seeks to answer the question: What is known about the perspectives of families and children who receive care in clinic-based child obesity treatment? This review aims to explore the scope of perspectives reported by families of children with obesity who have received individualized outpatient clinic-based obesity treatment.” 22
• EXAMPLE 5. Relationship research question (quantitative research)
• - Defines interactions between dependent variable (use of ankle strategies) and independent variable (changes in muscle tone)
• “Background: To maintain an upright standing posture against external disturbances, the human body mainly employs two types of postural control strategies: “ankle strategy” and “hip strategy.” While it has been reported that the magnitude of the disturbance alters the use of postural control strategies, it has not been elucidated how the level of muscle tone, one of the crucial parameters of bodily function, determines the use of each strategy. We have previously confirmed using forward dynamics simulations of human musculoskeletal models that an increased muscle tone promotes the use of ankle strategies. The objective of the present study was to experimentally evaluate a hypothesis: an increased muscle tone promotes the use of ankle strategies. Research question: Do changes in the muscle tone affect the use of ankle strategies ?” 23

EXAMPLES OF HYPOTHESES IN PUBLISHED ARTICLES

• EXAMPLE 1. Working hypothesis (quantitative research)
• - A hypothesis that is initially accepted for further research to produce a feasible theory
• “As fever may have benefit in shortening the duration of viral illness, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response when taken during the early stages of COVID-19 illness .” 24
• “In conclusion, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response . The difference in perceived safety of these agents in COVID-19 illness could be related to the more potent efficacy to reduce fever with ibuprofen compared to acetaminophen. Compelling data on the benefit of fever warrant further research and review to determine when to treat or withhold ibuprofen for early stage fever for COVID-19 and other related viral illnesses .” 24
• EXAMPLE 2. Exploratory hypothesis (qualitative research)
• - Explores particular areas deeper to clarify subjective experience and develop a formal hypothesis potentially testable in a future quantitative approach
• “We hypothesized that when thinking about a past experience of help-seeking, a self distancing prompt would cause increased help-seeking intentions and more favorable help-seeking outcome expectations .” 25
• “Conclusion
• Although a priori hypotheses were not supported, further research is warranted as results indicate the potential for using self-distancing approaches to increasing help-seeking among some people with depressive symptomatology.” 25
• EXAMPLE 3. Hypothesis-generating research to establish a framework for hypothesis testing (qualitative research)
• “We hypothesize that compassionate care is beneficial for patients (better outcomes), healthcare systems and payers (lower costs), and healthcare providers (lower burnout). ” 26
• Compassionomics is the branch of knowledge and scientific study of the effects of compassionate healthcare. Our main hypotheses are that compassionate healthcare is beneficial for (1) patients, by improving clinical outcomes, (2) healthcare systems and payers, by supporting financial sustainability, and (3) HCPs, by lowering burnout and promoting resilience and well-being. The purpose of this paper is to establish a scientific framework for testing the hypotheses above . If these hypotheses are confirmed through rigorous research, compassionomics will belong in the science of evidence-based medicine, with major implications for all healthcare domains.” 26
• EXAMPLE 4. Statistical hypothesis (quantitative research)
• - An assumption is made about the relationship among several population characteristics ( gender differences in sociodemographic and clinical characteristics of adults with ADHD ). Validity is tested by statistical experiment or analysis ( chi-square test, Students t-test, and logistic regression analysis)
• “Our research investigated gender differences in sociodemographic and clinical characteristics of adults with ADHD in a Japanese clinical sample. Due to unique Japanese cultural ideals and expectations of women's behavior that are in opposition to ADHD symptoms, we hypothesized that women with ADHD experience more difficulties and present more dysfunctions than men . We tested the following hypotheses: first, women with ADHD have more comorbidities than men with ADHD; second, women with ADHD experience more social hardships than men, such as having less full-time employment and being more likely to be divorced.” 27
• “Statistical Analysis
• ( text omitted ) Between-gender comparisons were made using the chi-squared test for categorical variables and Students t-test for continuous variables…( text omitted ). A logistic regression analysis was performed for employment status, marital status, and comorbidity to evaluate the independent effects of gender on these dependent variables.” 27

EXAMPLES OF HYPOTHESIS AS WRITTEN IN PUBLISHED ARTICLES IN RELATION TO OTHER PARTS

• EXAMPLE 1. Background, hypotheses, and aims are provided
• “Pregnant women need skilled care during pregnancy and childbirth, but that skilled care is often delayed in some countries …( text omitted ). The focused antenatal care (FANC) model of WHO recommends that nurses provide information or counseling to all pregnant women …( text omitted ). Job aids are visual support materials that provide the right kind of information using graphics and words in a simple and yet effective manner. When nurses are not highly trained or have many work details to attend to, these job aids can serve as a content reminder for the nurses and can be used for educating their patients (Jennings, Yebadokpo, Affo, & Agbogbe, 2010) ( text omitted ). Importantly, additional evidence is needed to confirm how job aids can further improve the quality of ANC counseling by health workers in maternal care …( text omitted )” 28
• “ This has led us to hypothesize that the quality of ANC counseling would be better if supported by job aids. Consequently, a better quality of ANC counseling is expected to produce higher levels of awareness concerning the danger signs of pregnancy and a more favorable impression of the caring behavior of nurses .” 28
• “This study aimed to examine the differences in the responses of pregnant women to a job aid-supported intervention during ANC visit in terms of 1) their understanding of the danger signs of pregnancy and 2) their impression of the caring behaviors of nurses to pregnant women in rural Tanzania.” 28
• EXAMPLE 2. Background, hypotheses, and aims are provided
• “We conducted a two-arm randomized controlled trial (RCT) to evaluate and compare changes in salivary cortisol and oxytocin levels of first-time pregnant women between experimental and control groups. The women in the experimental group touched and held an infant for 30 min (experimental intervention protocol), whereas those in the control group watched a DVD movie of an infant (control intervention protocol). The primary outcome was salivary cortisol level and the secondary outcome was salivary oxytocin level.” 29
• “ We hypothesize that at 30 min after touching and holding an infant, the salivary cortisol level will significantly decrease and the salivary oxytocin level will increase in the experimental group compared with the control group .” 29
• EXAMPLE 3. Background, aim, and hypothesis are provided
• “In countries where the maternal mortality ratio remains high, antenatal education to increase Birth Preparedness and Complication Readiness (BPCR) is considered one of the top priorities [1]. BPCR includes birth plans during the antenatal period, such as the birthplace, birth attendant, transportation, health facility for complications, expenses, and birth materials, as well as family coordination to achieve such birth plans. In Tanzania, although increasing, only about half of all pregnant women attend an antenatal clinic more than four times [4]. Moreover, the information provided during antenatal care (ANC) is insufficient. In the resource-poor settings, antenatal group education is a potential approach because of the limited time for individual counseling at antenatal clinics.” 30
• “This study aimed to evaluate an antenatal group education program among pregnant women and their families with respect to birth-preparedness and maternal and infant outcomes in rural villages of Tanzania.” 30
• “ The study hypothesis was if Tanzanian pregnant women and their families received a family-oriented antenatal group education, they would (1) have a higher level of BPCR, (2) attend antenatal clinic four or more times, (3) give birth in a health facility, (4) have less complications of women at birth, and (5) have less complications and deaths of infants than those who did not receive the education .” 30

Research questions and hypotheses are crucial components to any type of research, whether quantitative or qualitative. These questions should be developed at the very beginning of the study. Excellent research questions lead to superior hypotheses, which, like a compass, set the direction of research, and can often determine the successful conduct of the study. Many research studies have floundered because the development of research questions and subsequent hypotheses was not given the thought and meticulous attention needed. The development of research questions and hypotheses is an iterative process based on extensive knowledge of the literature and insightful grasp of the knowledge gap. Focused, concise, and specific research questions provide a strong foundation for constructing hypotheses which serve as formal predictions about the research outcomes. Research questions and hypotheses are crucial elements of research that should not be overlooked. They should be carefully thought of and constructed when planning research. This avoids unethical studies and poor outcomes by defining well-founded objectives that determine the design, course, and outcome of the study.

Disclosure: The authors have no potential conflicts of interest to disclose.

Author Contributions:

• Conceptualization: Barroga E, Matanguihan GJ.
• Methodology: Barroga E, Matanguihan GJ.
• Writing - original draft: Barroga E, Matanguihan GJ.
• Writing - review & editing: Barroga E, Matanguihan GJ.

• Health Tech
• Health Insurance
• Medical Devices
• Gene Therapy
• Neuroscience
• H5N1 Bird Flu
• Health Disparities
• Infectious Disease
• Mental Health
• Cardiovascular Disease
• Chronic Disease
• Alzheimer's
• Coercive Care
• The Obesity Revolution
• The War on Recovery
• Adam Feuerstein
• Matthew Herper
• Jennifer Adaeze Okwerekwu
• Ed Silverman
• CRISPR Tracker
• Breakthrough Device Tracker
• Generative AI Tracker
• Obesity Drug Tracker
• 2024 STAT Summit
• Wunderkinds Nomination
• STAT Madness
• STAT Brand Studio

Don't miss out

Subscribe to STAT+ today, for the best life sciences journalism in the industry

The newest tool to prevent STIs is not available to everyone: A call for participatory and inclusive research

By Oni Blackstock and Whitney C. Irie July 22, 2024

I n the midst of an “out of control” epidemic of sexually transmitted infections (STIs) in the United States, a newly recommended tool for preventing these infections has emerged. Sadly, this novel prevention strategy is not yet available to everyone who might benefit from it due to the insufficiently participatory and inclusive nature of the research used to study it.

In June 2024, the federal Centers for Disease Control and Prevention released eagerly awaited clinical guidelines for a strategy that significantly reduces the risk of sexually transmitted infections. These guidelines advise health care providers to counsel cisgender men (cisgender is when one’s current gender identity is the same as their assigned sex at birth) and transgender women who have sex with men about using doxycycline post-exposure prophylaxis (doxy PEP) if they have had a bacterial STI in the past year.

advertisement

Doxy PEP involves taking a dose of doxycycline, an oral antibiotic, within 72 hours after a potentially high-risk sexual exposure. The goal is to decrease the risk of bacterial STIs: chlamydia, syphilis, and gonorrhea.

Following shared decision-making, in which providers and patients work together to make a clinical care decision, providers can prescribe doxy PEP — but at least for now, only to cisgender men and transgender women who have sex with men, groups for which doxy PEP has been shown to be highly effective.

Doxy PEP also has the potential to substantially benefit cisgender women, transgender men, and other people assigned female at birth, but these groups must wait to see if doxy PEP will be officially available to them. And health care providers are once again forced to tell cisgender women that they must wait for more studies and to tell transgender men and other people assigned female at birth that there is no doxy PEP data in these groups because they were not included in initial doxy PEP studies.

Related: CDC endorses prophylactic antibiotics to reduce STIs in high-risk groups

The limited reach of a novel STI prevention tool such as doxy PEP is a missed opportunity because cisgender women in the U.S. and globally have the highest rates of chlamydia. Along with other people assigned female at birth, they bear the burden of long-term health problems related to chlamydia and gonorrhea, such as pelvic inflammatory disease, ectopic pregnancy, and infertility. Rates of syphilis among cisgender women and congenital syphilis in their babies have reached record highs over the past decade. Notably, STIs can also facilitate the transmission and acquisition of HIV.

In the U.S., the epidemic of sexually transmitted infections has been marked by stark racial inequities in rates and access to screening, testing, and treatment, with Black, Latinx, and Indigenous communities being disproportionately affected.

The persistent void in generalizable knowledge about strategies for preventing STIs and HIV for cisgender women, transgender men, and other people assigned female at birth highlights the critical and urgent need for participatory and inclusive research design in clinical trials.

The need for participatory research

Participatory research involves the meaningful collaboration of community members, especially those from the population of focus, in all phases of the research, including designing the clinical trial. This helps to ensure, in part, that recruitment of participants as well as planned measures to help keep participants in the study and using the intervention are responsive to the social and structural context in which the clinical trial is taking place.

To be fair, researchers did conduct a clinical trial of doxy PEP in cisgender women , but its outcomes were affected by the same issue that plagued early clinical trials of HIV pre-exposure prophylaxis (PrEP) among cisgender women. Doxy PEP was not effective at preventing bacterial STIs among cisgender women likely because many of those randomized to receive the treatment did not use it.

Related: CDC proposes new antibiotics policy aimed at reducing STIs

This echoes findings from early HIV PrEP trials among cisgender women in sub-Saharan Africa , where oral PrEP did not appear to be effective in preventing HIV because most women in the trial did not take the medication as prescribed. A follow-up qualitative study found that participants had concerns about the investigational nature of the treatment and potential side effects, and were discouraged from taking PrEP by peers, partners, and community members. Some participants also enrolled in the study to access quality health care which would not have been available them otherwise.

Using a participatory approach to designing STI and HIV biomedical prevention clinical trials in the U.S. as well as globally could help to ensure greater buy-in among potential participants and community members for the intervention being studied as well as for the overall clinical trial.

In the case of the doxy PEP clinical trial among cisgender women, an example of a more participatory approach would be working in close collaboration with community members to plan and conduct in-depth interviews and focus groups, or even listening sessions, with community members, particularly from the population of focus, during the trial’s design phase. This could have helped to identify the reasons that might compel women to take (or not take) the antibiotic.

Although the researchers had likely hoped to mitigate any adherence-related issues by enrolling women already using oral HIV PrEP, a comprehensive understanding of their motivations for taking doxy PEP, how these motivations may differ from those for taking oral PrEP, as well as existing social support from peers, partners, and the community for doxy PEP would have been advantageous.

When research questions originate from outside the community, where priorities may be different, formative research in collaboration with community input is essential so community priorities are considered and centered in the clinical trial design.

Ensuring inclusivity in STI prevention clinical trials

While strides have been made to at least include cisgender women in these types of clinical trials, transgender men, transmasculine, non-binary, and other people assigned female at birth have been largely left out, despite studies demonstrating a critical need for interventions among these groups.

For example, one U.S.-based study found that, compared with cisgender men, transgender men who have sex with men are less likely to have been tested for HIV or STIs. Another study found that the prevalence of HIV among transmasculine individuals who have sex with men is higher than in the general population.

Sign up for First Opinion

The smartest thinkers in life sciences on what's happening — and what's to come

The lack of inclusion of transgender men and other people assigned female at birth in clinical trials limits the options available to these groups to protect themselves from HIV and STIs, and further compounds an already existing scarcity of gender-affirming care for them. An inclusive approach means designing clinical trials to ensure that all groups who could benefit from an intervention are included, particularly those that have been historically excluded.

Moving forward: inclusive and participatory research for effective prevention

Researchers can help to broaden access to available STI and HIV prevention options for marginalized communities as we work towards ending the STI and HIV epidemics by leveraging a participatory and inclusive approach to biomedical prevention clinical trials. Meaningful and in-depth collaboration with the community in developing and implementing clinical trials, especially in the planning phases, is key to accurately determining an intervention’s efficacy.

Future clinical trials must be intentional in their inclusion of transgender men and other people assigned female at birth. It is encouraging to see this is beginning to happen. A recent clinical trial of twice-yearly injectable lenacapavir for HIV PrEP in cisgender women was stopped early after showing it was 100% effective. An ongoing trial of this same approach includes transgender men and gender non-binary individuals who have sex with partners assigned male at birth.

It is no longer tenable to simply tell our patients that we don’t have the data. Wider use of participatory and inclusive research would mean we don’t have to.

Oni J. Blackstock, M.D., M.H.S., is a primary care and HIV physician and researcher, and founder and executive director of Health Justice, a racial and health equity consulting firm. Whitney C. Irie, Ph.D., M.S.W., is a social scientist and assistant professor at Boston College’s School of Social Work, a lecturer on population medicine in the Department of Population Medicine at Harvard Medical School, and an adjunct faculty member at The Fenway Institute.

LETTER TO THE EDITOR

Have an opinion on this essay submit a letter to the editor here ., about the authors reprints, oni blackstock, whitney c. irie.

infectious disease

sexual health

STAT encourages you to share your voice. We welcome your commentary, criticism, and expertise on our subscriber-only platform, STAT+ Connect

To submit a correction request, please visit our Contact Us page .

Recommended

Recommended Stories

NIDA should beware of funding companies that violate people’s privacy

Readers respond to essays on philanthropy and nursing schools, opioid overdoses, and more

STAT Plus: Broad Institute, facing end of Microsoft cloud contract, shuffles data science leadership

STAT Plus: A cancer care startup is making a new bet with payers and providers: only pay if it works

STAT Plus: Top FDA officials weighing regulation of ultra-processed foods, internal documents show

• Open access
• Published: 15 July 2024

The role of duty, gender and intergenerational care in grandmothers’ parenting of grandchildren: a phenomenological qualitative study

• José Miguel Mansilla-Domínguez 1 ,
• Ana María Recio-Vivas 1 ,
• Laura Lorenzo-Allegue 1 ,
• José Miguel Cachón-Pérez 1 ,
• Laura Esteban-Gonzalo 1 , 2 &
• Domingo Palacios-Ceña 3  

BMC Nursing volume  23 , Article number:  477 ( 2024 ) Cite this article

107 Accesses

Metrics details

The aim of this study was to describe grandmothers’ experiences of taking care of their grandchildren in terms of their care-giving tasks, motivations and emotions.

A qualitative phenomenological study was conducted. Purposive sampling was used, based on the relevance of the research question. Seventeen participants were included, women ≥ 65 years old, grandmothers who care for their grandchildren at least 10 h per week and who attended the Nursing units of the Primary Care Health Centers (Madrid Public Health Service). Seventeen in-depth interviews were conducted. The interviews were audio-recorded, transcribed verbatim and thematic analysis was carried out from the perspective of hermeneutic phenomenology. For the analysis, the Excel program was used to organize and share the coding process. Also, we followed COREQ guidelines.

Four main themes were identified: (a) Care out of obligation, where participants feel an obligation to help their children by caring for grandchildren, regardless of their number, and prefer to do so voluntarily; (b) Care out of responsibility, where grandmothers see their role as a responsibility that includes saving costs by caring for grandchildren and facilitating their children’s work life balance; (c) Care as a social duty, reflecting a moral commitment inherited from their mothers to help future generations; and (d) Construction of care from a gender perspective, where grandmothers, as women, primarily assume the care and upbringing of grandchildren.

Our results contribute to increase knowledge about childcare provided by grandmothers to their grandchildren. Grandmothers become fundamental pillars of families by helping their children balance family and work. Behind this care there is a strong sense of obligation, duty and generational responsibility. Grandmothers’ help presents differences in the distribution of tasks and care by sex. Identifying factors that motivate grandmothers to care for their grandchildren helps nurses to perform higher quality comprehensive care.

Peer Review reports

Introduction

Aging populations as well as social and demographic change have led to the modification and reorganization of family relationships and particularly those taking place between grandparents and their grandchildren [ 1 ]. Previous studies [ 2 , 3 , 4 ] show increased involvement of grandmothers in the care of grandchildren, a shared experience among the elderly. [ 2 , 3 , 4 ].

Grandparents have always played a fundamental role in the transmission of values to their grandchildren. But this role has changed as they have taken over the care, upbringing and education of their grandchildren [ 5 ]. Grandmothers/grandfathers have become family welfare providers in a context of economic and social crisis [ 6 ]. In recent decades, the percentage of grandparents aged 65 or older who report caring for their grandchildren has doubled [ 7 ]. In their work on grandparents’, [ 8 ] Gleeson et al. [ 8 ] show how commitment to caring for grandchildren has increased from 11% in 2010 to 17% in 2018. Similarly, The Survey of Health, Ageing and Retirement in Europe [ 9 ] describes how 22.07% of grandparents care for their grandchildren. This same report found Spain to be the country where there are fewer grandparent caregivers, but those who do provide care, do so with greater intensity than grandparents in other European countries. Spanish grandmothers devote an average of 7.1 h a day to caregiving (including weekends), compared to 4.9 h a day in Europe. Moreover, this care is provided on a voluntary basis [ 10 ].

Voluntariness is sometimes experienced by grandmothers as an obligation, influenced by the values and education they have received and acquired during their lives [ 11 ]. In addition, there are gender differences in the distribution of caregiving tasks and responsibilities among grandmothers and grandfathers [ 12 ]. Grandmothers assume the main responsibility for caregiving and adopt the role of the caregiver, while grandfathers take on a secondary or complementary caregiver role [ 12 ]. Thus, unequal distribution of caregiving responsibilities during their own parenthood is perpetuated in the care of grandchildren [ 13 ].

Previous qualitative studies [ 7 ] have described how grandmothers who cared for their grandchildren were actively involved in their socialization and education. Lee & Blitz [ 14 ] note how grandmothers take on a greater burden because of their greater involvement in caring for grandchildren. Despite this burden, the reasons that influence them to accept the care of their grandchildren are based on the voluntariness of the care, and the acquired gender role [ 15 ]. Sometimes this decision is made jointly with the children’s parents or in response to a sense of obligation [ 16 ].

In contrast, the evidence regarding grandmother caregiving is scarce. Therefore, what are the lived experiences of grandmothers with grandchildren’s care? What are the reasons for this care? The objective of the present study was to describe the lived experiences of grandmothers who take care of their grandchildren regarding their care, motivations, and feelings.

Materials & Methods

A qualitative phenomenological study was conducted based on Husserl’s framework [ 17 , 18 , 19 , 20 ]. In the field of qualitative studies, phenomenology attempts to understand other people’s lived experiences by using first-person narratives and other sources such as personal letters, diaries and drawings [ 21 , 22 , 23 ]. Thus, in phenomenology, two approaches in particular are influential in healthcare: (a) descriptive phenomenology has the closest connection with Husserl’s original conception of phenomenology and focuses on creating detailed descriptions of the specific experiences of others [ 24 , 25 , 26 ] (ref) and (b) interpretative or hermeneutic phenomenology, which seeks to understand the nature of human beings and the meanings they bestow upon the world by examining language in its cultural context [ 22 ]. Husserl indicates the need to “retain” beliefs (bracketing) which, using the phenomenological reduction approach, would allow a critical examination of the phenomena without the influence of the researcher’s own beliefs [ 23 ]. In our study, bracketing was achieved by carefully recording the positions taken by the researchers beforehand and by using in-depth interviews as the main data collection tool [ 19 , 21 ]. In this way, we sought to avoid the researcher’s influence on the data and reveal the nature of the phenomenon through the patients’ accounts [ 23 ].

This study was carried out according to the Consolidated Criteria for Reporting Qualitative Research [ 27 ]. The study protocol was approved by the Ethical Committee of the Universidad Rey Juan Carlos (code: 01/2015) and the Local Ethical Committee of South Research-Madrid Health Service (code: 08/2012).

Research team, reflexivity and context

All participants provided oral informed consent prior to their inclusion. All eight participating researchers (five women) were research nurses and university professors, experienced in health science investigation. The study protocol was audited by an external researcher.

Prior to the study, the positioning of the researchers (Table  1 ) was established through two briefing sessions addressing the theoretical framework, the context, their prior experience and their motivation for the research [ 21 ]. Los investigadores no tenían conflicto de intereses con las participantes.

Participants and sampling strategies

Purposive sampling was used, based on relevance to the research question (not clinical representativeness [ 28 ]. In phenomenology, it is necessary to include those people who experienced the phenomenon under study [ 20 , 29 ]. In this case, participants who had relevant information about the phenomenon (caring for grandchildren) were included.

Sampling and data collection was pursued until the researchers achieved information redundancy, at which point no new information emerged from the data analysis [ 28 ], and/or when the information collected did not contribute anything new to the development of the properties and dimensions of the analysis [ 28 ]. In our study, this situation occurred after including 17 participants.

The study subjects included females ≥ 65 years old, grandmothers who care for their grandchildren at least 10 h per week, are retired from formal employment and do not live in the same household as their grandchildren. Exclusion criteria were grandmothers who cared for their grandchildren less than two days a week and/or less than one hour a day. The inclusion criterion determined by hours is based on the report “Productive ageing: Grandparents’ provision of care to grandchildren. Implications for their health and well-being” for the Institute of Elderly and Social Services (IMSERSO) of the Spanish Ministry of Labor and Social Affairs of 2008 [ 30 ], where it reflects that grandmothers who provide prolonged and intense care with great dedication to their grandchildren exceed at least 10 h per week.

The recruitment process of the participants consisted of an initial phase, where the Heads of Nursing and Centre Directors of the Health Centers of Pinto (Madrid) were contacted and a presentation of the project was made. The Heads of Nursing provided the telephone numbers of the grandparents recruited by Nursing and Medical professionals from the health centers in order to subsequently establish contact with the participants, thus guaranteeing a less invasive entry into the study by the researcher. The interviews were carried out in the Aula of the health center for two months, with all participants attending alone.

Data collection

Based on the phenomenological design, first-person data collection tools (in-depth interviews) and researchers field notes were used simultaneously [ 20 ]. Unstructured in-depth interviews were used as the main tool for data collection. The interview started with an open question: ‘What is your experience with the care of your grandchildren? Thereafter, the researchers listened carefully, noted key words and topics identified in the subjects’ responses and used their answers to ask for and clarify content [ 20 ]. The interviews followed a guide created exclusively for this study (supplementary material- Appendix 1 ). In this way, relevant information was collected from the perspective of the participants. Researchers also used prompts or probes during the interviews: (a) to encourage the participants to provide more detail (‘Can you tell me a bit more about that?’), (b) to encourage the participant to keep talking (‘Have you experienced the same thing since?’), (c) to resolve confusion (paraphrasing of something that the patient had said) and (d) to show full attention by the researcher (‘That’s really interesting, please tell me more’) [ 20 ].

All interviews were tape-recorded and transcribed verbatim. Each of them ranged from 45 to 75 min (mean 57.85; SD 5.60). No third party was present in the interviews.

First, a complete and literal transcription of each interview and the researchers’ field notes was drafted. Subsequently, an inductive analysis of the data was conducted [ 20 ] by two researchers (JMMD, DPC). During the data analysis stage, a model proposed by Amedeo Giorgi was used [ 31 ] which distinguishes 5 stages of data processing: (1) data collection; (2) reading literal transcription of the interviews; (3) breaking down the descriptions into separate units in order to identify the relevant meaning units for the phenomenon under study; (4) data organization and listing from the perspective of the discipline using an encoding process and, finally, (5) data synthesis and summarisation in order to communicate these to the scientific community. The breaking down and encoding process consisted of identifying the most descriptive content in order to obtain meaningful units and then reducing and identifying the most common meaningful groups. In this manner, groups of meaning units were formed, i.e., similar points or content allowing for emergence of themes that described the study participants’ experience Fig.  1 .

Example of analysis and coding procedure

For clarity, a matrix was built with results obtained from the analysis. Joint team meetings were held to display, combine, integrate, and identify themes. In case of difference of opinions, theme identification was based on consensus among research team members. No qualitative software was used to analyse the data. The Excel program (Microsoft Corp, Redmond, WA) was used to organize and share the coding process. In addition, researchers used the same Excel template shared through OneDrive (Microsoft Corp, Redmond, WA) to modify, highlight, and extract the results of the coding process simultaneously as part of a network.

The Lincoln and Guba criteria were used (Table  2 ) to establish data trustworthiness [ 32 ].

Seventeen grandmothers participated in this study. The mean age was 67.5 years (SD 3.79). The mean number of hours spent per week caring for grandchildren was 24.6 (SD 3.4). In addition, the mean number of days per week spent caring for grandchildren was 5.8 (SD 1.2). See Table  3 : Descriptive characteristics of the participants.

Four main themes were identified: (a) Caregiving out of obligation, (b) Caregiving out of responsibility, (c) Caregiving as a social duty, and (d) Constructing caregiving from a gender perspective (Fig.  2 ). In addition, examples of narratives obtained from participants are included in Appendix 2 , presented by themes and meaning clusters.

Map of themes and Common meaning groups

Caregiving out of duty and obligation

Participants described feeling a sense of obligation to help their children by caring for their grandchildren. This obligation to provide care applies to all the grandchildren, regardless of the number, understanding that the duty to help all their children is equal. This feeling of duty to help all their children equally, without discrimination, makes them unable to refuse even in the presence of physical limitations.

The grandmothers´ feeling of duty and obligation also appears when it is necessary to take their grandchild to a day care center and/or caregivers outside the family. The reasons for taking care of the grandchildren themselves are that they offer better, more exhaustive and individualized care than a day care center and provide love and affection. Also, avoiding the use of these day care centers saves money. In addition, they do not allow their grandchildren to be taken to a daycare center while they are available.

“I think I take better care of them. Because in a day care center there are many children , I think they don’t take care of them as well as I do. Also , I can’t consent to them putting her in daycare while I’m there.” (P7) .

Duty and obligation are experienced more satisfactorily if the grandmothers voluntarily choose to care for their grandchildren. Voluntary care combined with the feeling of duty to help the family means that it is not perceived as a sacrifice and that the experience is positive. Although grandmothers invest part of “their life” in caring for their grandchildren, understood as the loss of their leisure time and routines, they feel that they have “fulfilled their duty”. On the contrary, it is experienced negatively when the grandmother does not want to do it and it is imposed as “obligatory”.

“It is a job I have to do , it is a feeling , not of obligation , but that they are part of your life. They are your grandchildren , and your daughter is working and what are you going to do , we have to make do. And if we have to give up our activities , we do. It pays off for me” (P9).

Caregiving as a responsibility

The grandmothers described how they have “a responsibility” of generational support. Their help or support have certain purposes such as achieving economic savings for their children by caring for the grandchildren, thus avoiding the hire of external personnel or day care centers. Another purpose is to help their children balance work and family life by helping out with the grandchildren. On occasions, the grandmothers were satisfied to report how their help was instrumental in complementing the family salary and reconciling their working hours and shifts. Still another goal for the grandmothers is that of feeding their grandchildren with which they feel they guarantee a healthy, wholesome and balanced diet for them. And finally, grandmothers aim to educate their grandchildren; the transmission of values is considered a great responsibility.

“ My daughter works and , in addition , she is studying. Dedication to her work and her child is impossible for her. If I can help my family in any way , I must do it and I feel satisfied.” (P11).

The grandmothers narrated how their role becomes prominent in difficult and crisis situations, in which they do not hesitate to take care of their grandchildren as they feel responsible to help and support the family. Examples of crises are the onset of illness (grandchildren/children), and/or separation/divorce.

“ I started taking care of my granddaughter after a divorce. The child was frightened , scared and , well , how hard it is for children at that time. Your daughter is at home , you have to protect her more , she is divorced , her income has dropped by half. You have to help them ” (P1).

Caregiving as a social duty

The participants described how they have a social duty, a moral debt acquired when their own mothers took care of their own children. They believe they should reciprocate by helping the following generations. Our participants perceive “their grandmother figure” as a socially accepted and sought-after figure between mothers and daughters. It has always existed, and they contribute to generational continuity through their caregiving. Our participants narrated how their mothers and mothers-in-law were an important help in reconciling their work and personal lives and now want to “pay back” that generational debt.

“I was working when I had my daughter , so my mother took care of her. My mother put her to bed , fed her , dressed her , how can I not do it now for my daughter?” (P3).

Helping out is not only focused on taking care of the grandchildren, but also includes housework and cleaning. In the majority of the participants, caregiving is mainly provided for the children of the grandmothers ‘daughters, i.e., the maternal lineage. This behavior is a shared norm, which is maintained and perpetuated generationally (from mother to daughter). Greater intimacy and trust between mother and daughter are given as reasons by our participants.

“ I think that in the case of women there is always a tendency towards her mother. I guess it will be because of the trust. Right now , the girl I have belongs to my daughter , and I have more confidence to tell her “don’t put those clothes on the girl” and with my daughters-in-law I have kept quiet and have not said anything to them” (P8).

Constructing caregiving from a gender perspective

Our participants recounted how the burden of caring for grandchildren falls to them, that is, on the women. Grandmothers take care of their grandchildren, supervise their activities and are ultimately responsible for their upbringing. While grandmothers perform household chores and manage the feeding and hygiene of the grandchildren, the supervisory role may sometimes be relinquished to the grandfather. Grandmothers consider the grandfather to be a “helper” in caring for the grandchildren, but the grandmothers themselves, are ultimately responsible for childcare.

“He is at home , but managing the granddaughter is my job. I take care of everything , making the food , washing the clothes , sewing , ironing. Of course , he doesn’t do those things , because he doesn’t know how. There is a lot of difference between what grandmothers do and what grandfathers do (P9).

The grandfathers’ participation is centered on leisure-entertainment activities and transportation to school, and their tasks are related to leisure, play and entertainment (going out to the park to play), doing homework. There are also differences in the distribution of tasks by gender in relation to activities outside and inside the home. While grandmothers stay closer to home and take care of activities inside the home, grandfathers are in charge of everything that involves going out of the home with their grandchildren.

“I do see differences between us and grandfathers They have had that upbringing from our time. They have not been at home like us , who are more accustomed to the house and to taking care of things , and we manage it better. Because at home , I am so comfortable and he is so bored” (P5).

Our participants recognize that there are gender differences in the distribution of tasks and responsibilities in the care of their grandchildren. They feel they play a mayor and leading role, while grandfathers take on an “auxiliary” role.

Grandmothers feel that they have a duty and obligation to care for their grandchildren in order to help their children. This feeling of duty appears in the work of Badenes & López [ 33 ] in which they describe how grandparents feel an obligation to help rather than accept that their grandchildren be cared for by strangers. In situations where their children need outside help to care for their grandchildren, grandmothers feel a deep-rooted sense of responsibility, family commitment and obligation. However, this does not mean that there are no consequences. Previous studies [ 14 , 33 , 34 ] noted that grandparents experience emotional exhaustion when caring for their grandchildren for many hours and they give up other activities or wishes that would be appropriate for an older person. The voluntary aspect of caregiving is described by our participants. Authors such as Broese van Groenou & De Boer [ 10 ] and Buchanan & Rotkirch [ 1 ], point out that caring for grandchildren is experienced differently depending on whether it derives from a voluntary choice or an obligation. When childcare is not a voluntary choice, previous studies show that the experience can negatively affect the physical and emotional health of grandmothers [ 10 ]. Buchanan & Rotkirch [ 1 ] observe that caregiving produces satisfaction when it is chosen. But when it is an “imposition” by their children, against their will, or truncates life projects such as retirement, then it becomes a key factor in the presence of feelings of overload. Margolis &Verdery [ 35 ] and Lakomý & Kreidl [ 36 ] found that grandmothers play a key role in their children´s ability to balance their careers with family life. This help is an attempt by grandmothers to protect the family and support their children in advancing their careers or keeping their jobs. In Spain, Noriega et al. [ 7 ] show how grandmothers help many young couples to balance work and family life and also become a significant source of economic support.

Our results coincide with previous studies [ 14 , 37 , 38 ], where grandparents adopt the role of caregivers of their grandchildren, as a tool to help their children prevail over difficulties in reconciling work and family life. The results of the present study also coincide with Bordone et al. [ 39 ] who find that this help coming from grandmothers is accentuated in crisis situations such as children’s illnesses, separations, divorces or unemployment. Furthermore, Danielsbacka et al. [ 40 ] point out that grandparents perceive themselves as part of a generation whose dedication to the family contributes to the balance and economic sustainability of society. Leeson [ 41 ] depicts the role of the grandmother as a fundamental pillar for the maintenance of the family unit. They perform mediation functions in conflicts between their children and grandchildren and serve as a refuge from crisis situations in the family. Lee & Blitz [ 14 ] observed that grandmothers not only fulfill the role of helpers in the family. They feel they are “grandmothers for everything”, as they say, because their children resort to the saving figure of grandmothers as the only way to maintain their socioeconomic status and quality of life in the face of any difficulty that may come up.

The helping role to fulfill a debt has been described previously in the work of Di Gessa et al. [ 9 ], who identified how grandmothers’ helping role derived from their own observation and past experience with their mothers and their own grandmothers. These authors state that the helping role of grandmothers is progressively constructed as they imitate the role played by their own grandmothers, mothers and mothers-in-law. The involvement of grandmothers in helping families has facilitated the maintenance of their children’s work or professional development. Hill [ 42 ] proposes the incorporation of women into the labor market and aging of the population as factors leading grandmothers to take on childcare of their grandchildren. These authors argue that grandparents live longer, have acceptable health and are available to care for their grandchildren. On the other hand, Margolis & Verdery [ 35 ] identify the incorporation of both parents into the labor market, inflexible work schedules and the absence of social policies aimed at helping to balance family life and careers as factors that have contributed to the incorporation of grandmothers as an essential resource for families. The presence of generational duty between mothers and daughters is a form of intergenerational family solidarity. For Hank et al. [ 43 ] that solidarity consists of providing care and help among family members. Family solidarity is consolidated as fundamental social capital for the well-being of people, in which grandmothers become a key support as they come to the aid of their children and grandchildren in times of need.

Our results detect a gender distribution of tasks between grandmothers and grandfathers in caring for grandchildren. The burden of care and the final responsibility falls to grandmothers, who play a leading role, while the grandfather is an assistant. Our results coincide with Notter [ 13 ], who observes that household chores and childcare fall mainly to older women. This means that feminization of old age may perpetuate gender-based caregiving inequities. Differences in the distribution of activities between grandmothers and grandfathers continue to exist today and are based on the social construction that defines his or her roles and participation as caregivers [ 5 ]. The participants in the present study point out their importance as protagonists in the care of their grandchildren. Several authors such as Broese van Groenou & De Boer [ 10 ] and Notter [ 13 ] have observed how greater involvement of grandmothers in caregiving tasks for grandchildren leads to greater satisfaction with the role of caregiver. This factor may explain the grandmothers’ feeling of self-importance, as opposed to the “auxiliary” role assigned to the grandfather. Moreover, Perry & Daly [ 12 ] have identified differences between grandmothers and grandfathers as to the performance of caregiving tasks and degree of dedication to and involvement with their grandchildren. These authors have found that grandmothers are the first figure to be asked for help by their children in caring for their grandchildren. In this way, the grandfather becomes a secondary or complementary caregiver.

Strengths and limitations

One of the strengths of this study was to obtain a profile of our participants, which could be used to explore other dimensions (emotions, feelings, etc.) in other research. This profile consists of women over 65 years of age, with an average age of 68 years, living with their partner, and devoting a large part of their time per week to caring for the grandchildren (25 h and/or 6 days per week on average). The current study has not included the experience of grandfathers. The authors believe it would be of interest to strengthen the present study to include the perspective of grandfathers in order to comprehend the distribution of grandparenting tasks from a gender perspective.

There are different proposals in qualitative research [ 29 ]. Specifically, in this study we have focused on studying the lived experience through phenomenology. But it would be an important aspect of future research to study the cultural aspects of care in this group of participants using ethnography, and to analyse in depth the process of caring for grandchildren, which agents are involved (inside and outside the family) in which contexts and circumstances using grounded theory [ 29 , 32 ].

Implications for policy and practice

Our results would be of interest to develop support programs for grandmothers aiming to prevent fatigue and exhaustion and support them in caring for their grandchildren. In addition, from policy view it would be necessary to develop studies with a gender perspective on the care provided by grandmothers and grandfathers. It is also essential to promote awareness in various public institutions about the development of intergenerational policies and programmes between grandparents and grandchildren by developing psycho-educational and support programmes where grandparents are helped to: (a) recognise the valuable role of grandparents in today’s society (b) foster social and family cohesion (c) facilitate the transmission of values and strengthen family ties (d) promote active ageing through social participation.

Conclusions

This research adds knowledge about grandmothers’ experiences of caring for their grandchildren, in terms of their caregiving, motivations, and feelings. Grandmothers are a fundamental pillar of families, enabling the balancing of family and work through their help in caring for grandchildren. Grandmothers embrace the role of caregivers of their grandchildren due to a strong sense of duty, obligation and generational responsibility. However, caregiving tasks were found to be assigned to grandparents according to gender.

These results may help us understand the informal care provided by grandmothers within families and how this caregiving facilitates family life and work for the children’s parents. Knowing and understanding the motivations and experiences of grandmothers in caring for grandchildren will enable the application of comprehensive care. Furthermore, the study of grandmothers’ care of grandchildren and the knowledge of the construction and transmission of care between generations (grandmothers-mothers and daughters) would allow us to understand not only the grandmothers’ experiences, but also to identify the cultural and social factors involved, and how these may influence health and the gender perspective of care.

Data availability

The data that support the findings of the study are available on request from the corresponding author, upon reasonable request. The data are not public due to ethics restrictions.

Buchanan A, Rotkirch A. Twenty-first century grandparents: global perspectives on changing roles and consequences. Contemp Soc Sci. 2018;13(2):131–44. https://doi.org/10.1080/21582041.2018.1467034 .

Article   Google Scholar  

Qi X. Floating grandparents: rethinking family obligation and intergenerational support. Int Sociol. 2018;33(6):761–77. https://doi.org/10.1177/0268580918792777 .

Burn K, Szoeke C. Grandparenting predicts late-life cognition: results from the women’s healthy Ageing Project. Maturitas. 2015;81(2). https://doi.org/10.1016/j.maturitas.2015.03.013 .

Kim J, Park EC, Choi Y, Lee H, Lee SG. The impact of intensive grandchild care on depressive symptoms among older koreans. Int J Geriatr Psychiatry. 2017;32(12). https://doi.org/10.1002/gps.4625 .

Coall DA, Hilbrand S, Sear R, Hertwig R. Interdisciplinary perspectives on grandparental investment: a journey towards causality. Contemp Soc Sci. 2018;13(2). https://doi.org/10.1080/21582041.2018.1433317 .

Condon J, Luszcz M, McKee I. The transition to grandparenthood: a prospective study of mental health implications. Aging Ment Heal. 2018;22(3). https://doi.org/10.1080/13607863.2016.1248897 .

Noriega C, López J, Domínguez R, Velasco C. Perceptions of grandparents who provide auxiliary care: value transmission and child-rearing practices. Child Fam Soc Work. 2017;22(3). https://doi.org/10.1111/cfs.12339 .

Gleeson JP, Hsieh C, ming, Cryer-Coupet Q. Social support, family competence, and informal kinship caregiver parenting stress: the mediating and moderating effects of family resources. Child Youth Serv Rev. 2016;67. https://doi.org/10.1016/j.childyouth.2016.05.012 .

Di Gessa G, Glaser K, Tinker A. The Health Impact of Intensive and Nonintensive Grandchild Care in Europe: new evidence from SHARE. Journals Gerontol Ser B Psychol Sci Soc Sci. 2016;71(5):867–79. https://doi.org/10.1093/geronb/gbv055 .

van Broese MI, De Boer A. Providing informal care in a changing society. Eur J Ageing. 2016;13(3):271–9. https://doi.org/10.1007/s10433-016-0370-7 .

Komonpaisarn T, Loichinger E. Providing regular care for grandchildren in Thailand: an analysis of the impact on grandparents’ health. Soc Sci Med. 2019;229. https://doi.org/10.1016/j.socscimed.2018.05.031 .

Perry G, Daly M. Grandparental partnership status and its effects on caring for grandchildren in Europe. Helle S, editor. PLoS One. 2021;16(3):e0248915. https://doi.org/10.1371/journal.pone.0248915 .

Notter IR. Grandchild Care and Well-Being: Gender Differences in Mental Health Effects of Caregiving Grandparents. Carr DS, editor. Journals Gerontol Ser B. 2021; https://doi.org/10.1093/geronb/gbab164 .

Lee Y, Blitz LV. We’re GRAND: a qualitative design and development pilot project addressing the needs and strengths of grandparents raising grandchildren. Child Fam Soc Work. 2016;21(4):381–90. https://doi.org/10.1111/cfs.12153 .

Arpino B, Bordone V, Balbo N. Grandparenting, education and subjective well-being of older europeans. Eur J Ageing. 2018;15(3):251–63. https://doi.org/10.1007/s10433-018-0467-2 .

Article   PubMed   PubMed Central   Google Scholar  

Taylor MF, Marquis R, Coall DA, Werner J. The enjoyment rewards of fulfilling a Custodial Grandparenting Role in the lives of Grandchildren removed from their parents’ care. Child Care Pract. 2018;24(1):92–110. https://doi.org/10.1080/13575279.2017.1297772 .

Matua GA, Van Der Wal DM. Differentiating between descriptive and interpretive phenomenological research approaches. Nurse Res. 2015;22(6):22–7. https://doi.org/10.7748/nr.22.6.22.e1344 .

Article   PubMed   Google Scholar  

Neubauer BE, Witkop CT, Varpio L. How phenomenology can help us learn from the experiences of others. Perspect Med Educ. 2019;8(2):90–7. https://doi.org/10.1007/S40037-019-0509-2 .

Norlyk A, Harder I. What makes a phenomenological study phenomenological? An analysis of peer-reviewed empirical nursing studies. Qual Health Res. 2010;20(3):420–31. https://doi.org/10.1177/1049732309357435 .

Moser A, Korstjens I, Series. Practical guidance to qualitative research. Part 3: Sampling, data collection and analysis. Eur J Gen Pract. 2018;24(1):9–18. https://doi.org/10.1080/13814788.2017.1375091 .

Gearing RE. Bracketing in research: a typology. Qual Health Res. 2004;14(10):1429–52.

Dowling M. From Husserl to Van Manen. A review of different phenomenological approaches. Int J Nurs Stud. 2007;44(1):131–42. https://doi.org/10.1016/J.IJNURSTU.2005.11.026 .

Carpenter C, Suto M. Qualitative research for occupational and physical therapists: a practical guide. 2008;185. ISBN: 978-1-405-14435-3.

Giorgi A. The phenomenological movement and research in the human sciences. Nurs Sci Q. 2005;18(1):75–82. https://doi.org/10.1177/0894318404272112 .

Giorgi A. The status of Husserlian phenomenology in caring research. Scand J Caring Sci. 2000;14(1):3–10. https://doi.org/10.1111/J.1471-6712.2000.TB00554.X .

Article   CAS   PubMed   Google Scholar  

Giorgi AP. Concerning the application of phenomenology to caring research. Scand J Caring Sci. 2000;14(1):11–5. https://doi.org/10.1111/J.1471-6712.2000.TB00555.X .

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Heal Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042 .

Sebele-Mpofu FY. Saturation controversy in qualitative research: complexities and underlying assumptions. A literature review. Cogent Soc Sci. 2020;6(1). https://doi.org/10.1080/23311886.2020.1838706 .

Mills J, Birks M. Qualitative Methodology: A Practical Guide. 1st ed. Qualitative Methodology: A Practical Guide. London: SAGE Publications Ltd.; 2014. 1–288 p.

Triadó C, Celdrán M, Conde-Sala JM, Montoro-Rodríguez J, Pinazo S, Villar F. Envejecimiento productivo: la provisión De Cuidados De Los abuelos a Los Nietos. Implicaciones para su salud y bienestar. Madrid: IMSERSO; 2008. pp. 1–90.

Google Scholar  

Giorgi A. The theory, practice, and evaluation of the phenomenological method as a qualitative research procedure. J Clin Psychol. 1997;28(2):235–60.

Korstjens I, Moser A, Series. Practical guidance to qualitative research. Part 4: trustworthiness and publishing. Eur J Gen Pract. 2018;24(1):120–4. https://doi.org/10.1080/13814788.2017.1375092 .

Badenes NY, López MT. Doble Dependencia:abuelos que cuiddan nietos en España. Rev Serv Soc. 2011;49:107–25.

Falkingham J, Evandrou M, Qin M, Vlachantoni A. Informal care provision across multiple generations in China. Ageing Soc. 2020;40(9):1978–2005. https://doi.org/10.1017/S0144686X19000369 .

Margolis R, Verdery AM. A cohort perspective on the demography of Grandparenthood: past, Present, and future changes in race and sex disparities in the United States. Demography. 2019;56(4):1495–518. https://doi.org/10.1007/s13524-019-00795-1 .

Lakomý M, Kreidl M. Full-time versus part-time employment: does it influence frequency of grandparental childcare? Eur J Ageing. 2015;12(4):321–31. https://doi.org/10.1007/s10433-015-0349-9 .

Huang Y, Zhao Q, Reynolds NR, Tang S, Yan F, Wang Y, et al. The experience of Grandparenting Young Children. J Gerontol Soc Work. 2022;65(3):305–19. https://doi.org/10.1080/01634372.2021.1965687 .

Chan ACY, Lee S-K, Zhang J, Banegas J, Marsalis S, Gewirtz AH. Intensity of Grandparent Caregiving, Health, and Well-Being in Cultural Context: A Systematic Review. Heyn PC, editor. Gerontologist. 2022; https://doi.org/10.1093/geront/gnac026/6530315 .

Bordone V, Arpino B, Aassve A. Patterns of grandparental child care across Europe: the role of the policy context and working mothers’ need. Ageing Soc. 2017;37(4):845–73. https://doi.org/10.1017/S0144686X1600009X .

Danielsbacka M, Tanskanen AO, Coall DA, Jokela M. Grandparental childcare, health and well-being in Europe: a within-individual investigation of longitudinal data. Soc Sci Med. 2019;230:194–203. https://doi.org/10.1016/j.socscimed.2019.03.031 .

Leeson GW. Global demographic change and grandparenthood. Contemp Soc Sci. 2018;13(2):145–58.

Hill L. Haudenosaunee grandmothers Caring for their Grandchildren: the process of assuming the Caregiving Role. J Gerontol Soc Work. 2016;59(4):281–95. https://doi.org/10.1080/01634372.2016.1214660 .

Hank K, Cavrini G, Di Gessa G, Tomassini C. What do we know about grandparents? Insights from current quantitative data and identification of future data needs. Eur J Ageing. 2018;15(3):225–35. https://doi.org/10.1007/s10433-018-0468-1 .

Download references

Acknowledgements

We thank Primary Care Management of Servicio Madrileño de Salud for allowing us to carry out this study. We would also like to thank all our fellow nurses for their help.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Department of Nursing, Faculty of Biomedical and Health Sciences, European University of Madrid, Madrid, Spain

José Miguel Mansilla-Domínguez, Ana María Recio-Vivas, Laura Lorenzo-Allegue, José Miguel Cachón-Pérez & Laura Esteban-Gonzalo

Departament of Nursing, Universidad Complutense de Madrid, Madrid, Spain

Laura Esteban-Gonzalo

Research Group of Humanities and Qualitative Research in Health Science, Department of Physical Therapy, Occupational Therapy, Physical Medicine and Rehabilitation, Universidad Rey Juan Carlos (Hum&QRinHS), Alcorcón, Spain

Domingo Palacios-Ceña

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have made substantial contributions to the paper: JMMD and DPC conception and design of the work; JMMD, DPC and JMCP the acquisition, analysis and interpretation of dataJMMD, ARV, LLA, JMCP, LEG and DPC have drafted the work or substantively revised it and have approved the submitted version.All authors have agreed both to be personally accountable for the author’s own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Laura Lorenzo-Allegue .

Ethics declarations

Consent for publication.

Not applicable.

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

All methods were carried out in accordance with the relevant guidelines and regulations. In order to carry out this study, approval was requested from the Research Commission in charge of guaranteeing the scientific quality of the research studies carried out at this center, and the execution of what is established in the Declaration of Helsinki in those cases in which the research requires the participation of people. The study protocol was approved by the Ethical Committee of the Universidad Rey Juan Carlos (code: 01/2015) and the Local Ethical Committee of South Research-Madrid Health Service (code: 08/2012). Informed consent was obtained from all participants involved in this study. The permission for oral informed consent was obtained from the approving ethics committee.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mansilla-Domínguez, J.M., Recio-Vivas, A.M., Lorenzo-Allegue, L. et al. The role of duty, gender and intergenerational care in grandmothers’ parenting of grandchildren: a phenomenological qualitative study. BMC Nurs 23 , 477 (2024). https://doi.org/10.1186/s12912-024-02151-0

Download citation

Received : 17 November 2023

Accepted : 05 July 2024

Published : 15 July 2024

DOI : https://doi.org/10.1186/s12912-024-02151-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Grandparents
• Gender role
• Qualitative research

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]