mixed method case study evaluation

This website may not work correctly because your browser is out of date. Please update your browser .

Introduction to mixed methods in impact evaluation

Resource link.

• Introduction to mixed methods in impact evaluation (PDF, 1.74MB)

This guide, written by Michael Bamberger for InterAction  outlines the elements of a mixed methods approach with particular reference to how it can be used in an impact evaluation.

"There is rarely a single evaluation methodology that can fully capture all of the complexities of how programs operate in the real world. Consequently, evaluators must find creative ways to combine different evaluation frameworks, tools and techniques—hence the growing interest in MM approaches. The unique feature of mixed methods approaches is that they seek to integrate social science disciplines with predominantly QUANT and predominantly QUAL approaches to theory, data collection and data analysis and interpretation.

Although many evaluators now routinely use a variety of methods, “What distinguishes mixed-method evaluation is the intentional or planned use of diverse methods for particular mixed-method purposes using particular mixed-method designs” (Greene 2005:255). Most commonly, methods of data collection are combined to make an evaluation MM, but it is also possible to combine conceptual frameworks, hypothesis development, data analysis, or frameworks for the interpretation of the evaluation findings." (Bamberger 2012)

• Part I. Why mixed methods? 3
• What is a mixed-methods impact evaluation design? 3
• The limitations of an exclusive reliance on QUANT or QUAL evaluation approaches 3
• The benefits of a mixed methods approach 4
• Part II. The mixed methods approach 9
• Four decisions for designing a mixed methods evaluation 9
• Applying MM approaches at each stage of the evaluation 13
• Part III. Application of mixed methods  designs 19
• Sampling strategies for QUANT and QUAL oriented MM evaluations 19
• Using mixed methods to evaluate complex interventions 21
• Assessing processes of behavioural change 24
• Part IV. Managing mixed methods evaluations 27
• Mixed methods designs require a special management approach 27
• Tips for resource-constrained NGOs to mobilize the expertise and resources required to conduct mixed-methods evaluations 30
• Part V. Case Studies Illustrating Different Applications of mixed Methods designs 32
• Annexes  (available at http://www.interaction.org/impact-evaluation-notes (archived link))
• Annex 1. Strengths and weaknesses of quantitative evaluation designs
• Annex 2. Strengths and weaknesses of qualitative evaluation designs
• Annex 3. Examples of evaluation designs at each point on the QUANT - QUAL continuum
• Annex 4. Characteristics of QUANT and QUAL approaches at different stages of the evaluation
• Annex 5. How QUANT and QUAL approaches complement each other at different stages of an evaluation
• Annex 6. Comparing random and purposive sampling methods
• Annex 7. A range of quantitative, qualitative and theory-based approaches for defining the counterfactual
• Annex 8. Strategies for reducing the costs of data collection and analysis
• Annex 9. Example of triangulation: comparing estimates of household income and poverty from different sources
• Annex 10. Case studies of MM evaluation designs with predominant QUANT, QUAL and balanced orientations
• Annex 11. How mixed methods can strengthen QUANT evaluation designs

Bamberger, M. InterAction, (2012).  Introduction to mixed methods in impact evaluation  (No. 3.). Retrieved from website: https://www.interaction.org/wp-content/uploads/2019/03/Mixed-Methods-in-Impact-Evaluation-English.pdf

'Introduction to mixed methods in impact evaluation' is referenced in:

Framework/guide.

• Rainbow Framework :  Combine qualitative and quantitative data
• Impact evaluation

Back to top

© 2022 BetterEvaluation. All right reserved.

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

70k Accesses

30 Citations

12 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

24 Mixed Methods Evaluation

Donna M. Mertens has authored, co-authored or edited over 15 books relate to research and evaluation methods and human rights, most recently Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods (2014, 4th ed.), Indigenous Pathways into Social Research (co-edited with Fiona Cram and Bagele Chilisa, 2013), Program Evaluation Theory and Practice (co-authored with Amy Wilson, 2012) and Transformative Research and Evaluation (2009). Her scholarly work focuses on the intersection of research and evaluation with social justice and human rights and is situated within the philosophical assumptions of the transformative paradigm. She was a professor in the Department of Education at Gallaudet University for 32 years where she taught MA and PhD hearing and deaf students in education, psychology, social work, audiology, administration, and international development. She has been invited to conduct professional development related to transformative mixed methods in many context by many agencies, e.g., the Australasian Evaluation Society in Australia and New Zealand, the African Evaluation Association in South Africa, Niger, and Ghana, the Grupo de Institutos, Fundações e Empresas in Brazil, the World Bank in India and Nepal, the Community of Evaluators in India and Nepal, the Sri Lankan Evaluation Association in Sri Lanka, and UN Women and Evaluation Partners in Kazakhstan. Mertens also served as the Editor for the Journal of Mixed Methods Research 2010-2014.

Michele Tarsilla (Ph.D.) is Vice-President of the Evaluation Capacity Development Group. Dr. Tarsilla has designed and implemented mixed methods evaluations in over 25 countries in sub-Saharan Africa and Latin America for the UN, the World Bank and a variety of other international agencies and national governments for over a decade. Building on his gender-responsive and participatory evaluation work as well his methodological versatility, Dr. Tarsilla has been committed to enhancing his clients’ formulation of evidence-informed and right-based policies and strategies throughout his career. He is currently serving as Chair of the International and Cross-Cultural Evaluation Group at the American Evaluation Association (AEA), Associate Editor of the Journal of Multidisciplinary Evaluation (JMDE) and Peer-Reviewer of the African and American Journals of Evaluation. Given his interest in reflecting upon evaluation current and future trends, Dr. Tarsilla has published in a variety of journals. More recently, he has been working on a book on Evaluation Capacity Development as well as a number of articles (and blogs) on the use of language of evaluation and the need for mixing methods to capture unintended and unexpected outcomes of international development interventions. A Fulbright Scholar at the Georgetown University School of Foreign Service, a graduate from the World Bank-sponsored International Program on Development Evaluation Training (IPDET), and a Doctor in Interdisciplinary Evaluation from Western Michigan University, Michele has been receiving and delivering trainings on mixed methods and impact evaluation for over 10 years.

• Published: 19 January 2016
• Cite Icon Cite
• Permissions Icon Permissions

Understanding mixed methods approaches in evaluation involves understanding the philosophical stances, theoretical perspectives, and practical strategies that have emerged in the evaluation world. Building on such realization, this chapter provides a critical overview of why and how mixed methods have been used in evaluation for decades. After clarifying the definition of mixed and multimethods in evaluation, the chapter identifies five main paradigms underlying the use of mixed methods in contemporary evaluation practice (postpositivist, constructivist, pragmatic, transformative, and dialectical pluralism). Next, real-world evaluations are used to illustrate the assumptions and beliefs of such paradigms across a variety of dimensions (axiology, ontology, epistemology, methodology). The chapter also looks forward to future directions for a more informed use of mixed methods in evaluation.

The use of evaluation has dramatically increased over the past two decades across a variety of sectors. In the political arena, policymakers—especially at a time of economic turbulence and fiscal austerity— are constantly pressured by their constituencies to account for the results and economic return of publicly funded programs. Similarly, in the context of social programs, nongovernmental organizations evaluate the effectiveness of their implementation strategies on a routine basis to learn how to serve their beneficiaries better and secure funding for sustaining their programs. Furthermore, project managers in the private sector use evaluation to understand what corrective actions need to be taken to enhance their marketing and sales strategies to boost their clients’ satisfaction and increase their profits.

Concurrent with the development and strengthening of the evaluation practice in a number of domains, a plethora of definitions of evaluation have been used to explain what evaluation is. Despite the undeniable popularity of the term evaluation since the late 1960s, common understandings of evaluation’s most typical attributes included how evaluative efforts differentiate themselves from apparently similar endeavors, such as research, monitoring, and auditing are missing. In order to fill such a definitional gap and set the stage for informed discussions on mixed methods evaluation (MME), the introductory section of this chapter first clarifies the concept of evaluation by using Fournier’s definition:

Evaluation is an applied inquiry process for collecting and synthetizing evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance, or quality of a program, product, person, policy, proposal or plan. Conclusions made in evaluation encompass both an empirical aspect and a normative aspect (judgment about the value of something). It is the value feature that distinguishes evaluation from other types of inquiry, such as basic science research, clinical epidemiology, investigative journalism, or public polling. (pp. 139–140)

In line with the classic definitional distinction between research and evaluation, evaluation (and therefore MME) is unique for two main reasons:

Evaluation fosters knowledge creation (as research does), but its primary goal (promoting institutional and organizational learning for accountability and programmatic improvement) is broader and somewhat more political than the one pursued by research. As such, evaluation may help identify the positive/negative and intended/unintended effects of implementing an innovative educational policy sponsored by the Department of Education in the United States or it may assist in gauging the extent to which an agricultural value chain program funded by a European foundation in Kenya could be scaled up in two neighboring countries.

Evaluation is more intentional about the use of its findings than research is, as it is primarily aimed at influencing program management or decision-making on a variety of issues based on value judgments that are rendered after comparing findings with preestablished benchmarks. As a result, evaluation may, among other sources, inform the future allocation of funding for social programs, the scaling-up of current health or energy policies, and/or the interruption of ineffective projects and programs in a myriad of sectors.

Definitions and Mixed Methods in Evaluation and Research

Now we have defined evaluation’s features and main purposes (as distinct from those of research), we next clarify the meaning of MME. However, we first must define mixed methods as generally conceived in relation to the definition of mixed methods research (MMR). A second definition pointing to the distinctive features of MME is then provided.

Defining Mixed Methods Research

The definition of MMR we use is based on criteria that manuscripts submitted to the Journal of Mixed Methods Research need to fulfill to be considered for publication. As such, MMR generally includes the following:

Data collection and analysis, as well as integration of findings and formulation of inferences, based on the use of both qualitative and quantitative approaches (i.e., grounded theory or interpretative and hypothesis testing or confirmatory) or methods (i.e., structured interviews or surveys coupled with focus group discussions and ethnography);

Explicit integration of the quantitative and qualitative aspects of the study in question such as the measurement of quantitative variables associated with specific outcomes of interest (e.g., change in income or employment rate among youth age 18–23 as a result of their participation in a vocational skills training) paired with a more in-depth description of the interactions between the trainers and the training participants coupled with a thorough scan of the environmental factors affecting the implementation of the program;

Discussion of how the study in question not only adds to the literature on MMR but also makes a contribution to a substantive area in the scholar’s field of inquiry.

Defining Mixed Methods Evaluation

The MMR definition provided in the previous section adequately describes an increasingly popular practice among contemporary researchers. However, as evaluation is characterized by a stronger political connotation than research is (other than accruing knowledge on a given phenomenon, evaluation informs program design and policymaking based on evaluative judgments), a more in-depth discussion is needed before presenting a more exhaustive definition of MME. In particular, some further clarifications on mixed methods is necessary to fully appreciate the distinctive features of MME.

Mixed methods are more than data collection methods

A definition of MME would not be exhaustive if it were applicable only to data collection methods (e.g., the development of a case study and the administration of a survey for the sake of triangulation). MME are also—more frequently than not—characterized by the combination of both qualitative and quantitative data analysis, as well as hypothesis development processes and conceptual and interpretative frameworks, for either specific ( Creswell & Plano Cark, 2007 ) or multiple ( Greene, Caracelli, & Graham, 1989 ) purposes. More broadly, MMEs are the expression of different philosophical stances ( Tashakkori & Teddlie, 1998 ) and, as such, they honor “emergent and transgressive methodologies, inclusion, and a coloring of epistemologies” ( Denzin, 2010 , p. 420), such as those of the different evaluation team members and/or the evaluation primary and secondary users ( Patton, 2008 ).

Mixed methods evaluation: Brief history

Following the publication of the first conceptual framework for MME designs in the late 1980s ( Greene, Caracelli, & Graham, 1989 ) and based on the increased awareness among practitioners of the relevance of mixing methods within the scope of evaluations, a number of important developments occurred in the evaluation world, including the following:

A groundbreaking New Directions in Evaluation issue devoted to mixed methods ( Greene & Caracelli, 1997 ) aimed at summarizing the advances made in the use of mixed methods in evaluation as of the late 1990s.

The User-Friendly Handbook for Mixed Method Evaluations , published by the National Science Foundation ( Frechtling & Sharp, 1997 ).

The first and second editions of the Handbook of Mixed Methods in Social and Behavioral Research ( Tashakkori & Teddlie 2003 , 2012).

The launch of a peer-reviewed publication specifically dedicated to mixed methods: the Journal of Mixed Methods Research (2007) .

The publication of second New Directions in Evaluation volume dedicated to mixed methods in 2013 that focused on the contribution of mixed methods to credibility of evidence ( Mertens & Hesse-Biber, 2013 ).

Concomitantly with the proliferation of these developments, a number of professional groups and communities of practice with a specific focus on mixed methods were established, including the following:

The Special Interest Group on Mixed Methods created within the American Educational Research Association in 2009.

The Topical Interest Group on mixed methods established within the American Evaluation Association in 2010.

The launch of the Mixed Methods International Association in 2013.

Similarly, due to the increased popularity of the topic both within and outside academia, a series of conferences centered on mixed methods were held around the world, such as the following:

The Mixed Methods International Conference in Cambridge and Leeds (United Kingdom) and Baltimore, Maryland, 2005–2012.

The Day in Mixed Methods, a full-day learning event organized before the start of the International Qualitative Inquiry Congress Conference in 2012 and 2013 at the University of Illinois, Urbana/Champaign.

The Mixed Methods International Research Association annual conference in Boston in 2014.

More recently, MME has received special attention within two communities. In the health sciences community, the National Institutes of Health published a guidebook on best practices for mixed methods research ( Creswell, Klassen, Plano Clark, & Clegg Smith, 2011 ) that has influenced the work of many evaluators working in public health both in the United States and overseas. In the international development community, the UK Department for International Development (DFID; Stern et al., 2012 ) published a working paper on broadening the range of designs and methods for impact evaluations, and Bamberger (2012) wrote a technical note on mixed methods in impact evaluation. More precisely, interest in MME (mainly for the sake of triangulation and a better understanding of implementation processes and root causes of observed phenomena) was recently spurred by two main factors: (a) the identification of the limitations and practical challenges associated with the conduct of randomized controlled trials (RCT) funded generously by a variety of international donors and (b) the realization that the net distinction between qualitative and quantitative is no longer apparent, due to the variety of data collection and analysis tools and techniques available to evaluators today, as confirmed by the authors of the DFID publication mentioned earlier:

Combining methods has also become easier because the clear distinctions between quantitative (variables) and qualitative methods (cases) have become blurred, with quantitative methods that are non-statistical and new forms of within-case analysis made easier by computer aided tools. (DFID, 2012, p. ii)

Relevance for mixed methods evaluation in theory and practice

The eclectic use of MME is closely linked with the variety of purposes (e.g., exploration, explanation, and triangulation) that the combination of qualitative and quantitative methods generally serves ( Greene, Caracelli, & Graham, 1989 ). As a result, evaluations featuring mixed methods designs attest to the diversity of philosophical frameworks, theoretical lenses, methodological choices, and practices found within the evaluation community. Within the realm of MME, multiple perspectives of what represents evidence and robustness of findings exist, also based on the level of stakeholder participation, as well as the type of disciplinary aspects and the nature of paradigmatic choices. Hence, MME operates at many different levels.

Dialogues around the use of mixed methods are inherently linked with consideration of the credibility of evidence and the criteria used to establish that credibility ( Mertens & Hesse-Biber, 2013 ). Evaluators turn to methodologists from the postpositivist, constructivist, pragmatic, and transformative paradigms for criteria that are associated with credibility and rigor from different philosophical paradigm stances ( Mertens & Wilson, 2012 ). However, far from being solely preoccupied with the robustness of the data on which their findings are based, mixed methods evaluators struggle with how to accurately represent the voices of the less powerful stakeholders from marginalized communities whose views have historically been missing or inaccurately presented in past evaluative endeavors. In response to the question that they often ask themselves about how the inclusion of marginalized communities in evaluation enhances the credibility of their findings, mixed methods evaluators have an embarrassment of riches in terms of the options available to them for justifying their methodological choices. However, such diversity and richness of methodologies poses some challenges to the evaluators wanting to benefit from them, including the need to work within and between standpoints or world perspectives that are in tension with each other.

The Mixed Methods Paradox and the Default Mixed Methods Syndrome

As a result of both the growing body of related specialized literature and the proliferation of professional groups presented in the earlier section, the use of mixed methods is a distinctive feature of many evaluations conducted over the past decade ( Mertens & Hesse-Biber, 2013 ). The effort to combine qualitative and quantitative methods to enhance the validity and credibility of evaluation findings through the use of mixed methods in evaluation represents a commendable endeavor. However, the unsystematic and unquestioned use of mixed methods, referred to here as default mixed methods , casts some doubts over the effectiveness of their application. As attested by several evaluations conducted in international development evaluations, for instance, mixed methods are often used during only one of the evaluation stages (data collection) and with the objective of simply meeting the requests made by those who commission the evaluation. Therefore, evaluation teams often use the same dominant approach (in most cases, a quantitative one) that they have been using consistently in their whole career and couple it with some ancillary qualitative data collection methods (e.g., focus group discussions) as if they were “throwing a few M&Ms onto the top of the ice cream to make it look pretty, and it might taste a little bit better” ( Kemp, 2001 , p. 23). In response to this externally induced demand for the use of mixed methods in evaluation (this is not directly equivalent to a more genuine MME), there is a need for a more shared and solid understanding of mixed methods. Here we provide a rationale for their use.

Why Use Mixed Methods Evaluation?

Traditionally, the most frequent argument provided for justifying the use of MME has been that combining qualitative and quantitative methods helps mitigate the limitations of the “stand-alone” uses of either one. More recently, the use of MME has been associated with broader objectives, including the mix of conceptual frameworks and paradigms. As such, MME are regarded as particularly instrumental for the following:

“reducing the risk of bias and offset the limitations of a single method and thus increase the robustness of the estimated counterfactual” ( Gertler, Martinez, Premand, Rawlings, & Vermeersch, 2011 , p. 119)

“generating a more complete, contextual, contingent and complex understanding of the phenomena of interest than would have a single-method study” ( Greene, 2005 , p. 410)

“achieving the best of each method while overcoming their unique deficiencies” ( Denzin, 1989 , p. 244)

“celebrating the equality of difference” ( McGee, 2003 , p. 135)

throwing “up interesting, often surprising and sometimes counterintuitive relationships and patterns” through inductive qualitative research, as well as ask ‘how much?’ and establish how confident we can be in these ‘working hypotheses’ through quantitative research ( Garbarino & Holland, 2009 , p. 11)

As attested by this broadening of the idea of “methods” and further developing our understandings of why mixed methods are and should be used in evaluation, MME represents an innovation in evaluation. Mixed method evaluation contributes to the definition and pursuit of new approaches characterized by a variety of features, such as expanded epistemological and conceptual frameworks, scope, focus, data analysis approaches, and target audiences (Box 24.1 ).

MME is geared toward the generation of general findings as well as the promotion of a more solid understanding and explanation of complexity (e.g., of both programs and policies in relation to the multifaceted contexts in which they are implemented) based on the principle that phenomena are often unpredictable and not explained by linear causal mechanisms.

MME enhances triangulation, that is, the use of findings from different data sources in order to address the same evaluation question more effectively: the robustness of evaluation conclusions does not depend exclusively on the degree of convergence of findings but also on the extent to which the evaluator documents the strategies used to mitigate the divergence of findings (including the verification of the divergent finding and the return to the field for further data collection).

MME informs the use of multiple data collection strategies and instruments that build on one another (e.g., a series of ethnographic case studies on households’ dietary patterns, farming practices, and expenditures would allow the identification of the items that could be included in a subsequent survey aimed at measuring changes in nutritional status as a result of a school feeding program).

MME facilitates different but important understandings of the same evaluand, thus providing a common platform to fully appreciate the contributions of different conceptual frameworks and hypotheses on how changes occur as a result of a given intervention (e.g., as result of the identification, through thematic analysis, of behavioral patterns among program beneficiaries, a new variable could be created and incorporated in a regression analysis, thus increasing the predictive power of the statistical model underlying the program being evaluated.

MME provides complementarity of qualitative and quantitative findings for a variety of purposes (e.g., some case studies, developed after a training follow-up survey, would allow explaining why female respondents were four times less likely than men to have found a job within nine months from the completion of the training program being evaluated), including “completion, enhancement or detailing a more significant whole” ( Bazeley & Kemp, 2012 , p. 58).

MME combines understanding of the breadth and depth of human experiences.

MME enhances the accuracy of findings by qualifying them based on spatial and temporal attributes of an intervention as well as characteristics of the population(s) affected by the program being evaluated.

MME promotes data transformation : numerical data are converted to narrative and vice versa for understanding and communication purposes.

MME promotes extreme case analysis: important cases of very high or poor performance are identified and investigated further through a different method;

MME promotes theory development and testing: first, by identifying what should go in a typology rather than just using an a priori theory-driven approach for theory generating purposes, and second, by identifying a theory testing approach, as a sort of mixed methods grounded theory.

MME promotes data merging: qualitative/categorical and quantitative variables are created and merged into the same data set and then used in the statistical analysis of one’s explanatory model.

MME helps open the so-called “black box” or “grey box” of a program ( Scriven, 1994 ), by combining the identification of the end results of an intervention with that of the implementation processes and the dynamics (cultural, economic, political, social) influencing the effects of an intervention.

MME promotes the use of different samples (both probabilistic and purposely selected) in order to complete one’s fuller addressing of the evaluation question.

When MME is conducted in a participatory manner, a plurality of stakeholders’ concerns (they capture nuances within the communities/individuals studied) and multiple interests are adequately addressed.

MME is characterized by a certain flexibility in combining and analyzing both quantitative and qualitative findings in order to improve the effectiveness of findings dissemination and provide a wider understanding and acceptance of the corresponding conclusions and recommendations while also enhancing the accountability of the overall evaluation endeavor

The remainder of this chapter explores the diverse philosophical frameworks that provide guidance in the use of mixed methods in evaluation, along with illustrative examples and critical analysis of mixed methods approaches in evaluation.

Multiple Philosophical and Theoretical Lenses in Evaluation

Alkin (2004) provided a picture of the theoretical roots for evaluation in North America in the first edition of Evaluation Roots through the use of a metaphor of a tree to depict three branches of evaluation theories (i.e., Methods, Use, and Values). The Methods branch was populated by those who held that rigor in method achieved by the use of RCTs would produce credible knowledge for evaluators. The Use branch theorists noted that evaluators could rigidly follow postpositivist-based rules, but the results of their studies would not make a difference if no one used them. Hence, their focus was on identifying the intended users and designing studies that would be viewed as credible by that constituency. The Values branch theorists emphasized the importance of context and multiple stakeholders’ constructions of reality as the pathway to creating knowledge that was credible. The first edition of Evaluation Roots has been described as providing a good beginning to the process of documenting the evolution of theories in the field of evaluation; however, it has been criticized for not being inclusive of evaluators outside of North America and not having representation of ethnic-minority or Indigenous evaluators’ theoretical perspectives ( Davidson, 2006 ; Mertens, 2015 ).

In the second edition of Evaluation Roots , Alkin (2013) used the same three branches to depict theorists in evaluation, and two additional chapters were added to be more inclusive of evaluation theorists around the globe. Stame’s (2013) chapter on a European evaluation theory tree mentions the growing need for mixed methods 1 as a tool to settle the tension between an “agricultural-botany” paradigm utilizing the experimental and mental testing traditions in psychology ( Parlett & Hamilton, 1977 ) and the illuminative evaluation paradigm ascribing to anthropology, psychiatry, and participant observation in sociology 2 . Borrowing from Kushner (2000), Stame also reminds the reader that a shift in evaluative thinking is needed from “documenting the program and the lives of the individuals in that context [to] documenting the lives and work of people and to use that as a context within which we read the significance and meaning of programs” (p. 359).

Similarly, Rogers and Davidson’s (2013) chapter on New Zealand and Australian evaluation roots mentions the relevance of combining methods in order to both capture the substantive differences of human experiences (often dictated by one’s own cultural background as is the case of Aboriginals and Torres Strait Islanders) and making evaluation more culturally responsive ( Cram, 2009 ; Davidson, 2010 ; Wehipeihana, Davidson, McKegg, & Shanker, 2010 ).

While the addition of authors from Europe, New Zealand, and Australia certainly widened the scope of the evaluation theorists represented, it did not address the voices of evaluators from marginalized communities and other social justice advocates who are making significant contributions to the field of evaluation. Therefore, Mertens (2012) and Mertens and Wilson (2012) added a fourth branch to the evaluation tree labeled Social Justice. Mertens and Wilson also questioned the metaphor of a tree with separate branches and suggested a metaphor based on ocean currents that have distinct identities while sharing a deep water conveyer belt for sharing ideas. We use this four-branch structure (whether seen as branches of a tree or river or currents in the oceans) to frame our discussion of bases for the use of mixed methods in evaluation. Our main thesis is as follows: Understanding mixed methods approaches in evaluation involves understanding the philosophical stances, theoretical perspectives, and practical strategies that have emerged in the evaluation world associated with advances in mixed methods.

Evaluators lived through what were called the paradigm wars when post-positivists argued with constructivists about which methods would result in the creation of credible evidence ( Mertens & Wilson, 2012 ). The arguments seemed on the surface to be about methods, but they were more fundamentally about the worldviews of the evaluators and their assumptions about axiology (ethics), ontology (reality), epistemology (knowledge), and methodology (systematic inquiry) ( Shadish, 1998 ). Greene (2012) identified one important lesson from the paradigm wars as follows:

One important lesson from the ensuing “great qualitative-quantitative debate” was the recognition—across all methodological divides—that knowledge about human affairs is partially affected by the lenses and stances of the knower, the human inquirer. Accepting the impossibility of positivist raw empiricism and purely objective knowledge about human action meant, in tandem, that social inquirers had to accept the inevitable presence of self as researcher in the knowledge generated by social inquiry. One’s philosophical or paradigmatic assumptions about social reality and knowledge—alongside one’s favorite theories, the sensibilities of one’s mentor, important life experiences, values, and beliefs—all matter to the methodological decisions an inquirer makes and to the interpretive weft and warp of the knowledge attained. (p. 756)

Based on an expanded version of Guba and Lincoln’s (1989 , 2005) depiction of four paradigms (i.e., postpositivist, constructivist, pragmatism, and critical theory), evaluators are currently operating in a multiparadigmatic world that includes postpositivists, constructivists, pragmatists, and transformativists ( Mertens, 2009 , 2015 ; Mertens & Wilson, 2012 ). Any of these paradigms individually or in combination with each other can be used as a basis for the selection of mixed methods in evaluation. Hence, we use these paradigms and branches as the organizing framework for the next section.

Paradigms and Branches

The four major paradigms align with the four branches in evaluation theory, as depicted in Table 24.1 . Postpositivists align with Methods theorists, constructivists with Values theorists, transformativists with Social Justice theorists, and pragmatists with Use theorists. Theorists who choose to work across paradigms are not depicted in this table, however; these theorists are discussed in a later section on dialectical pluralism (DP).

Postpositivist Paradigm and the Methods Branch of Evaluation

Table 24.2 depicts the philosophical assumptions associated with the postpositivist paradigm. As the ontological and epistemological assumptions lead to a methodological assumption that prioritizes the use of RCT, one might wonder how mixed methods fit within this branch. Howard White’s 3 (2013) recent work tackles this issue and clarifies that the key question asked in impact evaluations (i.e., what difference did the intervention make?) can best be answered with an RCT or quasi-experimental design (if an RCT is not feasible). However, he also admits that impact evaluations can benefit from the addition of qualitative approaches to obtain answers to different types of questions (e.g., What are root causes of the observed changes? What is the quality of implementation? How relevant is the targeting of the intervention being evaluated? What are the barriers to either the adoption of a new behavior or the participation in the program in question?). Otherwise said, evaluations can be strengthened by adding qualitative data collection ( Bell, 2006 ; Briggs, 2006 ) to an essentially postpositivist study (RCT; Oakley 2005a , 2005b ). The postpositivist perspective remains one of the most popular in the research and evaluation community in North America and Europe. However, the fact that qualitative methods are treated as if they were “handmaiden or second best” to quantitative ( Hesse-Biber, 2010 , p. 457) has not failed to attract criticisms. In particular, often reproached for not having received advanced training in qualitative methodologies ( Howe, 2004 ), postpositivists have been accused of introducing an excessively quantitative lens and jargon in the evaluation community, thus “marginalizing the open-ended, free flowing, emergent nature of qualitative inquiry” and “leaving little space for issues connected to empowerment, social justice, and a politics of hope” ( Denzin, 2010 , p. 420).

Note . RCT = randomized control trials

The postpositivist paradigm in the real world: Examples

Given the many resources allocated to rigorous impact evaluations across a variety of sectors over the past decade, contemporary evaluation practice abounds with instances ascribed to the postpositivist paradigm, methods-based evaluations. Often suspected to be a sort of “marriage of interest” for either obtaining additional funding or responding to public concerns over the ethical issues associated with randomization (i.e., providing some individuals or communities with some goods and services while denying others access to those very same good and services, for the sake of rigor), the combination of qualitative and quantitative methods has gained increased popularity across a variety of sectors, especially within the international development arena, as illustrated by the following examples.

First, qualitative methods are increasingly used in impact evaluations conducted around the world by the Abdul Latif Jameel Poverty Action Lab affiliated with the Massachusetts Institute of Technology. In the course of over 100 RCT and quasi-experimental impact evaluations conducted over the past decade and based on the realization that findings of purely quantitative evaluations left key policy and programmatic questions unanswered, a certain number of Massachusetts Institute of Technology economists and quantitative evaluators started using (albeit marginally) qualitative methods. Traditionally skeptical about the utility of using nonstatistical methods for research and evaluation purposes, a large number of Jameel Poverty Action Lab impact evaluations in the mid-2000s started using qualitative methods, primarily as a means to identify and explain both the processes and the underlying causes of the impact(s) detected through statistical analysis (e.g., correlational and regression analysis). This is the case of the impact evaluation of the GoBifo 4 community development project funded by the World Bank in Sierra Leone 5 ( Casey, Glennerster, & Miguel, 2011 ). In the course of this evaluation, the organization of focus groups along with the conversion of the corresponding qualitative findings to variables and numbers—which served as the basis for the development of follow-up surveys—proved to be particularly instrumental in explaining social dynamics within the context of the villages targeted by the project.

Second, an illustrative example of this shift in thinking about methods within agencies that favored quantitative methods for decades is the recent hiring of a qualitative evaluator in the Gender Innovation Lab, a new office created within the Poverty Reduction and Economic Management unit of the African Region at the World Bank. Mandated to conduct 12 impact evaluations in the areas of agricultural productivity, entrepreneurship and employment, property rights, assets, and agency in 2012–2015, 6 the newly hired evaluator assists the team in the development of qualitative studies and integration of process- and context-related questions into existing quantitative surveys for the purposes of assessing the impact of development interventions more effectively. Based on the realization that quantitative studies had not highlighted how to provide women with consistent access to economic inputs in the past, the Gender Innovation Lab team acknowledged the critical role that qualitative methods play in the identification of possible solutions. 7

Third, a variety of predominantly quantitative evaluations recently conducted exemplify some of the key tenets of the postpositivist paradigm. These include, as attested in a recent publication on the use of mixed methods in impact evaluation ( Bamberger, 2012 ), (a) the impact evaluation of postconflict reconstruction in Liberia, funded by DFID and the International Rescue Committee; (b) the evaluation of a conditional cash transfer program in Kazakhstan, funded by Save the Children; and (c) the impact evaluation of Food and Agriculture Organization of the United Nations emergency and rehabilitation work in rural Democratic Republic of Congo.

Thus the postpositivist paradigm and its associated Methods branch exemplify quantitatively dominant mixed methods designs in which the qualitative component is used to gain insights and understandings related to the phenomenon under study. The Methods branch examples reflect the assumption that a quantitative measure could be used to capture the “reality” of the impact of a program and that evaluators, for the most part, should be objective and neutral in their stance. In the next section, we discuss the assumptions of the constructivist paradigm and how they align with the Values branch of evaluation.

Constructivist Paradigm and Values Branch of Evaluation

As mentioned earlier in the chapter, constructivist evaluators reject the ontological assumption that there is one reality out there waiting to be measured in favor of an ontology that holds that realities are multiple and socially constructed. (See Table 24.3 for a depiction of the philosophical assumptions associated with the constructivist paradigm.) This constructivist assumption leads to an emphasis on the use of qualitative methods because these allow the evaluator to interact with diverse members of the stakeholder community and together create the different versions of social reality needed to fully understand the context and the process of implementing an intervention. Even in the 1980s, constructivists acknowledged that quantitative data could be included as part of a primarily qualitative study ( Guba & Lincoln, 1989 ). Hence, the door has been and continues to be open to consider the character of mixed methods studies within the constructivist paradigm.

Denzin (2012) suggests that mixed methods studies that are rooted in the constructivist paradigm have greater potential to produce findings that can address the social good than can mixed methods approaches more closely aligned with the postpositivist paradigm. Mixed methods that begin in the constructivist paradigm afford the opportunity to assess the interpretive, contextual level of experience where meaning is created and provides a roadmap to address social justice. Constructivists view systematic inquiry as “an interactive process shaped by personal history, biography, gender, social class, race, and ethnicity of the people in the setting (p. 85).” Denzin proposes a moratorium on “mixed methods talk about designs and typologies and get back to the task at hand, which is changing the world” (p. 85). His position is that investigators need to employ constructivist strategies in order to create texts that challenge and stimulate readers to action. Denzin strongly argues for the use of the constructivist paradigm as the framework for social justice oriented research, but there is a large part of the constructivist community that does not embrace this as the primary goal of their work ( Merriam, 2009 ). Many constructivists have long claimed that their work should be descriptive and interpretive rather than activist. The transformative paradigm and Social Justice branch is discussed in the next section; this is a stance that has pursuit of social justice as one of its primary assumptions.

Hesse-Biber (2013) provides additional food for thought as to how a constructivist lens, informed by critical theory, can be used in combination with quantitative approaches, such as RCTs. In contrast to White’s (2013) view of the role of qualitative methods in evaluation, Hesse-Biber supports a stronger role for the qualitative aspects of a MME study. She argues that qualitative approaches can be used more extensively to answer such questions as

How well does the intervention respond to the culture and context of the target population?

How well do recruitment procedures work?

To what extent does the target population reflect the range of diversity with regard to the overall goals of the project? Who is left out? Why?

To what extent is ethics praxis (e.g., cultural responsiveness, attention to power inequities) built into the recruitment and evaluation process?

How well does the target population understand what they are consenting to?

To what extent do participants accept the outcome/s of randomization? Are participants willing to be randomized?

She argues that a RCT study would be enhanced by being enveloped within a subjectivist framework. This raises the question: Is it possible that qualitatively framed mixed methods are better suited to the ability of mixed methods researchers to demonstrate a causal relationship between variables? ( Mertens & Hesse-Biber, 2012 ).

The constructivist paradigm in the real world: Examples

A growing body of impact evaluations funded by DFID and other donors in northern Europe (Swedish International Development Agency and Danish International Development Agency) has given renewed prominence to the constructivist paradigm over the past few years. The idea that qualitative methods help represent the plurality of stakeholders’ perspectives on a variety of phenomena of interest was certainly a response to the frustrations and limitations associated with the very finite RCT efforts to measure linear causal links between interventions and effects. Specialized literature seemed to be particularly instrumental in the revitalization of this paradigm and the reaffirmation that qualitative methods ought to be included in the causal analysis toolbox ( Brady, 2002 ; Yin 2003 ; Pawson, 2007 ).

Knigge and Cope (2006 ; cited in Fielding, 2012 ) illustrate a qualitatively driven approach in their evaluation of community gardens established in a deprived area of Buffalo, New York, to determine their contribution to social capital accumulation. They relied heavily on the use of qualitative methods and modern technologies. Thanks to the use of web-based multimedia environment supported by GIS, the evaluation team first created a map showing community resources and facilities in proximity to the community gardens. Then a visual representation of the community garden blocks was enriched by additional information on ethnicity and land attributes as well as photos of neighborhoods and audio comments by local residents. Based on the specifications provided by the local residents, the evaluation planners agreed to redraw their map to better reflect the reality revealed by the multimedia representation. The methods included the conduct of interviews and observations, including bicycle rides in the target area as a way to gauge participants’ opinions about the value of the community gardens project. The evaluation team leader, Knigge, added labor market statistics to the GIS and used the original map’s counts of vacant land parcels to show the association of community gardens with adjacent house values. As Fielding (2012) reported in his review of data integration with new technologies:

Knigge realized that by solely looking at published quantitative data, she “may have missed the existence of community gardens, and a wholly ethnographic study might have missed potential correlations and clusters that were best analyzed through GIS” ( Knigge & Cope, 2006 , p. 2934, cited in Fielding, 2012 , p. 7).

A second illustrative example of the constructivist paradigm in evaluation is the impact evaluation of the India Gram Panchayat Community Development Reform Program ( Bamberger, 2012 ). Following the random assignment of 200 villages (Gram Panchayats) to project and control groups (quantitative conceptual framework), the evaluation relied on exploratory study of land tenure, ownership of public goods, participation, and social networks (qualitative data collection) that informed the content of the baseline survey administered prior to the start of training programs (quantitative data collection). These first few activities were followed by qualitative data collection in five project and five control areas to describe the processes of behavior changes (e.g., new community organization strategies, type and variety of community development projects, extent of women’s and scheduled castes’ participation). The evaluation, including field visits and community and households interviews, allowed uncovering the processes through which political and social dynamics, corruption, economic change, and network affiliation impacted on the project’s effectiveness, measured by the comparison between the effect of the projects before and after two years of implementation.

A third example of constructivism is a MME conducted in a largely rural state in India to examine women’s experiences with induced abortion, its determinants, and prevalence. The mixed methods approach included a combination of a quantitative survey that included open-ended questions, case studies, and focus group discussions about attempts and completed abortions, as well as women’s motivations and preferences regarding abortion, as influenced by underlying cultural factors. Edmeades et al. (2010) wrote the following:

The modified narrative approach significantly reduced underreporting of abortions, either through providing women with a safe space in which to discuss abortions or by improving women’s recall of past events. As a result, the data on abortions and the circumstances surrounding their use generated by this mixed methods approach are likely to be more representative of the totality of abortion experiences than those based on more conventional data collection approaches. This has the effect of increasing the validity and reliability of the conclusions reached based on the analyses of these data, both of which are of particular importance to policy makers and researchers. (p. 194)

The constructivist paradigm and its embodiment in the Values branch of evaluation is demonstrated in the sample MME studies in that the designs were not only qualitatively dominated but they also focused on constructing understandings of the participants’ realities through means that allowed for the complexity of their experiences to surface. Quantitative data also contributed to the quality of the MME, but they were couched within the knowledge gained through the qualitative methods. The next section discusses the pragmatic paradigm and the Use branch of evaluation; here the focus is on identifying the intended users of evaluation findings and designing the evaluation to meet their needs.

Pragmatic Paradigm and Use Branch of Evaluation

The pragmatic paradigm ( Morgan, 2007 ) supports the use of mixed methods based on the assumption that there is not one set of methods that is appropriate; rather, the criteria for choosing methods include what method fits with the evaluation questions. The philosophical assumptions associated with the pragmatic paradigm as it is sometimes depicted in the mixed methods literature appear in Table 24.4 .

Biesta (2010) , Hall (2009) , and Denzin (2012) warn against an overly simplistic application of the pragmatism as a philosophy in evaluation, basically suggesting, “If the method fits the question, then use it.” Biesta outlines the basic principles of Deweyan pragmatism as a philosophy that can inform mixed methods evaluators because Dewey held that no knowledge claim can be documented as providing the “truth.” Rather, different knowledge claims result from different ways of engaging with the social world.

Denzin (2012) asks: Have members of the mixed methods community done an injustice to pragmatism as a philosophical frame for mixed methods? Hall (2013) offers a more nuanced version of the pragmatic paradigm based on Dewey’s work and opens up possibilities for mixed methods evaluators. She supports the use of Deweyan pragmatism because it emphasizes reflection, ethics, and social justice and using systematic inquiry in the service of addressing societal problems by taking intelligent action. In order to take intelligent action, evaluators need to interact with the communities with which they work and be open to critical reflection about the meanings that are generated in terms of identification of problems and potential solutions. Solutions are not viewed as stagnant entities; rather, the evaluator has a responsibility to continually collect data on the consequences of the solutions and to interact with community members in an ongoing way to insure that adaptations are made to enhance success. Thus collection of evidence needs to be ongoing, responsive, and multifaceted. This provides the rationale for the use of both quantitative and qualitative methods in pragmatically rooted evaluations. Dewey emphasizes the importance of the consequences of action on the lives of people in a democratic sense (i.e., programs should enhance the lives of all stakeholders, both the privileged and members of marginalized communities).

The pragmatic paradigm in the real world: Examples

Robinson et al. (2011) provide an example of a mixed methods study that aligns with the assumptions of the pragmatic paradigm. The study was designed to evaluate the needs and experiences of family members who provided care for their relatives who had dementia. The evaluators justified their choice of mixed methods in this way:

We judged our objective measures of dementia care recipients and caregivers could be fully meaningful only if considered in the qualitative context of the family carers’ lives. This exemplifies why a mixed methods approach is fitting to dementia research: complex research problems warrant a complex research approach. Dementia is a profoundly complicated condition and mixed methods allow for a diverse range of means by which to address the problem. (p. 311)

They specifically mention that they chose the pragmatic paradigm because, according to Morgan (2007) , this provides a basis for using quantitative and qualitative methods without needing to give attention to metaphysical concerns such as ontology and epistemology. They also describe the pragmatic paradigm as allowing them to move back and forth between the deductive, quantitative aspects and the inductive, qualitative aspects of their study.

A second example of the pragmatic paradigm is the multilevel multiple methods evaluation of a microloan program managed by a Canadian credit union and primarily targeting unemployed individuals interested in either establishing or expanding their small businesses (Jackson & Tarsilla, 2012, Tarsilla, 2010 ). Several methods were used to address questions on both the effectiveness of the program (e.g., To what extent did the targeted borrowers strengthen their business as a result of their participation in the program? To what extent did the distributed microloans contribute to the well-being of the borrowers’ households?) and implementation processes (e.g., What aspects of the bank–borrower relationship could be strengthened in the near future? What other products would prove to be useful to the borrowers? How could the provincial government assist similar initiatives in the future?). Data collection took place at three different levels: (a) micro level, (the microloan borrowers and their households); (b) meso level (credit union staff); and (c) macro level (provincial and federal government staff). Data collection methods included the following:

Key-informant interviews with staff from the credit union distributing the microloans and Carleton University faculty members with expertise in community economic development and social return on investment.

Review of documents on micro-finance and community economic development in Canada and around the world (including audits and annual reports conducted in the past).

Semistructured questionnaire and focus groups: used to develop a semistructured questionnaire with questions on key outcome indicators (employment, credit status, welfare assistance, etc.).

Online, semistructured survey consisting of both closed-ended and open-ended questions.

A second virtual discussion group to validate some of the preliminary findings.

Case studies: a total of 10 (7 borrowers who had paid back their loan and 3 who had defaulted).

Expanded value added statement calculations ( Mook, Quarter, & Richmond, 2007 ): to quantify the “hidden” value of nonmonetary contributions from the program (e.g., the provision of free financial literacy sessions and earned articles in the local media).

A third example of this paradigm is the evaluation of children’s socialization and movement patterns in the center of an urban center in Denmark. As part of this evaluation, the team combined ethnographic fieldwork with global positioning system technology and used an interactive questionnaire that children completed via mobile phone ( Christensen, Mikkelsen, Sick Nielsen, & Harder, 2011 ). The innovative methodology adopted for this specific evaluative effort highlighted children’s subjective experiences with systematic observations, mapping, and survey data. As the evaluation team stated,

We were brought together by an almost coincidental recognition: two researchers, from transportation and mobility studies, experienced the need for qualitative social science input to understand the subjective and contextual basis for people’s movements, and two anthropologists needed to understand the broader patterning of children’s everyday mobility. In this study, therefore, the value of a pragmatic approach that accentuates the flexibility of methods and techniques, and interdisciplinary collaboration, led to the development of an integrated research design unifying our efforts in a holistic approach to understand the everyday mobility patterns of children. (p. 230)

The last example of the pragmatic paradigm is the impact evaluation of integrated community deliberative processes conducted by the National Democratic Institute (in press). The use of survey data collection instruments aimed at gauging changes in local government officials’ opinion, knowledge, and attitudes about governance mechanisms was combined with the use of more in-depth interviews with random samples of community members. Qualitative methods were used within the scope of this rigorous impact evaluation in order to assess any unexpected impact and/or identify environmental factors hindering the project effectiveness. The results were quite interesting: while the survey responses suggested that the program was attaining the expected objectives, the qualitative interviews and focus groups highlighted that government officials’ corruption and the unjustified land seizure occurring within communities severely hampered the success of the project and significantly questioned the democratic features of the deliberative processes implemented as part of the project.

The preceding examples illustrate the assumptions of the pragmatic paradigm and the Use branch of evaluation in that the primary goal of the studies is to enhance the use of the study findings by the targeted intended users. While some of the studies engaged with participants at the community level, most of the emphasis was on use by those in power. The final paradigm and branch of evaluation, the transformative paradigm and Social Justice branch, focuses on use to enhance the lives of the most marginalized members of society.

Transformative Paradigm and the Social Justice Branch of Evaluation

The transformative paradigm ( Mertens, 2009 ; Mertens & Wilson, 2012 ) philosophical assumptions emanate from an ethical stance that emphasizes the pursuit of social justice and the furtherance of human rights. 8 The evaluator derives implications for the nature of reality, knowledge, and systematic inquiry that are commensurate with this primary ethical assumption. Hence the nature of reality is looked upon as being multifaceted and reflective of different power positionalities in society. A summary of the transformative philosophical assumptions is presented in Table 24.5 .

The transformative axiological assumption prioritizes the values of equal human rights and social justice for all and leads evaluators to ask such questions as: If I start my evaluation with the goal of enhancing human rights and furthering social justice, what does this mean about how I view reality, knowledge, and systematic inquiry? How should this stance influence my philosophical assumptions and my practice as an evaluator? One immediate concern that arises from this primary ethical stance is the need to understand the culture of the community with all its complexity, including issues of power differentials associated with different social, cultural, and economic positionalities. Other issues include the importance of seeing the strengths within the communities and arranging conditions so members of teams are appreciated for the strengths they bring to the study. In addition, evaluators need to design their studies in ways that allow for reciprocity. What do the members of the community get out of the evaluation?

The transformative axiological assumption leads to consideration of the ontological and epistemological assumptions associated with this paradigm. The evaluator needs to engage in data collection in culturally respectful ways that allow for multiple versions of reality to emerge (ontology) and that build relationships of trust (epistemology). Thus mixed methods are not a requirement for doing transformative evaluations; however, they generally provide the opportunity to understand communities in their full complexity better than monomethods do.

In the transformative paradigm, reality is viewed as being constructed on the basis of different social and cultural positions, such as those related to gender, race, ethnicity, poverty, disability, deafness, religion, age, indigeneity, immigration status, and sexual identity. The evaluator has a responsibility to reveal the different versions of reality, exposing their source and the consequences of privileging one version of reality over another. Different versions of reality emerge from different positions in accord with the earned and unearned privileges associated with occupying different positions within the aforementioned dimensions of diversity. For example, hearing people might have a version of reality that holds that being hearing is better than being deaf. This version of reality contrasts with a deaf version of reality that holds that being deaf means one is a member of a culturally linguistic minority group. If the hearing version of reality is given privilege, this can result in denial of rights for the deaf people in terms of their use of a visual language, such as American Sign Language or British Sign Language. This is exactly what happened in the 1800s at the International Congress on the Education of the Deaf when hearing people decided that an oral approach to communication that requires lip reading and speech was the best method to educate deaf people. The consequence of accepting the hearing version of reality is that, for over a century, 9 deaf people around the world were denied access to a visual language in their education. Is it any wonder that deaf students graduating from high school in the United States read at a sixth-grade level?

Evaluators need to be cognizant of the versions of reality that are associated with unearned privileges within the communities in which they work, and they need to make visible those versions of reality that sustain an oppressive status quo and those that enhance the possibility of furthering human rights. Mixed methods provide a mechanism for revealing different versions of reality, their sources, and the consequences of privileging one version of reality over another. For example, evaluation of court access for deaf and hard-of-hearing people in the United States used qualitative data collection in the form of focus groups with deaf people to reveal the range of different experiences they had with the courts ( Mertens, 2009 ). Their experiences were influenced by the degree of hearing loss (e.g., hard of hearing or deaf), their language use (e.g., American Sign Language, Mexican Sign Language, spoken English), and their ability to benefit from supportive technologies (e.g., hearing aids, cochlear implants, real time captioning). This diversity also needed to be reflected in the training for court personnel and in the quantitative evaluation based on the courts’ implementation of plans to enhance accessibility.

Epistemologically, transformative evaluators are aware of the power relations in the communities in which they work, and they are aware of the need to be in a constant state of learning about how to interact in culturally appropriate ways. The nature of knowledge within the transformative paradigm is inclusive of different ways of knowing that are inherent in different stakeholder groups. For example, Indigenous ways of knowing have been explored by Chilisa (2012) , Cram (2009) , and others in this volume (see Cram and Mertens). This can involve knowledge of a history of oppression, broken treaties, stealing land, and colonialism that might not seem relevant to a non-Indigenous evaluator in the present time. Yet this is knowledge that is carried by members of Indigenous communities that might overshadow the issues that the program developers and evaluators have in mind. Mixed methods in terms of engaging in appropriate cultural rituals or practices may be as important to obtaining good data as is the use of linguistically appropriate data collection instruments.

Methodologically, a cyclical design is recommended for systematic inquiry in order to provide opportunities for understanding the cultural complexity of the community, designing mechanisms for working together, gathering knowledge that already exists, and designing data collection and interventions that are viewed as legitimate and hopeful by members of the community. This can mean beginning with qualitative inquiry in the form of dialogue between and among the stakeholder groups. It can also mean collection of demographic and incidence data that are disaggregated by the relevant dimensions of diversity (e.g., gender, tribal affiliation, age, language). Such qualitative and quantitative data need to be brought to the community to provide guidance on interpretation and to design next steps forward.

The transformative paradigm in the real world: Examples

Bledsoe (2010) conducted a transformative mixed methods study of an obesity reduction program that was situated in a city in which there was high poverty and many members of ethnic and racial minority groups, as well as immigrants from Africa and Eastern Europe. When Bledsoe joined the development team, they told her that they knew the reason for the obesity in the school: it was poor self-esteem. So they designed a program to raise the students’ self-esteem. Bledsoe asked if they had data to support this assumption, but they did not. As a first step, she proceeded to use quantitative and qualitative methods to ascertain the demographic characteristics of the students and their perspectives about obesity. The findings revealed that the students felt good about being big, but they wanted to avoid health problems associated with being obese such as heart problems or diabetes. Thus a different type of program was needed than that which was first planned. Figure 24.1 illustrates the full transformative cyclical mixed methods design used. It shows that the evaluators emphasized the importance of understanding the culture of the community and the forces within the community that prevented healthy living choices. The design is cyclical in that information from each stage of the project was used to inform the next stage of the project. For example, once there was an understanding that certain barriers were challenging the students’ abilities to lose weight, quantitative site mapping was used to show the lack of access to fresh fruits and vegetables and safe places to exercise. The interventions included things like negotiating with neighborhood food vendors to increase selections of healthy foods, youth wearing pedometers during a dance to count the number of steps taken, and food festivals that demonstrated how to cook traditional foods in a healthier manner.

Sample obesity reduction transformative MME.

Another example of transformative MME is the work conducted by Chilisa (2005 , 2012 ) in Botswana. Selected to assess the impact of an intervention designed to reduce the spread of HIV/AIDS in Africa, she started questioning the project design and, through her data collection and analysis, was able to demonstrate the variety of misunderstanding about the culture and language of Botswana. As attested by both the survey and in-depth interviews with youth, Chilisa was able to show that power dynamics between men and women were such that a woman would never have been able to tell a man to use a condom. Likewise, she was able to emphasize that the use of health campaign messages in English was not effective, as those at higher risk of HIV transmission spoke only local languages.

Some additional interesting examples of transformative MME are provided by the growing body of equity-focused evaluations funded by several United Nations agencies over the past few years. In the case of the UNICEF long-term evaluation of the Tostan program (aimed to reduce female circumcision in villages in Senegal), for instance, mixed methods proved to be critical to determine the overall impact of the intervention. In particular, the use of a quantitative district household survey was combined with observations, focus groups and in-depth interviews aimed at assessing the program’s implementation process, the dynamics surrounding the villages’ commitment to abandon the practice of circumcision, and the perception of the program’s impact among the targeted women.

A similar example is the real-time evaluation of the humanitarian response to Pakistan’s 2009 displacement crisis ( Bamberger & Segone, 2012 ). Thanks to the use of mixed methods, the findings of this evaluation pointed to the fact that the overall humanitarian response in Pakistan had been planned in consultation only with village elders and male household heads (accounting for the majority of survey respondents and focus group participants) and that no attention had been devoted to the special needs of women and children in the poorest and most vulnerable families. The evaluation of the UNICEF Education Program in Timor-L’Este 2003–2009 is also a good illustration of the transformative use of mixed methods, and it confirmed how the combination of quantitative and qualitative methods, besides serving relevant equity purposes, is key to making up for the limited in-country access to quantitative data. A final transformative example is also the evaluation of the impact of social assistance on reducing child poverty and child social exclusion in Albania. In the course of this evaluation, the mining of national data set allowed the identification of different typologies of vulnerable groups that were then interviewed, thus providing policymakers with actionable recommendations on how to enhance national social safety net mechanisms.

Combining Paradigms: Dialectical Pluralism

A dialectical approach places assumptions, practices, understandings, standpoints, and findings from one or more studies emanating from different paradigmatic positions in conversation with each other ( Greene & Hall, 2010 ; Johnson, 2011 ). Greene (2012) explains the complex challenges that evaluators encounter in terms of context, values, and the role of inquiry in society. She notes that the different methodological traditions in the evaluation community address these challenges differently, and that, in turn, shapes

the knowledge generated from a study and the warrants for that knowledge. . . [She] presents a mixed methodological response to these challenges and a primary argument that a mixed methods approach offers dialogic opportunities to generate better understanding of important social phenomena precisely because it legitimizes and respects multiple responses to these critical issues and invites dialogue among them. (p. 755)

DP is a philosophical stance that allows evaluators to engage in MME while experiencing the tensions created when different paradigmatic stances are put in conversation with each other. Johnson and Stefurak (2013) describe DP as encompassing a pluralistic ontology and reliance on a dialectical, dialogical, and hermeneutical process where stakeholders continually interact with differences (e.g., in philosophy, paradigm, culture, methodology, method). Dialectical methods engage parties holding seemingly disparate philosophical and knowledge positions in meaningful dialogue using multiple social psychological, dialogical, and negotiation strategies. They identify the following characteristics of DP: inclusive/heterogeneous team construction, equal power, joint construction of judgment criteria/standards, and commitment to deliberative and transformative democracy. Dialectic pluralism can include observation, experience, experiment, and so on, with the expectation that multiple empirical data will not typically converge on a simple conclusion and will often provide divergent results that bring fuller understandings of phenomena. Most human phenomena are complex; hence, there are many perspectives, viewpoints, and smaller truths that can be stated about almost any phenomenon studied.

Dialectical pluralism paradigm in the real world: Examples

An illustrative example of a DP mixed method study is the evaluation of the US Department of Agriculture Personnel Management Demonstration Project ( Mark, Fekv, & Button, 1997 ). The purpose of this evaluation was to test the effectiveness of a new personnel management and human resources strategy aimed at improving the recruitment and selection processes for hiring federal employees (e.g., decentralization and simplification of the administrative tasks and responsibilities associated with the hiring procedures in two of the department’s agencies) within the agency. Primarily relying on quantitative data (e.g., review of information about personnel actions and applicants and hired individuals for each vacancy), the evaluation made extensive use of qualitative methods (a theory of change was also developed before the start of the evaluation, based on semistructured interviews and document reviews and a total of 96 follow-up visits at 45 sites). Several features of DP were identified in association with this evaluation. First, the divergence of findings between the survey responses (according to them, managers appreciated the added value of the new hiring system) and in-person interviews (according to them, managers were not satisfied with the new system). Such plurality of discordant voices among the end users of the new human resources strategy eventually led to the revision of the instrument and the replacement of the term hiring system (leading to confusion and misunderstanding) with a more precise label. Second, the interview system in place at the US Department of Agriculture allowed the evaluators to qualify the extent to which the relationship between the applicant and the human resources person influenced the level of employees’ satisfaction with the overall hiring experience (qualitative data analysis confirmed the relevance of such relationship whereas the quantitative analysis had erroneously led them to think that this was an “error variance”).

A second DP mixed methods example is the evaluation of masculinity ideology conducted among a variety of ethnic groups in South Africa ( Luyt, 2011 ). As part of this evaluation, a quantitative instrument (the MANI-II, a multidimensional measure of masculinity ideology) developed in 2001 was revised based on qualitative findings that recognized basic conceptual differences between cultural groups in understanding masculinity. Based on a thematic analysis of focus group transcripts, the qualitative findings of this evaluation pointed to the lack of equivalence in cross-cultural content across the three versions of the tool (English, Afrikaans, and Xhosa). Simultaneously, a factor analysis of the instrument was conducted to assess the cultural construct validity of the instrument. At the end of this quantitative data collection phase, a large number of responses was analyzed (434 Afrikaans, 890 English, and 273 Xhosa) and the coefficient of congruence (Tucker’s phi) was calculated. As the authors put it:

“small-group descriptive” qualitative data and “large- group normative” quantitative data facilitated thorough exploration into varied participant perspectives within and between cultural groups [. . .] Quantitative data are considered before qualitative data. Yet qualitative data are not considered merely supplementary in that they provide unique as well as supplementary information concerning possible revision. This lends some support for the notion that equal status designs are possible in terms of the sum of their individual contributions. ( Edmeades et al., 2010 , p. 190)

In summary, DP typically relies on inclusion of methods that are likely to provide different understandings and perspectives of the evaluand. Purposive construction of a participatory multiple-divergent stakeholder team (including those with the least power) is a key strategy to include difference of voices in the evaluation.

Future Issues in Mixed Methods Evaluation and Conclusions

Perhaps because evaluation is inherently situated “out in the world,” practitioners embraced the use of mixed methods before philosophical or theoretical frameworks were developed that explicitly addressed the meaning of mixing methods at various levels. Rather than just default mixed methods, evaluators have a rich array of options from the philosophical level to the level of practice to encourage the critical thinking that is necessary in these complex studies. Differences in opinions about the appropriate methodologies and credibility of evidence can be traced back to the philosophical assumptions held by evaluators (or their clients), as has been demonstrated in this chapter. However, in order for that to happen, a serious introspection—a sort of “de-disciplining” process ( Richardson, 2006 ) will first need to take place among those many evaluators who continue to strive for “objectivity” in their profession and, as such, dismiss any remote association between their own design or methodological choices and any specific, value-based understanding of both the evaluand and the reality that surrounds it.

Many questions emerge as we explore the terrain of mixed methods in evaluation from various perspectives. Evaluators with different philosophical frameworks will most likely respond to the challenges in the use of mixed methods differently. For example, evaluators need to address the timing of the mixing methods, the integration of data analysis, and dealing with divergent findings. Also, critical reflection is needed to decide when the use of mixed methods is the better strategy as opposed to the use of multiple methods belonging to only one of the two spheres (quantitative or qualitative).

Given the existing paradox between the infinite opportunities for combining methods and the rather uninventive mix of methods in the current evaluation practice, more MME capacity development programs are needed. Geared toward both those who commission and those who conduct evaluations, such learning events could take the form of either online trainings catering to virtual communities of practice (e.g., American Evaluation Association and EvalPartners webinars, Evaluation Capacity Development Group list-serve, Mymande), research methodologies and evaluation graduate courses ( Frels, Onwuegbuzie, Leech, & Collins, 2012 ) or professional international workshops (International Program in Development Evaluation Training and International Development Evaluation Association workshops).

Building on the radical distinction from MMR and the old paradigm consisting in the often unquestioned combination of data collection methods, practitioners conducting MME will need to broaden the variety of the conceptual frameworks underlying their endeavors, and, by fully embracing the political and systemic functionality of MME, they will need to make epistemological frameworks and their values more explicit in the methodology sections of their evaluation reports.

In conclusion, the popularity of MME increasing and more debates on the why’s and how’s of MME should take place among evaluation practitioners working in a variety of sectors. Given the dynamic nature of the scholarship, the pervasive involvement of stakeholders, and the rapid developments in mixed methods in recent history in the evaluation field, we expect that evaluators will continue to contribute to the advancement of understandings at multiple levels in the mixed methods community. However, contrary to what happened with mixed methods in research settings, one will need to resist the temptation of developing an excessive number of new MME typologies and theoretical constructs centered around this topic. Evaluators work in a complex reality with a need for responsiveness to political and other contextual variables. Mixed methods in evaluation need to reflect this complexity, the realities the communities in which they work, diversity, and contextual variations in order to provide effective results that can be used to improve decision making and promote the desired social and individual changes.

Suggested Websites

www.eval.org

The American Evaluation Association website contains a list of evaluation organizations across the world and a library of resources for evaluators.

www.interaction.org

Alliance of US-based international nongovernmental organizations who focus on disaster relief and sustainable development programs.

www.ioce.net /

An international partnership of regional and national evaluation organizations that supports capacity development across the globe.

www.mymande.org/

An interactive platform to share knowledge on country-led monitoring and evaluation (M&E) systems and learning resources worldwide

Discussion Questions

Which of the paradigms discussed in this chapter resonates with you the most? What reasons can you give for your choice?

If applicable, where would your firm and your clients be positioned in relation to the evaluation paradigms, branches, and MM approaches?

To what extent do you believe default mixed methods will dominate the discourse within the evaluation community?

Is there any trend in MM evaluation that you foresee emerging in the near future and that is not yet adequately captured by any of the paradigms presented in this chapter? Please qualify your response by drawing on your direct evaluation experience and/or your readings.

The use of systematic reviews together with narrative reviews advocated by a number of theorists ( Pettigrew & Roberts, 2006 ) is one of the most illustrative examples of mixed methods in European evaluation theory and practice.

One evaluation approach that seems to be well suited to overcome this decade-long tension is realist evaluation. By predicating the possibility to uncover the hidden mechanisms that influence the processes underlying processes (the black box) and assuming the existence of meso-regularities (middle-range theories) in the world’s phenomena, the authors that ascribe to such approaches propose an alternative to both the positivist and constructivist movements in evaluation.

Howard White was the executive director of the International Initiative on Impact Evaluation (3ie), a special program providing funding and capacity development for the conduct of impact evaluations in conjunction with qualitative studies and systematic reviews of international development policies, projects, and programs. His article titled “Will We Ever Learn?” published by the Center of Global Development in 2007 emphasized the relevance of impact evaluation and stressed the ancillary nature of qualitative methods to address questions on effectiveness and impact of international development interventions.

GoBifo means “move forward” in the Krio language.

The project provided block grants to communities to use toward building local public goods like schools, latrines, and grain-drying floors and/or sponsoring skills training and income-generating activities.

Unlike past impact evaluation efforts geared toward the measurement of gender inequalities within the scope of any given development intervention, this new generation of impact evaluation is aimed to identify, develop, and test innovative policy solutions to alleviate underlying gender constraints.

In the past, it was through qualitative methods (e.g., case studies and in-depth unstructured interviews) that a series of effective strategies addressing gender issues were identified: (a) the provision of extension for women on the crops that they are currently growing, (b) the delivery of extension closer to women’s home (in order to deal with their mobility and time constraints), (c) the combination of extension with subsidies for inputs (given the numerous credit constraints reported during the in-depth interviews), and (d) the training of influential women in how to grow higher value crops that are traditionally male dominated without altering the existing extension programs.

Sweetman, Badjee, and Creswell (2010) conducted a literature review on mixed methods studies conducted with a transformative lens. They identified a large number (272) of mixed methods studies, and the most popular form of methods integration was the sequential form, in which one data set extended or added to the other data set.

The International Congress on the Education of the Deaf finally apologized for this oppressive policy at their meeting in 2010.

Alkin, M. C. (Ed.). ( 2004 ). Evaluation roots: Tracing theorists’ views and influences . Thousand Oaks, CA: Sage.

Google Scholar

Google Preview

Alkin, M. C. (Ed.). ( 2013 ). Evaluation roots (2nd ed.). Thousand Oaks, CA: Sage.

Bamberger, M. (2012). Introduction to mixed methods in impact evaluation . Interaction Impact Evaluation Notes No. 3. Retrieved from http://www.interaction.org/document/guidance-note-3-introduction-mixed-methods-impact-evaluation

Bamberger, M. , & Segone, M. ( 2012 ). How to design and manage equity-focused evaluations . New York, NY: UNICEF.

Bazeley, P. , & Kemp, L. ( 2012 ). Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research.   Journal of Mixed Methods Research , 6 , 55–72.

Bell, V. (2006). The Cochrane qualitative and implementation methods group. Retrieved from http://cqim.cochrane.org/

Bledsoe, K. (2010, November). A transformative mixed methods evaluation of an obesity reduction program . Paper presented at the annual meeting of the American Evaluation Association, Anaheim, CA.

Brady, H. E. (2002, July). Models of causal inference: Going beyond the Neyman-Rubin-Holland theory . Paper presented at the annual meeting of the Political Methodology Group, University of Washington, Seattle. Retrieved from http://www.polmeth.wustl.edu/media/Paper/brady02.pdf

Briggs, J. (2006). Cochrane qualitative research methods group. The Cochrane Collaboration. Available from http://www.joannabriggs.org

Casey, K. , Glennerster, R. , & Miguel, E. ( 2011 ). The GoBifo Project evaluation report: Assessing the impacts of community driven development in Sierra Leone . New York, NY: World Bank.

Chilisa, B. ( 2005 ). Education research within postcolonial Africa: A critique of HIV/AIDS research in Botswana.   International Journal of Qualitative Studies in Education , 18 , 659–684.

Chilisa, B. ( 2012 ). Indigenous research methodologies . Thousand Oaks, CA: Sage.

Christensen, C. , Mikkelsen, M. R. , Sick Nielsen, T. A. , & Harder, H. ( 2011 ). Children, mobility, and space: Using GPS and mobile phone technologies in ethnographic research.   Journal of Mixed Methods Research , 5 , 227–246. doi:10.1177/1558689811406121

Cram, F. ( 2009 ). Maintaining Indigenous voices. In D. M. Mertens & P. E. Ginsberg (Eds.), The handbook of social research ethics (pp. 308–322). Thousand Oaks, CA: Sage.

Creswell, J. W. , Klassen, A. C. , Plano Clark, V. L. , & Clegg Smith, K. ( 2011 ). Best practices for mixed methods research in the health sciences . Bethesda, MD: National Institutes of Health. http://obssr.od.nih.gov/mixed_methods_research/pdf/Best_Practices_for_Mixed_Methods_Research.pdf

Creswell, J.W. & Plano Clark, V. L. ( 2007 ). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.

Davidson, E. J. ( 2006 ). Book review: Evaluation roots: Tracing theorists’ views and influences . American Journal of Evaluation , 2 , 273–276.

Davidson, E. J. ( 2010 ). “Process values” and “deep values” in evaluation.   Journal of Multidisciplinary Evaluation , 6 (13), 206–208.

Denzin, N. K. ( 1989 ). The research act: A theoretical introduction to sociological methods (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.

Denzin, N. K. ( 2010 ). Moments, mixed methods, and paradigm dialogs.   Qualitative Inquiry , 16 (6), 419–427. doi:10.1177/1077800410364608

Denzin, N. ( 2012 ). Triangulation 2.0.   Journal of Mixed Methods Research , 6 (2), 80–88.

Edmeades, J. , Nyblade, L. , Malhotra, A. , MacQuarrie, K. , Parasuraman, S. , & Walia, S. ( 2010 ). Methodological innovation in studying abortion in developing countries: A “narrative” quantitative survey in Madhya Pradesh, India.   Journal of Mixed Methods Research , 4 (3), 176–198.

Fournier, D. M. ( 2010 ). Evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 139–140). Thousand Oaks, CA: Sage.

Frechtling, J. , & Sharp, L. (Eds.). ( 1997 ). User-friendly handbook for mixed method evaluations . Arlington, VA: National Science Foundation. Retrieved from http://www.nsf.gov/pubs/1997/nsf97153/

Frels, R. K. , Onwuegbuzie, A. J. , Leech, N. L. , & Collins, K. M. T. ( 2012 ). Challenges to teaching mixed research courses.   The Journal of Effective Teaching , 12 (2), 23–44.

Garbarino, S. , & Holland, J. ( 2009 ). Quantitative and qualitative methods in impact evaluation and measuring results . (DFID Issue Paper). London, England: Department for International Development. Retrieved from http://www.gsdrc.org/docs/open/EIRS4.pdf

Gertler, P. , Martinez, S. , Premand, P. , Rawlings, L. B. , & Vermeersch C. M. J. ( 2011 ). Impact evaluation in practice . Washington, DC: World Bank.

Greene, J. C. ( 2005 ). Synthesis: A reprise on mixing methods. In T. S. Weisner (Ed.), Discovering successful pathways in children’s development: Mixed methods in the study of childhood and family life (pp. 405–420). Chicago, IL: University of Chicago Press.

Greene, J. ( 2012 ). Engaging critical issues in social inquiry by mixing methods.   American Behavioral Scientist , 56 (6), 755–773.

Greene, J. C. , & Caracelli, V. J. (Eds.). ( 1997 ). Advances in mixed-methods evaluation: The challenges and benefits of integrating diverse paradigms.   New Directions in Evaluation , 74 . San Francisco, CA: Jossey-Bass.

Greene, J. C. , Caracelli, V. J. , & Graham, W. F. ( 1989 ). Toward a conceptual framework for mixed-method evaluation design.   Education Evaluation and Policy Analysis , 11 (3), 255–274.

Guba, E. , & Lincoln, Y. S. ( 1989 ). Fourth generation evaluation . Newbury Park, CA: Sage.

Hall, J. ( 2009 ). Introduction to mixed methods research for nursing and the health sciences. In S. Andrew & E. J. Halcomb (Eds.), Mixed methods research for nursing and the health sciences (pp. 3–12). Oxford, UK: Blackwell.

Hesse-Biber, S. ( 2010 ). Qualitative approaches to mixed methods practice.   Qualitative Inquiry , 16 , 455–468.

Hesse-Biber, S. ( 2013 ). Thinking outside the randomized controlled trials experimental box: Strategies for enhancing credibility and social justice. In New Directions in Evaluation , 138 , 49–60.

Howe, K. R. ( 2004 ). A critique of experimentalism.   Qualitative Inquiry , 10 , 42–61.

Jackson, E. T. , & Tarsilla, M. ( 2013 ). Mixed methods in social accounting: Evaluating the micro-loan program of Alterna Savings Credit. In L. Mook (Ed.), Accounting for social value (pp. 117–138). Toronto: University of Toronto Press.

Johnson, R. B. (2011, May). Dialectical pluralism: A metaparadigm to help us hear and “combine” our valued differences . Paper presented at the Seventh International Congress of Qualitative Inquiry, University of Illinois, Urbana-Champaign.

Johnson, R. B. , & Stefurak, T. ( 2013 ). Dialectical pluralism: A process philosophy for understanding, valuing, and “dynamically combining” our differences.   New Directions in Evaluation , 138 , 37–48.

Kemp, L. (2001, May). The DNA of integrated methods . Paper presented at the annual conference of the Australian Association for Social Research, Wollongong, New South Wales, Australia.

Luyt, R. ( 2011 ). A framework for mixing methods in quantitative measurement development, validation, and revision: A case study.   Journal of Mixed Methods Research 6 (4), 294–316.

Mark, M. , Fekv, I. , & Button, S. B. ( 1997 ). Integrating qualitative methods in a predominantly quantitative evaluation: A case study and some reflections.   New Directions for Evaluation , 74 , 47–60.

Merriam, S. ( 2009 ). Qualitative research . San Francisco, CA: Jossey-Bass.

Mertens, D. M. ( 2009 ). Transformative research and evaluation . New York, NY: Guilford Press.

Mertens, D. M. ( 2015 ). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (4th ed.). Thousand Oaks, CA: Sage.

Mertens, D. M. , & Hesse-Biber, S. ( 2013 ). Mixed methods and credibility of evidence in evaluation.   New Directions in Evaluation , 138 , 5–14.

Mertens, D. M. , & Wilson, A. T. ( 2013 ). Program evaluation theory and practice: A comprehensive guide . New York, NY: Guilford Press.

McGee, R. ( 2003 ). Qualitative and quantitative poverty appraisal workshop: Some reflections and responses. In R. Kanbur (Ed.), Q-squared: Qualitative and quantitative methods of poverty appraisal (pp. 132–140). Delhi, India: Permanent Black.

Mook, L. , Quarter, J. , & Richmond, B. J. ( 2007 ). What counts: Social accounting for non-profits and cooperatives (2nd ed.). London, England: Sigel Press.

Morgan, D. L. ( 2007 ). Pardigms lost and pragmatism regained.   Journal of Mixed Methods Research , 1 (1), 48–76.

Oakley, A. ( 2005 a). Paradigm wars: Some thoughts on a personal and public trajectory. In A. Oakley , The Ann Oakley reader (pp. 245–250). Cambridge, UK: Polity.

Oakley, A. ( 2005 b). Who’s afraid of the randomized controlled trial? Some dilemmas of the scientific method and good research practice. In A. Oakley , The Ann Oakley reader (pp. 233–244). Cambridge, UK: Polity.

Parlett, M. , & Hamilton, D. ( 1977 ). Evaluation as illumination: A new approach to the study of innovatory programs. In D. Hamilton , D. Jenkins , C. King , B. MacDonald , & M. Parlett (Eds.), Beyond the numbers game: A reader in educational evaluation (pp. 6–22). London, England: Macmillan.

Patton, M. Q. ( 2008 ). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Pawson, R. (2007). Causality for beginners . Paper presented at the NCRM Research Methods Festival 2008. University of Leeds. Retrieved from http://eprints.ncrm.ac.uk/245/

Pettigrew, M. , & Roberts, H. ( 2006 ). Systematic reviews in the social sciences . Oxford, UK: Blackwell.

Richardson, V. ( 2006 ). Stewards of a field, stewards of an enterprise: The doctorate in education. In C. M. Golde & G. E. Walker (Eds.), Envisioning the future of doctoral education: Preparing stewards of the discipline . San Francisco, CA: Jossey-Bass.

Robinson, A. L. , Emden, C. G. , Croft, T. D. , Vosper, G. C. , Elder, J. A. , Siirling, C. & Vickers, J. C. ( 2011 ). Mixed methods data collection in dementia research: A “progressive engagement” approach.   Journal of Mixed Methods Research , 5 (4), 330–344.

Scriven, M. ( 1994 ). The fine line between evaluation and explanation.   Evaluation Practice , 15 (1), 75–77.

Shadish, W. ( 1998 ). Evaluation theory is who we are.   American Journal of Evaluation , 19 , 1–19.

Stame, N. ( 2013 ). A European evaluation theory tree. In M. Alkin (Ed.), Evaluation roots (2nd ed.) (pp. 355–370). Thousand Oaks, CA: Sage.

Stern, E. , Stame, N. , Mayne, J. , Forss, K. , Davies, R. , & Befani, B. ( 2012 ). Broadening the range of designs and methods for impact evaluations (DFID Working Paper 38). London, England: Department for International Development.

Sweetman, D. , Badjee, M. , & Creswell, J. W. ( 2010 ). Use of the transformative framework in mixed methods studies.   Qualitative Inquiry , 16 (6), 441–454. doi:10.1177/1077800410364610

Tarsilla, M. ( 2010 ). Social impact evaluation of the Alterna Savings Community micro-loan program . Ottawa, ON: Carleton Centre for Community Innovation, Carleton University.

Teddlie, C. , & Tashakkori, A. ( 2003 ). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3–50). Thousand Oaks, CA: Sage.

Wehipeihana, N. , Davidson, E. J. , McKegg, K. , & Shanker, V. ( 2010 ). What does it take to do evaluation in communities and cultural contexts other than our own?   Journal of Multidisciplinary Evaluation , 16 (3), 182–192.

White, H. ( 2013 ). The use of mixed methods in randomized controlled trials.   New Directions in Evaluation , 138 , 61–74.

Yin, R. K. ( 2003 ). Case study research: design and methods (3rd ed.). Thousand Oaks, CA: Sage.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Evidence Matters

Towards equitable, inclusive and sustainable development

Using mixed methods to strengthen process and impact evaluation

As necessary as conventional quantitative methods might be, the importance of adopting a mixed-methods approach in order to understand and answer complex development questions cannot be overemphasized. This blog is the second of a two-part series by 3ie Senior Research Fellow Michael Bamberger in which he offers detailed guidance on how to design, implement and utilize mixed-methods evaluations. Click here to read the first part in which he explains why an integrated mixed-methods approach  is more effective for addressing real-world problems.  

Mixed-methods approaches can be applied at all stages of an evaluation and not just during data collection. They seek to combine the strengths of both quantitative and qualitative evaluations – having different purposes and methods when used by researchers working within either of these dominant traditions. While there is no standard mixed-methods approach, a wide range of tools and techniques can be adapted to the specific context of each evaluation. Many mixed-methods evaluations will have three components: a qualitative, a quantitative, and a mixed-methods component that synthesizes the other two.

Designing, implementing, and utilizing mixed-methods evaluations requires the following key steps. (Figure 2.1)

Step 1:  Understanding the dominant approach of the researchers  

Mixed-methods approaches are used quite differently when used to complement a quantitative-dominant evaluation compared to a qualitative-dominant approach ( section 2.1 ):

• For quantitative-dominant evaluation (research), mixed methods are often used to provide a deeper understanding of the social and political context within which a program operates, the organizational processes, behavior, and motivations, and to explain why, how, and for whom intended outcomes were achieved (or not achieved).     
• For qualitative-dominant evaluations, mixed methods are often used to enhance the representativity and generalizability of the findings (how much change, for whom, and the statistical significance).   

Step 2:  Building the research team

Conducting a mixed-methods evaluation will often require the research team to broaden the range of research expertise. Over the short run, this often means sub-contracting the parts of the research with which the team is not familiar or contracting a consultant. Over the longer run, the research team may consider hiring staff with the required expertise or who can provide the capacity development required by current staff members. When bringing in consultants or sub-contracting parts of the study, it is important to incorporate the new research expertise early in the research planning to allow time for each group to become familiar the approach of the other and to understand how the research design must be adapted to fully benefit from the integrated approach. However, it is commonly the case that the evaluation is designed before the consultants are brought in and they are asked to implement a certain number of research activities that have already been finalized. While this latter scenario can be considered as a multi-method evaluation (with parallel research strands), it is not a true mixed-methods design and much of the potential synergy is lost.

Additionally, fully incorporating the new team members requires strong team leadership and time for team-building.

Step 3: Defining the research questions

For understanding the program being studied and the key research questions, four main approaches are combined under mixed methods. These include: a review of secondary data sources, factoring in experience from earlier projects, and interviews with stakeholders (and key informants). The fourth one – diagnostic studies – helps understand the economic, political, socio-cultural, historical, and other relevant dimensions of the broader context within which the program operates (or will operate).

Diagnostic studies is a key feature of the mixed-methods approach, which emphasizes that development issues are multi-dimensional and that different stakeholders have different perspectives, expectations, and priorities. Furthermore, it considers the interaction between sectors – so, for example, an education program will be influenced by the status of health, transport and infrastructure, and agriculture, among others. This approach involves spending significant amounts of time visiting the affected populations to understand their experiences and perspectives, the history of the community, and their expectations from and about the proposed project.

Step 4: Defining the research hypotheses

Quantitative research hypotheses are normally derived deductively from existing theories often supported by literature reviews. In contrast, qualitative hypotheses are usually derived inductively based on observations in the field ( section 2.2 ). So, while quantitative deductive hypotheses are usually defined at the start of the evaluation before data collection begins, qualitative inductive hypotheses evolve as data is collected and there is a better understanding of the issues being studied. A strength of the deductive approach is that the hypotheses are clearly defined at the start of the study so that sample design and data collection can be designed to permit rigorous testing of the hypothesis. The limitation is that the evaluation is implemented without a full understanding of the specific project context. The mixed-methods approach permits the use of a three-stage hypothesis development:

• Stage 1: diagnostic study to provide a deep understanding of the context and generate an inductive hypothesis that captures the unique characteristics of the program context ( section 2.3 )
• Stage 2: expanded deductive hypothesis that incorporates the in-depth understanding provided by the inductive hypothesis
• Stage 3:  updated inductive hypothesis that can be incorporated at a later point in the evaluation

Step 5:  The research design

5A Sample design

There are four sampling decisions in a mixed-methods design:

• Is the sample design random or purposive? ( section 2.4 )
• Is the study conducted at a single level (e.g., the community level) or at several levels (e.g., community health centers and district health offices)?
• Is the design sequential (quantitative and qualitative designs are used one after the other) or parallel (both used simultaneously) ( section 2.5 )? Sequential designs are the most common as they are logistically easier to manage (it is very difficult to simultaneously coordinate two different studies at the same time). However, triangulation during data collection, to test the reliability and validity of the data collection process, is an example of a parallel mixed-methods design.
• Is there a use for nested designs? ( section 2.6 )

5B Data collection methods

Mixed-methods studies normally combine conventional quantitative and qualitative data collection methods ( section 2.7 ) with the integration of both into mixed-methods data collection.

5C Enhancing data quality

A priority of mixed methods is to combine different tools to enhance data quality. This is done through triangulation and using techniques such as diagnostic studies, in-depth interviews, participant observation to understand the context within which indicators are used.

Step 6:  Data analysis and inference

Mixed-methods data analysis includes three components:

• Qualitative analysis: the main types are categorizing, contextualizing, theming and comparing, and displaying ( section 2.8 )  
• Quantitative analysis: the main types are descriptive versus inferential analysis, and parametric versus non-parametric
• Integrated mixed-methods analysis: where the above two are incorporated

The basic models of integrated mixed-methods analysis are:

• Parallel analysis: when the quantitative and qualitative data sets are analyzed independently and the findings of one do not affect the design of the other. In this case, the analysis of the linkages between the quantitative and qualitative findings is done at the interpretation stage
• Sequential analysis: when the findings of one part of the analysis determine the structure of the analysis of the other part ( section 2.9 )
• Conversion: when qualitative data is converted (“quantized”) into a metric that permits statistical analysis. One common approach is the use of rating scales ( section 2.10 ), another is the use of non-parametric statistics such as Chi-square.  Similarly, quantitative data is classified (“qualitized”) into descriptive categories that can be used to select case studies or for other descriptive purposes
• Outlier analysis is used to study cases falling outside of the normal pattern of responses ( section 2.11 )
• Multi-level mixed data analysis: when data is organized into levels (e.g., household, village school, school district).  The data is analyzed separately for each level and then patterns between levels are identified.
• Fully-integrated mixed-methods data analysis: when inherently mixed-methods analysis techniques (Tashakkori et al 2020) are used. These have been developed specifically for mixed-methods analysis ( section 2.12 )
• Cross-over analysis: when analytical techniques from one tradition are applied to the other [ Section 2.13 ] These kinds of analysis do not just refer to switching between quantitative and qualitative analysis, but also between different traditions within each approach.

The interpretation of the findings and inference from mixed-methods analysis combines the different approaches used for assessing threats to validity and adequacy of quantitative and qualitative data analysis (Bamberger and Mabry 2020 Chapter 7).

Step 7:  Presentation, dissemination and use of mixed-methods evaluations

A number of approaches to presentation, dissemination and use are emphasized in mixed methods evaluations, according to which an evaluation must capture the perspectives and voices of a wider range of stakeholders – often not the case in other evaluations. All these perspectives must be represented in the evaluation findings – which should reflect the multiple perspectives and not try to present a single perspective, usually of the funding agency or a government office. For example, while the construction of a road through a village may be estimated to have a positive economic rate of return, it may be negatively evaluated by some residents, particularly mothers, concerned for the safety of their children, who can no longer walk around the village on their own. The mixed-methods evaluation findings must include both perspectives. Before finalizing the report, appropriate mechanisms should be included to ensure feedback is obtained from a wide range of stakeholders, who will require the findings to be disseminated in different ways. While some may expect a formal written report, many community-level and indigenous groups may prefer a discussion in a community meeting or the use of posters, or in some cases community theater or dance. Finally, recommendations on utilization must take into consideration the broader political, economic, and socio-cultural environment.

Hi, A very informative blog. Came to know about a new terminology 'inductive hypothesis'. Would have been great if more clarity was provided on the term by providing any external link.

Add new comment

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Most popular blogs

Too difficult, too disruptive and too slow? Innovative approaches to common challenges in conducting humanitarian impact evaluations

Moving the debate forward on community-driven development

• impact evaluations
• systematic reviews
• 2020hindsight
• evidence-based policy

Related blogs

Importance of mixed-methods approaches in development research and evaluation

How qual improves quant in impact evaluations

Reflections on the impact of agricultural certification on well-being

Recent blogs.

Teaching critical thinking about health: impacts and implications

Insights from the Development Evidence Portal – the Middle East and North Africa

Strengthening Cote d'Ivoire's nutrition strategy with rapid evidence

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

Prevent plagiarism. Run a free check.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved April 8, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Language: English | German

How to Construct a Mixed Methods Research Design

Wie man ein mixed methods-forschungs-design konstruiert, judith schoonenboom.

1 Institut für Bildungswissenschaft, Universität Wien, Sensengasse 3a, 1090 Wien, Austria

R. Burke Johnson

2 Department of Professional Studies, University of South Alabama, UCOM 3700, 36688-0002 Mobile, AL USA

This article provides researchers with knowledge of how to design a high quality mixed methods research study. To design a mixed study, researchers must understand and carefully consider each of the dimensions of mixed methods design, and always keep an eye on the issue of validity. We explain the seven major design dimensions: purpose, theoretical drive, timing (simultaneity and dependency), point of integration, typological versus interactive design approaches, planned versus emergent design, and design complexity. There also are multiple secondary dimensions that need to be considered during the design process. We explain ten secondary dimensions of design to be considered for each research study. We also provide two case studies showing how the mixed designs were constructed.

Zusammenfassung

Der Beitrag gibt einen Überblick darüber, wie das Forschungsdesign bei Mixed Methods-Studien angelegt sein sollte. Um ein Mixed Methods-Forschungsdesign aufzustellen, müssen Forschende sorgfältig alle Dimensionen von Methodenkombinationen abwägen und von Anfang an auf die Güte und damit verbundene etwaige Probleme achten. Wir erklären und diskutieren die für Forschungsdesigns relevanten sieben Dimensionen von Methodenkombinationen: Untersuchungsziel, Rolle von Theorie im Forschungsprozess, Timing (Simultanität und Abhängigkeit), Schnittstellen, an denen Integration stattfindet, systematische vs. interaktive Design-Ansätze, geplante vs. emergente Designs und Komplexität des Designs. Es gibt außerdem zahlreiche sekundäre Dimensionen, die bei der Aufstellung des Forschungsdesigns berücksichtigt werden müssen, von denen wir zehn erklären. Der Beitrag schließt mit zwei Fallbeispielen ab, anhand derer konkret gezeigt wird, wie Mixed Methods-Forschungsdesigns aufgestellt werden können.

What is a mixed methods design?

This article addresses the process of selecting and constructing mixed methods research (MMR) designs. The word “design” has at least two distinct meanings in mixed methods research (Maxwell 2013 ). One meaning focuses on the process of design; in this meaning, design is often used as a verb. Someone can be engaged in designing a study (in German: “eine Studie konzipieren” or “eine Studie designen”). Another meaning is that of a product, namely the result of designing. The result of designing as a verb is a mixed methods design as a noun (in German: “das Forschungsdesign” or “Design”), as it has, for example, been described in a journal article. In mixed methods design, both meanings are relevant. To obtain a strong design as a product, one needs to carefully consider a number of rules for designing as an activity. Obeying these rules is not a guarantee of a strong design, but it does contribute to it. A mixed methods design is characterized by the combination of at least one qualitative and one quantitative research component. For the purpose of this article, we use the following definition of mixed methods research (Johnson et al. 2007 , p. 123):

Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e. g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration.

Mixed methods research (“Mixed Methods” or “MM”) is the sibling of multimethod research (“Methodenkombination”) in which either solely multiple qualitative approaches or solely multiple quantitative approaches are combined.

In a commonly used mixed methods notation system (Morse 1991 ), the components are indicated as qual and quan (or QUAL and QUAN to emphasize primacy), respectively, for qualitative and quantitative research. As discussed below, plus (+) signs refer to concurrent implementation of components (“gleichzeitige Durchführung der Teilstudien” or “paralleles Mixed Methods-Design”) and arrows (→) refer to sequential implementation (“Sequenzielle Durchführung der Teilstudien” or “sequenzielles Mixed Methods-Design”) of components. Note that each research tradition receives an equal number of letters (four) in its abbreviation for equity. In this article, this notation system is used in some depth.

A mixed methods design as a product has several primary characteristics that should be considered during the design process. As shown in Table  1 , the following primary design “dimensions” are emphasized in this article: purpose of mixing, theoretical drive, timing, point of integration, typological use, and degree of complexity. These characteristics are discussed below. We also provide some secondary dimensions to consider when constructing a mixed methods design (Johnson and Christensen 2017 ).

List of Primary and Secondary Design Dimensions

On the basis of these dimensions, mixed methods designs can be classified into a mixed methods typology or taxonomy. In the mixed methods literature, various typologies of mixed methods designs have been proposed (for an overview see Creswell and Plano Clark 2011 , p. 69–72).

The overall goal of mixed methods research, of combining qualitative and quantitative research components, is to expand and strengthen a study’s conclusions and, therefore, contribute to the published literature. In all studies, the use of mixed methods should contribute to answering one’s research questions.

Ultimately, mixed methods research is about heightened knowledge and validity. The design as a product should be of sufficient quality to achieve multiple validities legitimation (Johnson and Christensen 2017 ; Onwuegbuzie and Johnson 2006 ), which refers to the mixed methods research study meeting the relevant combination or set of quantitative, qualitative, and mixed methods validities in each research study.

Given this goal of answering the research question(s) with validity, a researcher can nevertheless have various reasons or purposes for wanting to strengthen the research study and its conclusions. Following is the first design dimension for one to consider when designing a study: Given the research question(s), what is the purpose of the mixed methods study?

A popular classification of purposes of mixed methods research was first introduced in 1989 by Greene, Caracelli, and Graham, based on an analysis of published mixed methods studies. This classification is still in use (Greene 2007 ). Greene et al. ( 1989 , p. 259) distinguished the following five purposes for mixing in mixed methods research:

1.  Triangulation seeks convergence, corroboration, correspondence of results from different methods; 2.  Complementarity seeks elaboration, enhancement, illustration, clarification of the results from one method with the results from the other method; 3.  Development seeks to use the results from one method to help develop or inform the other method, where development is broadly construed to include sampling and implementation, as well as measurement decisions; 4.  Initiation seeks the discovery of paradox and contradiction, new perspectives of frameworks, the recasting of questions or results from one method with questions or results from the other method; 5.  Expansion seeks to extend the breadth and range of inquiry by using different methods for different inquiry components.

In the past 28 years, this classification has been supplemented by several others. On the basis of a review of the reasons for combining qualitative and quantitative research mentioned by the authors of mixed methods studies, Bryman ( 2006 ) formulated a list of more concrete rationales for performing mixed methods research (see Appendix). Bryman’s classification breaks down Greene et al.’s ( 1989 ) categories into several aspects, and he adds a number of additional aspects, such as the following:

(a)  Credibility – refers to suggestions that employing both approaches enhances the integrity of findings. (b)  Context – refers to cases in which the combination is justified in terms of qualitative research providing contextual understanding coupled with either generalizable, externally valid findings or broad relationships among variables uncovered through a survey. (c)  Illustration – refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bones” of “dry” quantitative findings. (d)  Utility or improving the usefulness of findings – refers to a suggestion, which is more likely to be prominent among articles with an applied focus, that combining the two approaches will be more useful to practitioners and others. (e)  Confirm and discover – this entails using qualitative data to generate hypotheses and using quantitative research to test them within a single project. (f)  Diversity of views – this includes two slightly different rationales – namely, combining researchers’ and participants’ perspectives through quantitative and qualitative research respectively, and uncovering relationships between variables through quantitative research while also revealing meanings among research participants through qualitative research. (Bryman, p. 106)

Views can be diverse (f) in various ways. Some examples of mixed methods design that include a diversity of views are:

• Iteratively/sequentially connecting local/idiographic knowledge with national/general/nomothetic knowledge;
• Learning from different perspectives on teams and in the field and literature;
• Achieving multiple participation, social justice, and action;
• Determining what works for whom and the relevance/importance of context;
• Producing interdisciplinary substantive theory, including/comparing multiple perspectives and data regarding a phenomenon;
• Juxtaposition-dialogue/comparison-synthesis;
• Breaking down binaries/dualisms (some of both);
• Explaining interaction between/among natural and human systems;
• Explaining complexity.

The number of possible purposes for mixing is very large and is increasing; hence, it is not possible to provide an exhaustive list. Greene et al.’s ( 1989 ) purposes, Bryman’s ( 2006 ) rationales, and our examples of a diversity of views were formulated as classifications on the basis of examination of many existing research studies. They indicate how the qualitative and quantitative research components of a study relate to each other. These purposes can be used post hoc to classify research or a priori in the design of a new study. When designing a mixed methods study, it is sometimes helpful to list the purpose in the title of the study design.

The key point of this section is for the researcher to begin a study with at least one research question and then carefully consider what the purposes for mixing are. One can use mixed methods to examine different aspects of a single research question, or one can use separate but related qualitative and quantitative research questions. In all cases, the mixing of methods, methodologies, and/or paradigms will help answer the research questions and make improvements over a more basic study design. Fuller and richer information will be obtained in the mixed methods study.

Theoretical drive

In addition to a mixing purpose, a mixed methods research study might have an overall “theoretical drive” (Morse and Niehaus 2009 ). When designing a mixed methods study, it is occasionally helpful to list the theoretical drive in the title of the study design. An investigation, in Morse and Niehaus’s ( 2009 ) view, is focused primarily on either exploration-and-description or on testing-and-prediction. In the first case, the theoretical drive is called “inductive” or “qualitative”; in the second case, it is called “deductive” or “quantitative”. In the case of mixed methods, the component that corresponds to the theoretical drive is referred to as the “core” component (“Kernkomponente”), and the other component is called the “supplemental” component (“ergänzende Komponente”). In Morse’s notation system, the core component is written in capitals and the supplemental component is written in lowercase letters. For example, in a QUAL → quan design, more weight is attached to the data coming from the core qualitative component. Due to the decisive character of the core component, the core component must be able to stand on its own, and should be implemented rigorously. The supplemental component does not have to stand on its own.

Although this distinction is useful in some circumstances, we do not advise to apply it to every mixed methods design. First, Morse and Niehaus contend that the supplemental component can be done “less rigorously” but do not explain which aspects of rigor can be dropped. In addition, the idea of decreased rigor is in conflict with one key theme of the present article, namely that mixed methods designs should always meet the criterion of multiple validities legitimation (Onwuegbuzie and Johnson 2006 ).

The idea of theoretical drive as explicated by Morse and Niehaus has been criticized. For example, we view a theoretical drive as a feature not of a whole study, but of a research question, or, more precisely, of an interpretation of a research question. For example, if one study includes multiple research questions, it might include several theoretical drives (Schoonenboom 2016 ).

Another criticism of Morse and Niehaus’ conceptualization of theoretical drive is that it does not allow for equal-status mixed methods research (“Mixed Methods Forschung, bei der qualitative und quantitative Methoden die gleiche Bedeutung haben” or “gleichrangige Mixed Methods-Designs”), in which both the qualitative and quantitative component are of equal value and weight; this same criticism applies to Morgan’s ( 2014 ) set of designs. We agree with Greene ( 2015 ) that mixed methods research can be integrated at the levels of method, methodology, and paradigm. In this view, equal-status mixed methods research designs are possible, and they result when both the qualitative and the quantitative components, approaches, and thinking are of equal value, they take control over the research process in alternation, they are in constant interaction, and the outcomes they produce are integrated during and at the end of the research process. Therefore, equal-status mixed methods research (that we often advocate) is also called “interactive mixed methods research”.

Mixed methods research can have three different drives, as formulated by Johnson et al. ( 2007 , p. 123):

Qualitative dominant [or qualitatively driven] mixed methods research is the type of mixed research in which one relies on a qualitative, constructivist-poststructuralist-critical view of the research process, while concurrently recognizing that the addition of quantitative data and approaches are likely to benefit most research projects. Quantitative dominant [or quantitatively driven] mixed methods research is the type of mixed research in which one relies on a quantitative, postpositivist view of the research process, while concurrently recognizing that the addition of qualitative data and approaches are likely to benefit most research projects. (p. 124) The area around the center of the [qualitative-quantitative] continuum, equal status , is the home for the person that self-identifies as a mixed methods researcher. This researcher takes as his or her starting point the logic and philosophy of mixed methods research. These mixed methods researchers are likely to believe that qualitative and quantitative data and approaches will add insights as one considers most, if not all, research questions.

We leave it to the reader to decide if he or she desires to conduct a qualitatively driven study, a quantitatively driven study, or an equal-status/“interactive” study. According to the philosophies of pragmatism (Johnson and Onwuegbuzie 2004 ) and dialectical pluralism (Johnson 2017 ), interactive mixed methods research is very much a possibility. By successfully conducting an equal-status study, the pragmatist researcher shows that paradigms can be mixed or combined, and that the incompatibility thesis does not always apply to research practice. Equal status research is most easily conducted when a research team is composed of qualitative, quantitative, and mixed researchers, interacts continually, and conducts a study to address one superordinate goal.

Timing: simultaneity and dependence

Another important distinction when designing a mixed methods study relates to the timing of the two (or more) components. When designing a mixed methods study, it is usually helpful to include the word “concurrent” (“parallel”) or “sequential” (“sequenziell”) in the title of the study design; a complex design can be partially concurrent and partially sequential. Timing has two aspects: simultaneity and dependence (Guest 2013 ).

Simultaneity (“Simultanität”) forms the basis of the distinction between concurrent and sequential designs. In a  sequential design , the quantitative component precedes the qualitative component, or vice versa. In a  concurrent design , both components are executed (almost) simultaneously. In the notation of Morse ( 1991 ), concurrence is indicated by a “+” between components (e. g., QUAL + quan), while sequentiality is indicated with a “→” (QUAL → quan). Note that the use of capital letters for one component and lower case letters for another component in the same design suggest that one component is primary and the other is secondary or supplemental.

Some designs are sequential by nature. For example, in a  conversion design, qualitative categories and themes might be first obtained by collection and analysis of qualitative data, and then subsequently quantitized (Teddlie and Tashakkori 2009 ). Likewise, with Greene et al.’s ( 1989 ) initiation purpose, the initiation strand follows the unexpected results that it is supposed to explain. In other cases, the researcher has a choice. It is possible, e. g., to collect interview data and survey data of one inquiry simultaneously; in that case, the research activities would be concurrent. It is also possible to conduct the interviews after the survey data have been collected (or vice versa); in that case, research activities are performed sequentially. Similarly, a study with the purpose of expansion can be designed in which data on an effect and the intervention process are collected simultaneously, or they can be collected sequentially.

A second aspect of timing is dependence (“Abhängigkeit”) . We call two research components dependent if the implementation of the second component depends on the results of data analysis in the first component. Two research components are independent , if their implementation does not depend on the results of data analysis in the other component. Often, a researcher has a choice to perform data analysis independently or not. A researcher could analyze interview data and questionnaire data of one inquiry independently; in that case, the research activities would be independent. It is also possible to let the interview questions depend upon the outcomes of the analysis of the questionnaire data (or vice versa); in that case, research activities are performed dependently. Similarly, the empirical outcome/effect and process in a study with the purpose of expansion might be investigated independently, or the process study might take the effect/outcome as given (dependent).

In the mixed methods literature, the distinction between sequential and concurrent usually refers to the combination of concurrent/independent and sequential/dependent, and to the combination of data collection and data analysis. It is said that in a concurrent design, the data collection and data analysis of both components occurs (almost) simultaneously and independently, while in a sequential design, the data collection and data analysis of one component take place after the data collection and data analysis of the other component and depends on the outcomes of the other component.

In our opinion, simultaneity and dependence are two separate dimensions. Simultaneity indicates whether data collection is done concurrent or sequentially. Dependence indicates whether the implementation of one component depends upon the results of data analysis of the other component. As we will see in the example case studies, a concurrent design could include dependent data analysis, and a sequential design could include independent data analysis. It is conceivable that one simultaneously conducts interviews and collects questionnaire data (concurrent), while allowing the analysis focus of the interviews to depend on what emerges from the survey data (dependence).

Dependent research activities include a redirection of subsequent research inquiry. Using the outcomes of the first research component, the researcher decides what to do in the second component. Depending on the outcomes of the first research component, the researcher will do something else in the second component. If this is so, the research activities involved are said to be sequential-dependent, and any component preceded by another component should appropriately build on the previous component (see sequential validity legitimation ; Johnson and Christensen 2017 ; Onwuegbuzie and Johnson 2006 ).

It is under the purposive discretion of the researcher to determine whether a concurrent-dependent design, a concurrent-independent design, a sequential-dependent design, or a sequential-dependent design is needed to answer a particular research question or set of research questions in a given situation.

Point of integration

Each true mixed methods study has at least one “point of integration” – called the “point of interface” by Morse and Niehaus ( 2009 ) and Guest ( 2013 ) –, at which the qualitative and quantitative components are brought together. Having one or more points of integration is the distinguishing feature of a design based on multiple components. It is at this point that the components are “mixed”, hence the label “mixed methods designs”. The term “mixing”, however, is misleading, as the components are not simply mixed, but have to be integrated very carefully.

Determining where the point of integration will be, and how the results will be integrated, is an important, if not the most important, decision in the design of mixed methods research. Morse and Niehaus ( 2009 ) identify two possible points of integration: the results point of integration and the analytical point of integration.

Most commonly, integration takes place in the results point of integration . At some point in writing down the results of the first component, the results of the second component are added and integrated. A  joint display (listing the qualitative and quantitative findings and an integrative statement) might be used to facilitate this process.

In the case of an analytical point of integration , a first analytical stage of a qualitative component is followed by a second analytical stage, in which the topics identified in the first analytical stage are quantitized. The results of the qualitative component ultimately, and before writing down the results of the analytical phase as a whole, become quantitative; qualitizing also is a possible strategy, which would be the converse of this.

Other authors assume more than two possible points of integration. Teddlie and Tashakkori ( 2009 ) distinguish four different stages of an investigation: the conceptualization stage, the methodological experimental stage (data collection), the analytical experimental stage (data analysis), and the inferential stage. According to these authors, in all four stages, mixing is possible, and thus all four stages are potential points or integration.

However, the four possible points of integration used by Teddlie and Tashakkori ( 2009 ) are still too coarse to distinguish some types of mixing. Mixing in the experiential stage can take many different forms, for example the use of cognitive interviews to improve a questionnaire (tool development), or selecting people for an interview on the basis of the results of a questionnaire (sampling). Extending the definition by Guest ( 2013 ), we define the point of integration as “any point in a study where two or more research components are mixed or connected in some way”. Then, the point of integration in the two examples of this paragraph can be defined more accurately as “instrument development”, and “development of the sample”.

It is at the point of integration that qualitative and quantitative components are integrated. Some primary ways that the components can be connected to each other are as follows:

(1) merging the two data sets, (2) connecting from the analysis of one set of data to the collection of a second set of data, (3) embedding of one form of data within a larger design or procedure, and (4) using a framework (theoretical or program) to bind together the data sets (Creswell and Plano Clark 2011 , p. 76).

More generally, one can consider mixing at any or all of the following research components: purposes, research questions, theoretical drive, methods, methodology, paradigm, data, analysis, and results. One can also include mixing views of different researchers, participants, or stakeholders. The creativity of the mixed methods researcher designing a study is extensive.

Substantively, it can be useful to think of integration or mixing as comparing and bringing together two (or more) components on the basis of one or more of the purposes set out in the first section of this article. For example, it is possible to use qualitative data to illustrate a quantitative effect, or to determine whether the qualitative and the quantitative component yield convergent results ( triangulation ). An integrated result could also consist of a combination of a quantitatively established effect and a qualitative description of the underlying process . In the case of development, integration consists of an adjustment of an, often quantitative, for example, instrument or model or interpretation, based on qualitative assessments by members of the target group.

A special case is the integration of divergent results. The power of mixed methods research is its ability to deal with diversity and divergence. In the literature, we find two kinds of strategies for dealing with divergent results. A first set of strategies takes the detected divergence as the starting point for further analysis, with the aim to resolve the divergence. One possibility is to carry out further research (Cook 1985 ; Greene and Hall 2010 ). Further research is not always necessary. One can also look for a more comprehensive theory, which is able to account for both the results of the first component and the deviating results of the second component. This is a form of abduction (Erzberger and Prein 1997 ).

A fruitful starting point in trying to resolve divergence through abduction is to determine which component has resulted in a finding that is somehow expected, logical, and/or in line with existing research. The results of this research component, called the “sense” (“Lesart”), are subsequently compared to the results of the other component, called the “anti-sense” (“alternative Lesart”), which are considered dissonant, unexpected, and/or contrary to what had been found in the literature. The aim is to develop an overall explanation that fits both the sense and the anti-sense (Bazeley and Kemp 2012 ; Mendlinger and Cwikel 2008 ). Finally, a reanalysis of the data can sometimes lead to resolving divergence (Creswell and Plano Clark 2011 ).

Alternatively, one can question the existence of the encountered divergence. In this regard, Mathison ( 1988 ) recommends determining whether deviating results shown by the data can be explained by knowledge about the research and/or knowledge of the social world. Differences between results from different data sources could also be the result of properties of the methods involved, rather than reflect differences in reality (Yanchar and Williams 2006 ). In general, the conclusions of the individual components can be subjected to an inference quality audit (Teddlie and Tashakkori 2009 ), in which the researcher investigates the strength of each of the divergent conclusions. We recommend that researchers first determine whether there is “real” divergence, according to the strategies mentioned in the last paragraph. Next, an attempt can be made to resolve cases of “true” divergence, using one or more of the methods mentioned in this paragraph.

Design typology utilization

As already mentioned in Sect. 1, mixed methods designs can be classified into a mixed methods typology or taxonomy. A typology serves several purposes, including the following: guiding practice, legitimizing the field, generating new possibilities, and serving as a useful pedagogical tool (Teddlie and Tashakkori 2009 ). Note, however, that not all types of typologies are equally suitable for all purposes. For generating new possibilities, one will need a more exhaustive typology, while a useful pedagogical tool might be better served by a non-exhaustive overview of the most common mixed methods designs. Although some of the current MM design typologies include more designs than others, none of the current typologies is fully exhaustive. When designing a mixed methods study, it is often useful to borrow its name from an existing typology, or to construct a superior and nuanced clear name when your design is based on a modification of one or more of the designs.

Various typologies of mixed methods designs have been proposed. Creswell and Plano Clark’s ( 2011 ) typology of some “commonly used designs” includes six “major mixed methods designs”. Our summary of these designs runs as follows:

• Convergent parallel design (“paralleles Design”) (the quantitative and qualitative strands of the research are performed independently, and their results are brought together in the overall interpretation),
• Explanatory sequential design (“explanatives Design”) (a first phase of quantitative data collection and analysis is followed by the collection of qualitative data, which are used to explain the initial quantitative results),
• Exploratory sequential design (“exploratives Design”) (a first phase of qualitative data collection and analysis is followed by the collection of quantitative data to test or generalize the initial qualitative results),
• Embedded design (“Einbettungs-Design”) (in a traditional qualitative or quantitative design, a strand of the other type is added to enhance the overall design),
• Transformative design (“politisch-transformatives Design”) (a transformative theoretical framework, e. g. feminism or critical race theory, shapes the interaction, priority, timing and mixing of the qualitative and quantitative strand),
• Multiphase design (“Mehrphasen-Design”) (more than two phases or both sequential and concurrent strands are combined over a period of time within a program of study addressing an overall program objective).

Most of their designs presuppose a specific juxtaposition of the qualitative and quantitative component. Note that the last design is a complex type that is required in many mixed methods studies.

The following are our adapted definitions of Teddlie and Tashakkori’s ( 2009 ) five sets of mixed methods research designs (adapted from Teddlie and Tashakkori 2009 , p. 151):

• Parallel mixed designs (“paralleles Mixed-Methods-Design”) – In these designs, one has two or more parallel quantitative and qualitative strands, either with some minimal time lapse or simultaneously; the strand results are integrated into meta-inferences after separate analysis are conducted; related QUAN and QUAL research questions are answered or aspects of the same mixed research question is addressed.
• Sequential mixed designs (“sequenzielles Mixed-Methods-Design”) – In these designs, QUAL and QUAN strands occur across chronological phases, and the procedures/questions from the later strand emerge/depend/build on on the previous strand; the research questions are interrelated and sometimes evolve during the study.
• Conversion mixed designs (“Transfer-Design” or “Konversionsdesign”) – In these parallel designs, mixing occurs when one type of data is transformed to the other type and then analyzed, and the additional findings are added to the results; this design answers related aspects of the same research question,
• Multilevel mixed designs (“Mehrebenen-Mixed-Methods-Design”) – In these parallel or sequential designs, mixing occurs across multiple levels of analysis, as QUAN and QUAL data are analyzed and integrated to answer related aspects of the same research question or related questions.
• Fully integrated mixed designs (“voll integriertes Mixed-Methods-Design”) – In these designs, mixing occurs in an interactive manner at all stages of the study. At each stage, one approach affects the formulation of the other, and multiple types of implementation processes can occur. For example, rather than including integration only at the findings/results stage, or only across phases in a sequential design, mixing might occur at the conceptualization stage, the methodological stage, the analysis stage, and the inferential stage.

We recommend adding to Teddlie and Tashakkori’s typology a sixth design type, specifically, a  “hybrid” design type to include complex combinations of two or more of the other design types. We expect that many published MM designs will fall into the hybrid design type.

Morse and Niehaus ( 2009 ) listed eight mixed methods designs in their book (and suggested that authors create more complex combinations when needed). Our shorthand labels and descriptions (adapted from Morse and Niehaus 2009 , p. 25) run as follows:

• QUAL + quan (inductive-simultaneous design where, the core component is qualitative and the supplemental component is quantitative)
• QUAL → quan (inductive-sequential design, where the core component is qualitative and the supplemental component is quantitative)
• QUAN + qual (deductive-simultaneous design where, the core component is quantitative and the supplemental component is qualitative)
• QUAN → qual (deductive-sequential design, where the core component is quantitative and the supplemental component is qualitative)
• QUAL + qual (inductive-simultaneous design, where both components are qualitative; this is a multimethod design rather than a mixed methods design)
• QUAL → qual (inductive-sequential design, where both components are qualitative; this is a multimethod design rather than a mixed methods design)
• QUAN + quan (deductive-simultaneous design, where both components are quantitative; this is a multimethod design rather than a mixed methods design)
• QUAN → quan (deductive-sequential design, where both components are quantitative; this is a multimethod design rather than a mixed methods design).

Notice that Morse and Niehaus ( 2009 ) included four mixed methods designs (the first four designs shown above) and four multimethod designs (the second set of four designs shown above) in their typology. The reader can, therefore, see that the design notation also works quite well for multimethod research designs. Notably absent from Morse and Niehaus’s book are equal-status or interactive designs. In addition, they assume that the core component should always be performed either concurrent with or before the supplemental component.

Johnson, Christensen, and Onwuegbuzie constructed a set of mixed methods designs without these limitations. The resulting mixed methods design matrix (see Johnson and Christensen 2017 , p. 478) contains nine designs, which we can label as follows (adapted from Johnson and Christensen 2017 , p. 478):

• QUAL + QUAN (equal-status concurrent design),
• QUAL + quan (qualitatively driven concurrent design),
• QUAN + qual (quantitatively driven concurrent design),
• QUAL → QUAN (equal-status sequential design),
• QUAN → QUAL (equal-status sequential design),
• QUAL → quan (qualitatively driven sequential design),
• qual → QUAN (quantitatively driven sequential design),
• QUAN → qual (quantitatively driven sequential design), and
• quan → QUAL (qualitatively driven sequential design).

The above set of nine designs assumed only one qualitative and one quantitative component. However, this simplistic assumption can be relaxed in practice, allowing the reader to construct more complex designs. The Morse notation system is very powerful. For example, here is a three-stage equal-status concurrent-sequential design:

The key point here is that the Morse notation provides researchers with a powerful language for depicting and communicating the design constructed for a specific research study.

When designing a mixed methods study, it is sometimes helpful to include the mixing purpose (or characteristic on one of the other dimensions shown in Table  1 ) in the title of the study design (e. g., an explanatory sequential MM design, an exploratory-confirmatory MM design, a developmental MM design). Much more important, however, than a design name is for the author to provide an accurate description of what was done in the research study, so the reader will know exactly how the study was conducted. A design classification label can never replace such a description.

The common complexity of mixed methods design poses a problem to the above typologies of mixed methods research. The typologies were designed to classify whole mixed methods studies, and they are basically based on a classification of simple designs. In practice, many/most designs are complex. Complex designs are sometimes labeled “complex design”, “multiphase design”, “fully integrated design”, “hybrid design” and the like. Because complex designs occur very often in practice, the above typologies are not able to classify a large part of existing mixed methods research any further than by labeling them “complex”, which in itself is not very informative about the particular design. This problem does not fully apply to Morse’s notation system, which can be used to symbolize some more complex designs.

Something similar applies to the classification of the purposes of mixed methods research. The classifications of purposes mentioned in the “Purpose”-section, again, are basically meant for the classification of whole mixed methods studies. In practice, however, one single study often serves more than one purpose (Schoonenboom et al. 2017 ). The more purposes that are included in one study, the more difficult it becomes to select a design on the basis of the purpose of the investigation, as advised by Greene ( 2007 ). Of all purposes involved, then, which one should be the primary basis for the design? Or should the design be based upon all purposes included? And if so, how? For more information on how to articulate design complexity based on multiple purposes of mixing, see Schoonenboom et al. ( 2017 ).

It should be clear to the reader that, although much progress has been made in the area of mixed methods design typologies, the problem remains in developing a single typology that is effective in comprehensively listing a set of designs for mixed methods research. This is why we emphasize in this article the importance of learning to build on simple designs and construct one’s own design for one’s research questions. This will often result in a combination or “hybrid” design that goes beyond basic designs found in typologies, and a methodology section that provides much more information than a design name.

Typological versus interactive approaches to design

In the introduction, we made a distinction between design as a product and design as a process. Related to this, two different approaches to design can be distinguished: typological/taxonomic approaches (“systematische Ansätze”), such as those in the previous section, and interactive approaches (“interaktive Ansätze”) (the latter were called “dynamic” approaches by Creswell and Plano Clark 2011 ). Whereas typological/taxonomic approaches view designs as a sort of mold, in which the inquiry can be fit, interactive approaches (Maxwell 2013 ) view design as a process, in which a certain design-as-a-product might be the outcome of the process, but not its input.

The most frequently mentioned interactive approach to mixed methods research is the approach by Maxwell and Loomis ( 2003 ). Maxwell and Loomis distinguish the following components of a design: goals, conceptual framework, research question, methods, and validity. They argue convincingly that the most important task of the researcher is to deliver as the end product of the design process a design in which these five components fit together properly. During the design process, the researcher works alternately on the individual components, and as a result, their initial fit, if it existed, tends to get lost. The researcher should therefore regularly check during the research and continuing design process whether the components still fit together, and, if not, should adapt one or the other component to restore the fit between them. In an interactive approach, unlike the typological approach, design is viewed as an interactive process in which the components are continually compared during the research study to each other and adapted to each other.

Typological and interactive approaches to mixed methods research have been presented as mutually exclusive alternatives. In our view, however, they are not mutually exclusive. The interactive approach of Maxwell is a very powerful tool for conducting research, yet this approach is not specific to mixed methods research. Maxwell’s interactive approach emphasizes that the researcher should keep and monitor a close fit between the five components of research design. However, it does not indicate how one should combine qualitative and quantitative subcomponents within one of Maxwell’s five components (e. g., how one should combine a qualitative and a quantitative method, or a qualitative and a quantitative research question). Essential elements of the design process, such as timing and the point of integration are not covered by Maxwell’s approach. This is not a shortcoming of Maxwell’s approach, but it indicates that to support the design of mixed methods research, more is needed than Maxwell’s model currently has to offer.

Some authors state that design typologies are particularly useful for beginning researchers and interactive approaches are suited for experienced researchers (Creswell and Plano Clark 2011 ). However, like an experienced researcher, a research novice needs to align the components of his or her design properly with each other, and, like a beginning researcher, an advanced researcher should indicate how qualitative and quantitative components are combined with each other. This makes an interactive approach desirable, also for beginning researchers.

We see two merits of the typological/taxonomic approach . We agree with Greene ( 2007 ), who states that the value of the typological approach mainly lies in the different dimensions of mixed methods that result from its classifications. In this article, the primary dimensions include purpose, theoretical drive, timing, point of integration, typological vs. interactive approaches, planned vs. emergent designs, and complexity (also see secondary dimensions in Table  1 ). Unfortunately, all of these dimensions are not reflected in any single design typology reviewed here. A second merit of the typological approach is the provision of common mixed methods research designs, of common ways in which qualitative and quantitative research can be combined, as is done for example in the major designs of Creswell and Plano Clark ( 2011 ). Contrary to other authors, however, we do not consider these designs as a feature of a whole study, but rather, in line with Guest ( 2013 ), as a feature of one part of a design in which one qualitative and one quantitative component are combined. Although one study could have only one purpose, one point of integration, et cetera, we believe that combining “designs” is the rule and not the exception. Therefore, complex designs need to be constructed and modified as needed, and during the writing phase the design should be described in detail and perhaps given a creative and descriptive name.

Planned versus emergent designs

A mixed methods design can be thought out in advance, but can also arise during the course of the conduct of the study; the latter is called an “emergent” design (Creswell and Plano Clark 2011 ). Emergent designs arise, for example, when the researcher discovers during the study that one of the components is inadequate (Morse and Niehaus 2009 ). Addition of a component of the other type can sometimes remedy such an inadequacy. Some designs contain an emergent component by their nature. Initiation, for example, is the further exploration of unexpected outcomes. Unexpected outcomes are by definition not foreseen, and therefore cannot be included in the design in advance.

The question arises whether researchers should plan all these decisions beforehand, or whether they can make them during, and depending on the course of, the research process. The answer to this question is twofold. On the one hand, a researcher should decide beforehand which research components to include in the design, such that the conclusion that will be drawn will be robust. On the other hand, developments during research execution will sometimes prompt the researcher to decide to add additional components. In general, the advice is to be prepared for the unexpected. When one is able to plan for emergence, one should not refrain from doing so.

Dimension of complexity

Next, mixed methods designs are characterized by their complexity. In the literature, simple and complex designs are distinguished in various ways. A common distinction is between simple investigations with a single point of integration versus complex investigations with multiple points of integration (Guest 2013 ). When designing a mixed methods study, it can be useful to mention in the title whether the design of the study is simple or complex. The primary message of this section is as follows: It is the responsibility of the researcher to create more complex designs when needed to answer his or her research question(s) .

Teddlie and Tashakkori’s ( 2009 ) multilevel mixed designs and fully integrated mixed designs are both complex designs, but for different reasons. A multilevel mixed design is more complex ontologically, because it involves multiple levels of reality. For example, data might be collected both at the levels of schools and students, neighborhood and households, companies and employees, communities and inhabitants, or medical practices and patients (Yin 2013 ). Integration of these data does not only involve the integration of qualitative and quantitative data, but also the integration of data originating from different sources and existing at different levels. Little if any published research has discussed the possible ways of integrating data obtained in a multilevel mixed design (see Schoonenboom 2016 ). This is an area in need of additional research.

The fully-integrated mixed design is more complex because it contains multiple points of integration. As formulated by Teddlie and Tashakkori ( 2009 , p. 151):

In these designs, mixing occurs in an interactive manner at all stages of the study. At each stage, one approach affects the formulation of the other, and multiple types of implementation processes can occur.

Complexity, then, not only depends on the number of components, but also on the extent to which they depend on each other (e. g., “one approach affects the formulation of the other”).

Many of our design dimensions ultimately refer to different ways in which the qualitative and quantitative research components are interdependent. Different purposes of mixing ultimately differ in the way one component relates to, and depends upon, the other component. For example, these purposes include dependencies, such as “x illustrates y” and “x explains y”. Dependencies in the implementation of x and y occur to the extent that the design of y depends on the results of x (sequentiality). The theoretical drive creates dependencies, because the supplemental component y is performed and interpreted within the context and the theoretical drive of core component x. As a general rule in designing mixed methods research, one should examine and plan carefully the ways in which and the extent to which the various components depend on each other.

The dependence among components, which may or may not be present, has been summarized by Greene ( 2007 ). It is seen in the distinction between component designs (“Komponenten-Designs”), in which the components are independent of each other, and integrated designs (“integrierte Designs”), in which the components are interdependent. Of these two design categories, integrated designs are the more complex designs.

Secondary design considerations

The primary design dimensions explained above have been the focus of this article. There are a number of secondary considerations for researchers to also think about when they design their studies (Johnson and Christensen 2017 ). Now we list some secondary design issues and questions that should be thoughtfully considered during the construction of a strong mixed methods research design.

• Phenomenon: Will the study be addressing (a) the same part or different parts of one phenomenon? (b) different phenomena?, or (c) the phenomenon/phenomena from different perspectives? Is the phenomenon (a) expected to be unique (e. g., historical event, particular group)?, (b) something expected to be part of a more regular and predictable phenomenon, or (c) a complex mixture of these?
• Social scientific theory: Will the study generate a new substantive theory, test an already constructed theory, or achieve both in a sequential arrangement? Or is the researcher not interested in substantive theory based on empirical data?
• Ideological drive: Will the study have an explicitly articulated ideological drive (e. g., feminism, critical race paradigm, transformative paradigm)?
• Combination of sampling methods: What specific quantitative sampling method(s) will be used? What specific qualitative sampling methods(s) will be used? How will these be combined or related?
• Degree to which the research participants will be similar or different: For example, participants or stakeholders with known differences of perspective would provide participants that are quite different.
• Degree to which the researchers on the research team will be similar or different: For example, an experiment conducted by one researcher would be high on similarity, but the use of a heterogeneous and participatory research team would include many differences.
• Implementation setting: Will the phenomenon be studied naturalistically, experimentally, or through a combination of these?
• Degree to which the methods similar or different: For example, a structured interview and questionnaire are fairly similar but administration of a standardized test and participant observation in the field are quite different.
• Validity criteria and strategies: What validity criteria and strategies will be used to address the defensibility of the study and the conclusions that will be drawn from it (see Chapter 11 in Johnson and Christensen 2017 )?
• Full study: Will there be essentially one research study or more than one? How will the research report be structured?

Two case studies

The above design dimensions are now illustrated by examples. A nice collection of examples of mixed methods studies can be found in Hesse-Biber ( 2010 ), from which the following examples are taken. The description of the first case example is shown in Box 1.

Box 1

Summary of Roth ( 2006 ), research regarding the gender-wage gap within Wall Street securities firms. Adapted from Hesse-Biber ( 2010 , pp. 457–458)

Louise Marie Roth’s research, Selling Women Short: Gender and Money on Wall Street ( 2006 ), tackles gender inequality in the workplace. She was interested in understanding the gender-wage gap among highly performing Wall Street MBAs, who on the surface appeared to have the same “human capital” qualifications and were placed in high-ranking Wall Street securities firms as their first jobs. In addition, Roth wanted to understand the “structural factors” within the workplace setting that may contribute to the gender-wage gap and its persistence over time. […] Roth conducted semistructured interviews, nesting quantitative closed-ended questions into primarily qualitative in-depth interviews […] In analyzing the quantitative data from her sample, she statistically considered all those factors that might legitimately account for gendered differences such as number of hours worked, any human capital differences, and so on. Her analysis of the quantitative data revealed the presence of a significant gender gap in wages that remained unexplained after controlling for any legitimate factors that might otherwise make a difference. […] Quantitative findings showed the extent of the wage gap while providing numerical understanding of the disparity but did not provide her with an understanding of the specific processes within the workplace that might have contributed to the gender gap in wages. […] Her respondents’ lived experiences over time revealed the hidden inner structures of the workplace that consist of discriminatory organizational practices with regard to decision making in performance evaluations that are tightly tied to wage increases and promotion.

This example nicely illustrates the distinction we made between simultaneity and dependency. On the two aspects of the timing dimension, this study was a concurrent-dependent design answering a set of related research questions. The data collection in this example was conducted simultaneously, and was thus concurrent – the quantitative closed-ended questions were embedded into the qualitative in-depth interviews. In contrast, the analysis was dependent, as explained in the next paragraph.

One of the purposes of this study was explanation: The qualitative data were used to understand the processes underlying the quantitative outcomes. It is therefore an explanatory design, and might be labelled an “explanatory concurrent design”. Conceptually, explanatory designs are often dependent: The qualitative component is used to explain and clarify the outcomes of the quantitative component. In that sense, the qualitative analysis in the case study took the outcomes of the quantitative component (“the existence of the gender-wage gap” and “numerical understanding of the disparity”), and aimed at providing an explanation for that result of the quantitative data analysis , by relating it to the contextual circumstances in which the quantitative outcomes were produced. This purpose of mixing in the example corresponds to Bryman’s ( 2006 ) “contextual understanding”. On the other primary dimensions, (a) the design was ongoing over a three-year period but was not emergent, (b) the point of integration was results, and (c) the design was not complex with respect to the point of integration, as it had only one point of integration. Yet, it was complex in the sense of involving multiple levels; both the level of the individual and the organization were included. According to the approach of Johnson and Christensen ( 2017 ), this was a QUAL + quan design (that was qualitatively driven, explanatory, and concurrent). If we give this study design a name, perhaps it should focus on what was done in the study: “explaining an effect from the process by which it is produced”. Having said this, the name “explanatory concurrent design” could also be used.

The description of the second case example is shown in Box 2.

Box 2

Summary of McMahon’s ( 2007 ) explorative study of the meaning, role, and salience of rape myths within the subculture of college student athletes. Adapted from Hesse-Biber ( 2010 , pp. 461–462)

Sarah McMahon ( 2007 ) wanted to explore the subculture of college student athletes and specifically the meaning, role, and salience of rape myths within that culture. […] While she was looking for confirmation between the quantitative ([structured] survey) and qualitative (focus groups and individual interviews) findings, she entered this study skeptical of whether or not her quantitative and qualitative findings would mesh with one another. McMahon […] first administered a survey [instrument] to 205 sophomore and junior student athletes at one Northeast public university. […] The quantitative data revealed a very low acceptance of rape myths among this student population but revealed a higher acceptance of violence among men and individuals who did not know a survivor of sexual assault. In the second qualitative (QUAL) phase, “focus groups were conducted as semi-structured interviews” and facilitated by someone of the same gender as the participants (p. 360). […] She followed this up with a third qualitative component (QUAL), individual interviews, which were conducted to elaborate on themes discovered in the focus groups and determine any differences in students’ responses between situations (i. e., group setting vs. individual). The interview guide was designed specifically to address focus group topics that needed “more in-depth exploration” or clarification (p. 361). The qualitative findings from the focus groups and individual qualitative interviews revealed “subtle yet pervasive rape myths” that fell into four major themes: “the misunderstanding of consent, the belief in ‘accidental’ and fabricated rape, the contention that some women provoke rape, and the invulnerability of female athletes” (p. 363). She found that the survey’s finding of a “low acceptance of rape myths … was contradicted by the findings of the focus groups and individual interviews, which indicated the presence of subtle rape myths” (p. 362).

On the timing dimension, this is an example of a sequential-independent design. It is sequential, because the qualitative focus groups were conducted after the survey was administered. The analysis of the quantitative and qualitative data was independent: Both were analyzed independently, to see whether they yielded the same results (which they did not). This purpose, therefore, was triangulation. On the other primary dimensions, (a) the design was planned, (b) the point of integration was results, and (c) the design was not complex as it had only one point of integration, and involved only the level of the individual. The author called this a “sequential explanatory” design. We doubt, however, whether this is the most appropriate label, because the qualitative component did not provide an explanation for quantitative results that were taken as given. On the contrary, the qualitative results contradicted the quantitative results. Thus, a “sequential-independent” design, or a “sequential-triangulation” design or a “sequential-comparative” design would probably be a better name.

Notice further that the second case study had the same point of integration as the first case study. The two components were brought together in the results. Thus, although the case studies are very dissimilar in many respects, this does not become visible in their point of integration. It can therefore be helpful to determine whether their point of extension is different. A  point of extension is the point in the research process at which the second (or later) component comes into play. In the first case study, two related, but different research questions were answered, namely the quantitative question “How large is the gender-wage gap among highly performing Wall Street MBAs after controlling for any legitimate factors that might otherwise make a difference?”, and the qualitative research question “How do structural factors within the workplace setting contribute to the gender-wage gap and its persistence over time?” This case study contains one qualitative research question and one quantitative research question. Therefore, the point of extension is the research question. In the second case study, both components answered the same research question. They differed in their data collection (and subsequently in their data analysis): qualitative focus groups and individual interviews versus a quantitative questionnaire. In this case study, the point of extension was data collection. Thus, the point of extension can be used to distinguish between the two case studies.

Summary and conclusions

The purpose of this article is to help researchers to understand how to design a mixed methods research study. Perhaps the simplest approach is to design is to look at a single book and select one from the few designs included in that book. We believe that is only useful as a starting point. Here we have shown that one often needs to construct a research design to fit one’s unique research situation and questions.

First, we showed that there are there are many purposes for which qualitative and quantitative methods, methodologies, and paradigms can be mixed. This must be determined in interaction with the research questions. Inclusion of a purpose in the design name can sometimes provide readers with useful information about the study design, as in, e. g., an “explanatory sequential design” or an “exploratory-confirmatory design”.

The second dimension is theoretical drive in the sense that Morse and Niehaus ( 2009 ) use this term. That is, will the study have an inductive or a deductive drive, or, we added, a combination of these. Related to this idea is whether one will conduct a qualitatively driven, a quantitatively driven, or an equal-status mixed methods study. This language is sometimes included in the design name to communicate this characteristic of the study design (e. g., a “quantitatively driven sequential mixed methods design”).

The third dimension is timing , which has two aspects: simultaneity and dependence. Simultaneity refers to whether the components are to be implemented concurrently, sequentially, or a combination of these in a multiphase design. Simultaneity is commonly used in the naming of a mixed methods design because it communicates key information. The second aspect of timing, dependence , refers to whether a later component depends on the results of an earlier component, e. g., Did phase two specifically build on phase one in the research study? The fourth design dimension is the point of integration, which is where the qualitative and quantitative components are brought together and integrated. This is an essential dimension, but it usually does not need to be incorporated into the design name.

The fifth design dimension is that of typological vs. interactive design approaches . That is, will one select a design from a typology or use a more interactive approach to construct one’s own design? There are many typologies of designs currently in the literature. Our recommendation is that readers examine multiple design typologies to better understand the design process in mixed methods research and to understand what designs have been identified as popular in the field. However, when a design that would follow from one’s research questions is not available, the researcher can and should (a) combine designs into new designs or (b) simply construct a new and unique design. One can go a long way in depicting a complex design with Morse’s ( 1991 ) notation when used to its full potential. We also recommend that researchers understand the process approach to design from Maxwell and Loomis ( 2003 ), and realize that research design is a process and it needs, oftentimes, to be flexible and interactive.

The sixth design dimension or consideration is whether a design will be fully specified during the planning of the research study or if the design (or part of the design) will be allowed to emerge during the research process, or a combination of these. The seventh design dimension is called complexity . One sort of complexity mentioned was multilevel designs, but there are many complexities that can enter designs. The key point is that good research often requires the use of complex designs to answer one’s research questions. This is not something to avoid. It is the responsibility of the researcher to learn how to construct and describe and name mixed methods research designs. Always remember that designs should follow from one’s research questions and purposes, rather than questions and purposes following from a few currently named designs.

In addition to the six primary design dimensions or considerations, we provided a set of additional or secondary dimensions/considerations or questions to ask when constructing a mixed methods study design. Our purpose throughout this article has been to show what factors must be considered to design a high quality mixed methods research study. The more one knows and thinks about the primary and secondary dimensions of mixed methods design the better equipped one will be to pursue mixed methods research.

Acknowledgments

Open access funding provided by University of Vienna.

Biographies

1965, Dr., Professor of Empirical Pedagogy at University of Vienna, Austria. Research Areas: Mixed Methods Design, Philosophy of Mixed Methods Research, Innovation in Higher Education, Design and Evaluation of Intervention Studies, Educational Technology. Publications: Mixed methods in early childhood education. In: M. Fleer & B. v. Oers (Eds.), International handbook on early childhood education (Vol. 1). Dordrecht, The Netherlands: Springer 2017; The multilevel mixed intact group analysis: A mixed method to seek, detect, describe and explain differences between intact groups. Journal of Mixed Methods Research 10, 2016; The realist survey: How respondents’ voices can be used to test and revise correlational models. Journal of Mixed Methods Research 2015. Advance online publication.

1957, PhD, Professor of Professional Studies at University of South Alabama, Mobile, Alabama USA. Research Areas: Methods of Social Research, Program Evaluation, Quantitative, Qualitative and Mixed Methods, Philosophy of Social Science. Publications: Research methods, design and analysis. Boston, MA 2014 (with L. Christensen and L. Turner); Educational research: Quantitative, qualitative and mixed approaches. Los Angeles, CA 2017 (with L. Christensen); The Oxford handbook of multimethod and mixed methods research inquiry. New York, NY 2015 (with S. Hesse-Biber).

Bryman’s ( 2006 ) scheme of rationales for combining quantitative and qualitative research 1

• Triangulation or greater validity – refers to the traditional view that quantitative and qualitative research might be combined to triangulate findings in order that they may be mutually corroborated. If the term was used as a synonym for integrating quantitative and qualitative research, it was not coded as triangulation.
• Offset – refers to the suggestion that the research methods associated with both quantitative and qualitative research have their own strengths and weaknesses so that combining them allows the researcher to offset their weaknesses to draw on the strengths of both.
• Completeness – refers to the notion that the researcher can bring together a more comprehensive account of the area of enquiry in which he or she is interested if both quantitative and qualitative research are employed.
• Process – quantitative research provides an account of structures in social life but qualitative research provides sense of process.
• Different research questions – this is the argument that quantitative and qualitative research can each answer different research questions but this item was coded only if authors explicitly stated that they were doing this.
• Explanation – one is used to help explain findings generated by the other.
• Unexpected results – refers to the suggestion that quantitative and qualitative research can be fruitfully combined when one generates surprising results that can be understood by employing the other.
• Instrument development – refers to contexts in which qualitative research is employed to develop questionnaire and scale items – for example, so that better wording or more comprehensive closed answers can be generated.
• Sampling – refers to situations in which one approach is used to facilitate the sampling of respondents or cases.
• Credibility – refer s to suggestions that employing both approaches enhances the integrity of findings.
• Context – refers to cases in which the combination is rationalized in terms of qualitative research providing contextual understanding coupled with either generalizable, externally valid findings or broad relationships among variables uncovered through a survey.
• Illustration – refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bones” of “dry” quantitative findings.
• Utility or improving the usefulness of findings – refers to a suggestion, which is more likely to be prominent among articles with an applied focus, that combining the two approaches will be more useful to practitioners and others.
• Confirm and discover – this entails using qualitative data to generate hypotheses and using quantitative research to test them within a single project.
• Diversity of views – this includes two slightly different rationales – namely, combining researchers’ and participants’ perspectives through quantitative and qualitative research respectively, and uncovering relationships between variables through quantitative research while also revealing meanings among research participants through qualitative research.
• Enhancement or building upon quantitative/qualitative findings – this entails a reference to making more of or augmenting either quantitative or qualitative findings by gathering data using a qualitative or quantitative research approach.
• Other/unclear.
• Not stated.

1 Reprinted with permission from “Integrating quantitative and qualitative research: How is it done?” by Alan Bryman ( 2006 ), Qualitative Research, 6, pp. 105–107.

Contributor Information

Judith Schoonenboom, Email: [email protected] .

R. Burke Johnson, Email: ude.amabalahtuos@nosnhojb .

• Bazeley, Pat, Lynn Kemp Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research. 2012; 6 :55–72. doi: 10.1177/1558689811419514. [ CrossRef ] [ Google Scholar ]
• Bryman A. Integrating quantitative and qualitative research: how is it done? Qualitative Research. 2006; 6 :97–113. doi: 10.1177/1468794106058877. [ CrossRef ] [ Google Scholar ]
• Cook TD. Postpositivist critical multiplism. In: Shotland RL, Mark MM, editors. Social science and social policy. Beverly Hills: SAGE; 1985. pp. 21–62. [ Google Scholar ]
• Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 2. Los Angeles: SAGE; 2011. [ Google Scholar ]
• Erzberger C, Prein G. Triangulation: Validity and empirically-based hypothesis construction. Quality and Quantity. 1997; 31 :141–154. doi: 10.1023/A:1004249313062. [ CrossRef ] [ Google Scholar ]
• Greene JC. Mixed methods in social inquiry. San Francisco: Jossey-Bass; 2007. [ Google Scholar ]
• Greene JC. Preserving distinctions within the multimethod and mixed methods research merger. Sharlene Hesse-Biber and R. Burke Johnson. New York: Oxford University Press; 2015. [ Google Scholar ]
• Greene JC, Valerie J, Caracelli, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis. 1989; 11 :255–274. doi: 10.3102/01623737011003255. [ CrossRef ] [ Google Scholar ]
• Greene JC, Hall JN. Dialectics and pragmatism. In: Tashakkori A, Teddlie C, editors. SAGE handbook of mixed methods in social & behavioral research. 2. Los Angeles: SAGE; 2010. pp. 119–167. [ Google Scholar ]
• Guest, Greg Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research. 2013; 7 :141–151. doi: 10.1177/1558689812461179. [ CrossRef ] [ Google Scholar ]
• Hesse-Biber S. Qualitative approaches to mixed methods practice. Qualitative Inquiry. 2010; 16 :455–468. doi: 10.1177/1077800410364611. [ CrossRef ] [ Google Scholar ]
• Johnson BR. Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research. 2017; 11 :156–173. doi: 10.1177/1558689815607692. [ CrossRef ] [ Google Scholar ]
• Johnson BR, Christensen LB. Educational research: Quantitative, qualitative, and mixed approaches. 6. Los Angeles: SAGE; 2017. [ Google Scholar ]
• Johnson BR, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educational Researcher. 2004; 33 (7):14–26. doi: 10.3102/0013189X033007014. [ CrossRef ] [ Google Scholar ]
• Johnson BR, Onwuegbuzie AJ, Turner LA. Toward a definition of mixed methods research. Journal of Mixed Methods Research. 2007; 1 :112–133. doi: 10.1177/1558689806298224. [ CrossRef ] [ Google Scholar ]
• Mathison S. Why triangulate? Educational Researcher. 1988; 17 :13–17. doi: 10.3102/0013189X017002013. [ CrossRef ] [ Google Scholar ]
• Maxwell JA. Qualitative research design: An interactive approach. 3. Los Angeles: SAGE; 2013. [ Google Scholar ]
• Maxwell, Joseph A., and Diane M. Loomis. 2003. Mixed methods design: An alternative approach. In Handbook of mixed methods in social & behavioral research , Eds. Abbas Tashakkori and Charles Teddlie, 241–271. Thousand Oaks: Sage.
• McMahon S. Understanding community-specific rape myths: Exploring student athlete culture. Affilia. 2007; 22 :357–370. doi: 10.1177/0886109907306331. [ CrossRef ] [ Google Scholar ]
• Mendlinger S, Cwikel J. Spiraling between qualitative and quantitative data on women’s health behaviors: A double helix model for mixed methods. Qualitative Health Research. 2008; 18 :280–293. doi: 10.1177/1049732307312392. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morgan DL. Integrating qualitative and quantitative methods: a pragmatic approach. Los Angeles: Sage; 2014. [ Google Scholar ]
• Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nursing Research. 1991; 40 :120–123. doi: 10.1097/00006199-199103000-00014. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morse JM, Niehaus L. Mixed method design: Principles and procedures. Walnut Creek: Left Coast Press; 2009. [ Google Scholar ]
• Onwuegbuzie AJ, Burke Johnson R. The “validity” issue in mixed research. Research in the Schools. 2006; 13 :48–63. [ Google Scholar ]
• Roth LM. Selling women short: Gender and money on Wall Street. Princeton: Princeton University Press; 2006. [ Google Scholar ]
• Schoonenboom J. The multilevel mixed intact group analysis: a mixed method to seek, detect, describe and explain differences between intact groups. Journal of Mixed Methods Research. 2016; 10 :129–146. doi: 10.1177/1558689814536283. [ CrossRef ] [ Google Scholar ]
• Schoonenboom, Judith, R. Burke Johnson, and Dominik E. Froehlich. 2017, in press. Combining multiple purposes of mixing within a mixed methods research design. International Journal of Multiple Research Approaches .
• Teddlie CB, Tashakkori A. Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Los Angeles: Sage; 2009. [ Google Scholar ]
• Yanchar SC, Williams DD. Reconsidering the compatibility thesis and eclecticism: Five proposed guidelines for method use. Educational Researcher. 2006; 35 (9):3–12. doi: 10.3102/0013189X035009003. [ CrossRef ] [ Google Scholar ]
• Yin RK. Case study research: design and methods. 5. Los Angeles: SAGE; 2013. [ Google Scholar ]

• Open access
• Published: 02 April 2024

Towards universal health coverage in Vietnam: a mixed-method case study of enrolling people with tuberculosis into social health insurance

• Rachel Forse   ORCID: orcid.org/0000-0002-0716-3342 1 , 2 ,
• Clara Akie Yoshino 2 ,
• Thanh Thi Nguyen 1 ,
• Thi Hoang Yen Phan 3 ,
• Luan N. Q. Vo 1 , 2 ,
• Andrew J. Codlin 1 , 2 ,
• Lan Nguyen 4 ,
• Chi Hoang 4 ,
• Lopa Basu 5 ,
• Minh Pham 5 ,
• Hoa Binh Nguyen 6 ,
• Luong Van Dinh 6 ,
• Maxine Caws 7 , 8 ,
• Tom Wingfield 2 , 7 ,
• Knut Lönnroth 2 &
• Kristi Sidney-Annerstedt 2  

Health Research Policy and Systems volume  22 , Article number:  40 ( 2024 ) Cite this article

146 Accesses

4 Altmetric

Metrics details

Vietnam’s primary mechanism of achieving sustainable funding for universal health coverage (UHC) and financial protection has been through its social health insurance (SHI) scheme. Steady progress towards access has been made and by 2020, over 90% of the population were enrolled in SHI. In 2022, as part of a larger transition towards the increased domestic financing of healthcare, tuberculosis (TB) services were integrated into SHI. This change required people with TB to use SHI for treatment at district-level facilities or to pay out of pocket for services. This study was conducted in preparation for this transition. It aimed to understand more about uninsured people with TB, assess the feasibility of enrolling them into SHI, and identify the barriers they faced in this process.

A mixed-method case study was conducted using a convergent parallel design between November 2018 and January 2022 in ten districts of Hanoi and Ho Chi Minh City, Vietnam. Quantitative data were collected through a pilot intervention that aimed to facilitate SHI enrollment for uninsured individuals with TB. Descriptive statistics were calculated. Qualitative interviews were conducted with 34 participants, who were purposively sampled for maximum variation. Qualitative data were analyzed through an inductive approach and themes were identified through framework analysis. Quantitative and qualitative data sources were triangulated.

We attempted to enroll 115 uninsured people with TB into SHI; 76.5% were able to enroll. On average, it took 34.5 days to obtain a SHI card and it cost USD 66 per household. The themes indicated that a lack of knowledge, high costs for annual premiums, and the household-based registration requirement were barriers to SHI enrollment. Participants indicated that alternative enrolment mechanisms and greater procedural flexibility, particularly for undocumented people, is required to achieve full population coverage with SHI in urban centers.

Conclusions

Significant addressable barriers to SHI enrolment for people affected by TB were identified. A quarter of individuals remained unable to enroll after receiving enhanced support due to lack of required documentation. The experience gained during this health financing transition is relevant for other middle-income countries as they address the provision of financial protection for the treatment of infectious diseases.

Peer Review reports

Contributing to universal health coverage (UHC) by improving access to fair and sustainable health financing, of which one mechanism is health insurance, has become a priority among low- and middle-income countries [ 1 , 2 ]. Many countries in the Asia Pacific region have made steady progress towards UHC coverage through sustained political commitments and fiscal policy aligned with their commitment [ 3 ]. By 2020, 27 countries had implemented a social health insurance (SHI) financing mechanism, which typically includes open enrollment for the full population along with partial or full subsidization of healthcare costs for vulnerable groups [ 4 ].

Vietnam’s first SHI scheme was piloted in 1989 and grew through successive pilots and expansions. In 2009 the national-level Health Insurance Law (HIL) went into effect, uniting the existing health insurance programs and schemes for the poor [ 5 ]. Amendments to the HIL effective in 2015 made SHI compulsory for all and pooled risk by re-structuring registration around the household unit [ 4 ]. A household in Vietnam is defined by inclusion in the ‘family book ’, the national system of family and address registration [ 6 ].

Access to SHI in Vietnam increased rapidly, principally through subsidization of premiums. Specific groups were enrolled automatically with full subsidy, including vulnerable populations (e.g., households classified as ‘poor’, children aged < 6, people aged > 80), pensioners and meritorious groups (e.g., veterans). Partial premium subsidization was also available for students, households classified as ‘near-poor’ and some farmers [ 7 ]. More than half of SHI members are entitled to 80% coverage with a 20% co-payment for services [ 8 ]. However, co-payments are reduced to 5% or are eliminated for subsidized groups (e.g., households classified as ‘poor’ and ‘near-poor’, children < 6) [ 4 ].

By 2020, Vietnam recorded a 91% national SHI coverage rate [ 7 ]. Those remaining uninsured mainly consisted of informally employed individuals [ 7 ]. Enrollment rates were highest among low- and high-income groups, leaving the so-called “missing middle” of uninsured [ 5 ].

Vietnam continues to transition to domestic financing of healthcare from donor financing by expanding the breadth of the national SHI. The Ministry of Health and Vietnam Social Security (VSS) have begun to close service gaps and integrate vertical health programs (e.g., those with stand-alone budget allocations and/or direct donor financing) into SHI financing [ 7 ]. The costs for antiretroviral therapy (ART) were transitioned from donor funding to SHI in 2019 [ 9 ], COVID-19 treatments were covered by SHI in 2020, and financing for tuberculosis (TB) care was fully transitioned to SHI in 2022 [ 7 ].

Until this financing transition, anti-TB medications and consultations were provided free of charge in the public sector, funded by a mixture of domestic and international funding [ 10 ]. While first-line TB medications were included in the SHI-reimbursable list of essential medicines, the government network of District TB Units (DTUs) were ineligible for registration with VSS, or reimbursement for services provided. Since July 2022, TB health facilities that met certain conditions could register with VSS and receive reimbursements for TB consultations, diagnostics and anti-TB medications [ 11 ]. The financing for drug-resistant (DR-)TB tests and medications remains largely unchanged, co-financed by the Global Fund and domestic budgets [ 12 ].

This transition of the TB financing model in Vietnam is a large undertaking as the country has the world’s 10th highest TB burden and the SHI benefits package is already considered to be generous, and the sustainability of the SHI fund is a concern [ 4 , 13 ] An estimated 169,000 individuals developed TB in 2021, and the disease killed approximately 14,200 [ 14 ]. A national costing survey of TB-affected households showed that 63% experienced catastrophic costs, spending ≥ 20% of their annual income on TB [ 10 ]. Many face food insecurity and cope with TB-related costs by taking loans, dissavings and informally borrowing money [ 10 , 15 , 16 ].

As Vietnam continues to expand SHI financing for the TB program, it is now vital for people with TB to have SHI. Those without SHI coverage will need to finance their care out of pocket (OOP) or purchase SHI and make co-payments for their care to be subsidized. For these reasons, it is important to understand why certain people with TB are uninsured, the feasibility of enrolling them in insurance when they begin treatment, and the challenges they may face with enrolling in SHI.

We conducted a convergent parallel mixed-method case study [ 17 ]. A case study was selected because it is well-suited to describe a complex issue in a real-life setting [ 18 ]. We used a naturalistic design with theoretical sampling of uninsured persons with TB using an interpretivist approach [ 19 ]. Mixed methods were selected to facilitate comparisons between quantitative and qualitative data and interpretation of the findings. An intervention, assisting TB-affected households to enroll in SHI, was conducted between November 2019 and January 2022, prior to the integration of the TB program into the SHI financing scheme. Quantitative data collection sought to answer questions regarding enrollment success rate, time to enrollment and cost of SHI enrollment for uninsured TB-affected households upon TB treatment initiation. The qualitative data explored barriers to SHI enrollment to explain and contextualize the quantitative findings. The quantitative and qualitative data were weighted equally [ 17 ].

Intervention description

A pilot intervention was conducted to facilitate SHI enrollment for people with TB in ten districts of Ha Noi and Ho Chi Minh City (HCMC). The standard process for first-time enrollment into SHI was mapped and costed from a household’s perspective (Additional file 1 ). Uninsured individuals were identified from the TB treatment register when they were enrolled in drug-susceptible (DS-)TB treatment at DTUs [ 20 ]. Study staff then attempted to facilitate enrollment of the person with TB and up to three household members into SHI.

SHI enrollment support included home visits by study staff to provide detailed information and counseling about the process of SHI enrollment, assistance with SHI application preparation including obtaining photocopies of all required documents, follow-up to obtain missing documentation within the household, accompaniment to the SHI office for application submission, and direct payment of the annual SHI premium for the household. For people who did not have the paperwork certifying temporary residence in Hanoi or Ho Chi Minh City, staff visited the local government office to obtain the information about the process for individual cases to obtain residency certificates and support participants with navigation of the bureaucracy. TB-affected people and their household members were also provided with a hotline number to call and receive support during working hours from the social workers who were employed by the study. Study staff attempted to facilitate the SHI enrollment process throughout the entire 6-month duration of DS-TB treatment. After a TB treatment outcome was recorded by the DTU, study staff stopped assisting with SHI enrollment and participants were recorded as ‘not enrolled in SHI’ in the study’s evaluation.

Quantitative methods

Case-level TB treatment notification data and SHI status were exported from VITIMES, the government-implemented electronic TB register for Vietnam, for all individuals who started TB treatment during the intervention period. The pilot intervention recruited participants from two TB treatment support projects (Project 1, n  = 59 and Project 2, n  = 56) [ 21 , 22 ] and tracked study forms housed in ONA.io. The sample size was determined by the availability of funding provided by the donor for treatment support service delivery, rather than to measure a specific end point of SHI enrollment. Descriptive statistics summarizing the enrollment cascade and turnaround time of enrollment were calculated using Stata v17 (Stata17 Corp, College Station, USA). To obtain the mean costs for household SHI enrollment, total direct costs for purchasing SHI were summed and divided by the total number of participants. Costs were captured in Vietnamese Dong (VND) and converted to United States Dollars (USD) using the exchange rate from the mid-point of the pilot intervention (1 June 2020) from OANDA.com.

Qualitative methods

Individuals were purposively sampled for maximum variation to ensure representation of all implementation areas and provide gender balance [ 23 ].The concept of information power guided the sample size [ 24 ]. Given the well-defined study aim, high quality in-depth responses from the participants and the authors’ expertise in the subject area, the sample size of 19 individual interviews and three focus group discussions was deemed appropriate. These were conducted in Ha Noi and HCMC. A total of 34 individuals participated in the interviews (Table  1 ).

They included 14 people enrolled in the pilot intervention, five community members who were non-beneficiaries of the treatment support intervention, 13 TB program staff from the national-, provincial- and district-levels and two study staff. Interviews were conducted at two time points: June 2019 and 2020. SHI enrollment barriers were collected as part of a qualitative study on the acceptability of providing cash transfers and SHI enrollment to adults with TB [ 25 ]. During the second round of interviews in 2020, study staff were included due to their in-depth knowledge of the challenges faced by TB-affected households when attempting to enroll in SHI and their ability to suggest programmatic-level solutions to these challenges. These interviews were conducted one-on-one, after the other interviews and focus groups had been conducted to reduce bias. The interviews were conducted at the National Lung Hospital, HCMC Provincial Lung Hospital, study office or DTUs. All interviews were conducted and transcribed in Vietnamese, translated into English, checked and finalized by a lead translator.

The interviews were analyzed through an inductive approach and themes were drawn through a framework analysis [ 26 ] to identify barriers to enrolling in SHI using Dedoose Version 7.0.23 (SocioCultural Research Consultants, Los Angeles, USA).

Data triangulation

Quantitative and qualitative data were collected in parallel. Triangulation of quantitative and qualitative data was conducted to synthesize findings and assess the level of agreement, convergence, and divergence from the findings generated by the different methods [ 17 ].

During the study, 5887 individuals were treated for DS-TB across the 10 intervention districts (Table  2 ). TB registers indicated that 2846 (48.3%) individuals were uninsured upon treatment initiation, or their SHI enrollment status was not recorded. Among 115 uninsured study participants, 88 (76.5%) were successfully enrolled in SHI before the end of their TB treatment. Among those, the household had an average of two members, resulting in a total of 206 individuals living in TB-affected households receiving SHI coverage through the pilot intervention.

The median time between DS-TB treatment initiation and SHI card issuance was 34.5 days (IQR 24–68): 11 days (IQR 5–23) between treatment initiation and pilot enrollment, 7 days (IQR 1–19.5) for SHI application preparation and submission, and 12 days (IQR 9–20) for application processing and SHI card provision.

The qualitative data showed that participants across all participant groups broadly understood that SHI is a system designed to prevent catastrophic OOP medical expenditure. As shown in Table  3 , National and provincial-level TB staff described SHI as a human right and spoke about achieving UHC as a nation; no other participant groups discussed SHI in this way. However, district-level doctors and intervention beneficiaries spoke in greater details about coverage and service gaps, and the practicalities of utilizing SHI. These participant groups expressed that when individuals purchase SHI only after a negative health event, such as a TB diagnosis, then the social safety net is unavailable to provide support until SHI coverage begins. Drawn from these views, the first theme indicated that the optimal time to purchase SHI is prior to a TB diagnosis.

One DTU staff member described how the standard processing time, or delays in processing SHI applications led to periods of high OOP expenditure:

“Unfortunately, claims are not immediately paid upon [SHI registration] submission. They may be handled in about 2 or 3 weeks, or even one month. That is why the insurance is not available at the time that they want to go for an examination and treat their condition using insurance.” (Female, District-level TB staff)

A complementary theme was that perceived lack of knowledge about SHI enrollment procedures prevents or delays enrollment. District-level TB doctors and program staff identified a lack of understanding and knowledge of the SHI enrollment process as a main contributor to lack of insurance or delays in obtaining coverage.

“Actually, for some people [with TB] who do not clearly understand the [enrollment] procedures… it will take a lot of time [to obtain SHI]. It also depends on the staff who handle the files at the commune; some staff are very enthusiastic and they help patients complete forms. There are cases [...] where they [people with TB] are required to fill in all information and write specific codes of each insurance card [from other family members] on a form. Meanwhile some people in their family work far from home and cannot send their insurance cards home in a timely manner.” (Female, program staff)

Participants tended to believe that individuals who lacked information about SHI made up the small minority of uninsured people in Vietnamese society. The above quote illustrated that the complicated administrative process prohibits enrollment; however, a factor potentially facilitating SHI enrollment may be the helpfulness of the person processing the SHI application.

The average cost per household to obtain SHI enrollment for one year (Table  2 ) was VND 1,503,313 (USD 65.52). (For detailed information on the costs of SHI enrollment, see Additional file 1 ). A third theme contextualized this finding and showed that SHI enrollment costs were perceived as prohibitively high for some. Cost was a greater challenge for lower income families, who did not meet the government’s criterion of households classified as ‘poor’ or ‘near-poor’, and were therefore ineligible for premium subsidies and SHI registration with lower co-payment rates. One DTU doctor reported that:

“We think that it is simple to buy health insurance cards, but that is only true for those who have sustainable income - when our income is much higher than the fee for buying health insurance. For some people, buying health insurance is a luxury.” (Male, District-level TB staff)

Twenty-seven people with TB (23.5%) were unable to obtain SHI coverage. The primary reason (70.4%) was missing documentation. In four instances (14.8%) a household member other than the person with TB refused to enroll in SHI. One individual (3.7%) died during the enrollment process. Three individuals (11.1%) did not enroll for other reasons.

SHI refusal by household members was not identified as a barrier to SHI enrollment in the qualitative data. However, a fourth theme confirmed the primary reason for non-enrollment by showing that some individuals do not possess the required documentation to obtain SHI, such as their identity card or ‘family book.’ [See Supplementary File] Even with six months of support from study staff, some TB-affected households were unable to gather the required documents for enrollment. The following quotation by an undocumented, elderly woman with TB illustrates the prolonged challenges she faced with obtaining formal employment, access to government services and SHI:

“I have had problems with my personal papers for a few decades and I cannot adjust my papers because I don’t have the money. […] I searched for my Identity Card and found out that I had lost it. Then I came back there [my hometown] to get the family book, to reissue my ID and to get my CV certified so I could join a company. I was very young at that time, just a little bit more than thirty years old, and I learned that I was cut from the family book.” (Female, pilot beneficiary)

To address challenges with documentation, one DTU officer in HCMC suggested that individuals who had never been insured required a change to the SHI registration requirements to ensure that everyone in Vietnam can access SHI:

“I think we should be flexible with these cases or we can find another way. Normally, the people who really need the support and the insurance or cash support, they are the people who have less information. […] We cannot have the same requirements for these people as for other people. Actually, for those who have [met] all conditions, they already have health insurance cards.” (Male, District-level TB staff)

Participants expressed that the uninsured had often not purchased SHI for a reason, and alternative registration procedures were needed to make SHI accessible for all. A fifth theme was identified indicating that current SHI enrollment procedures may prevent full population coverage.

Beyond the undocumented, some participants reported the enrollment mandate for the entire household (made under the Amendment to the HIL) for first-time enrollees was viewed as prohibitive of SHI coverage.

“Because in the old days, health insurance was sold individually for each person, but now it is sold to households, and many households do not have as good economic [situation]… so they can only afford to buy it for 50% or 60% of the household. Unskilled labor or low-income labor cannot afford to buy it for the whole family. That is to say, it is easier to buy it for each individual and it is difficult to buy for the whole family.” (Male, community member)

Though individual registration would make SHI more accessible to individuals with TB due to lower annual costs, household members with high vulnerability to TB would not be covered if policy promoted individual enrollment solely for TB.

This mixed-methods case study showed that by providing full subsidy and registration assistance, most uninsured people with TB could access SHI. However, the median time to insurance coverage meant that approximately 20% of a person’s DS-TB treatment duration remained uncovered by SHI despite successful enrollment. A substantial number of participants were unable to enroll in SHI and are likely to be perpetually locked out of SHI due to lack of personal documentation. Additional barriers to SHI enrollment were found to be lack of knowledge, the cost of obtaining coverage, and the household-based registration requirement.

The pilot intervention had dedicated staff who facilitated SHI application development and submission, yet it still took a median of 34.5 days for SHI coverage to take effect. In a context where this level of support is not available to all people with TB, it is likely that the turnaround time for SHI coverage is longer due to the complicated bureaucracy involved. This poses a major challenge, as TB-affected households incur the highest cost during the first two months of treatment [ 15 ]. One cost avoidance/mitigation strategy that people with a TB diagnosis may employ following the health financing transition is delaying TB treatment initiation until SHI coverage commences. This will likely lead to worse outcomes and sustained community transmission. The time between diagnosis and treatment should be rigorously monitored to ensure that this coping strategy is not employed, and alternative support should be made available to ensure that people diagnosed with TB are able to receive immediate treatment.

With the TB health financing transition, the uninsured will be asked to pay OOP for TB treatment and most insured individuals must co-pay for TB services which were previously provided free of cost. A national patient cost survey in 2018 found that 63% of TB-affected households experienced catastrophic costs under the previous health financing model [ 10 ]. There is a risk that the proportion of TB-affected households experiencing catastrophic costs could increase with the introduction of fees. This was not found to be the case for people living with HIV (PLHIV) when the costs of ART transitioned to SHI in Vietnam, but a new nationally representative TB costing survey is needed to assess this risk [ 9 ]. Several domestic solutions could ameliorate these challenges. As suggested for the Indian context, domestic revenues allocated by the Ministry of Finance to VSS could be increased to better support TB care [ 27 ]. VSS could also reclassify the category of TB disease and thus ensure that SHI paid for all diagnostics and drugs associated with TB treatment, without the need for a co-payment. A mid-term review of the Global Fund program in Vietnam has also called for a SHI package specifically designed to cover the OOP medical costs of TB care [ 28 ]. There are several potential mechanisms to prevent costs from falling on TB-affected households. A deeper investigation is needed to understand the fiscal space available within the Vietnamese government to cover such costs.

This case study showed that 23.5% of the uninsured people with TB were never able to enroll for the duration of their treatment, primarily due to lack of documentation. Specific provisions need to be made for the undocumented to receive free TB diagnosis, consultations, and medications through routine practice of the TB program. Multi- and bi-lateral funding mechanisms can also play a role in filling gaps by paying for TB tests for the uninsured, purchasing SHI for those diagnosed with TB, subsidizing or reimbursing OOP expenditure in the period before SHI coverage takes effect, and fully financing TB care for the undocumented. Furthermore, longer-term health system strengthening initiatives, such as creating a legal mechanism for the undocumented to obtain SHI, are likely needed to address the challenges faced by the 9% of the general population that remain uninsured. The ILO has called for “determining new strategies, which may include extension of state budget-funded subsidies to further support the participation of workers in the informal economy [ 7 ].” These forms of inclusive initiatives would solve the TB-specific challenges identified in this study and have a large positive impact on society.

We found that addressing the cost of SHI premiums and knowledge gaps in the enrollment procedures may improve SHI coverage. These findings mirror those following the transition of HIV financing to SHI in 2017. A study among PLHIV identified burdensome processes, lack of information about SHI registration procedures, and high SHI premium costs for a household as key barriers to SHI coverage [ 29 ]. However, a cluster randomized control trial which provided education, a 25% premium subsidy, or both to uninsured households found that these interventions had limited effects on SHI enrollment. Yet, “less healthy” individuals had higher SHI enrollment rates [ 30 ]. This suggests that people who have just received a TB diagnosis could be more receptive to interventions promoting SHI enrollment through premium subsidization and education. Vietnam’s National TB Program (NTP) has established a fund to subsidize SHI enrollment costs for TB-affected individuals. The size of the fund could be increased with additional support while access to the fund and the procedures for receiving support could be optimized [ 31 ]. Given the SHI transition, the NTP should also consider providing educational materials about the SHI enrollment process through the DTU network to uninsured persons with TB.

TB registers indicated that 52% of people starting TB treatment in the urban intervention districts had recorded SHI coverage. This rate is lower than other recent SHI coverage reports. A 2018–2022 DS-TB costing survey reported a SHI coverage of 70% [ 32 ], while in a DR-TB costing survey (2020–2022) it was 85% [ 16 ]. All available data sources indicate that SHI coverage among people with TB is lower than the general population, which is indicative of their socioeconomic vulnerability [ 33 ]. However, this large SHI coverage rate discrepancy may be explained by people with TB not revealing they had SHI coverage, or DTU staff could have also inconsistently recorded an individual’s SHI status in the paper TB registers since these data did not have much clinical relevance for TB treatment at the time. Now that DTUs receive financial reimbursements for the TB services from VSS, SHI coverage rates in treatment registers are likely to increase. Further research should be conducted to understand the national SHI coverage rate for people receiving TB treatment, along with the risk factors associated with being uninsured.

Limitations

This case study was conducted in the two largest cities of Vietnam and findings may not be representative of the entire country. Quantitative data were collected in a programmatic setting, and SHI coverage data for all individuals initiating TB treatment in the intervention areas appear to be underreported for reasons described above. Lastly, we were unable to collect SHI enrollment data from a control population, either prospectively during the pilot intervention or retrospectively during the pilot evaluation. As a result, we do not have information on the enrollment status or time to obtain SHI coverage among a population that did not receive assistance from the pilot intervention. However, given the substantial additional support provided by study staff for the enrollment process, we believe it is safe to assume that if left alone, TB-affected households would be slower in the enrollment process and likely enroll in lower rates.

Vietnam is viewed as a leader among Southeast Asian nations in its commitment and progress towards UHC. This mixed-methods case study illustrated the progress that Vietnam has made in its path to greater domestic financing of healthcare through SHI. This study is one of the first to examine the integration of TB services into SHI in Vietnam and define the challenges that people with TB face while attempting to gain access to financial protection after receiving a TB diagnosis. In order to make strides towards UHC in Vietnam and to close population coverage gaps, initiatives are required to specifically address the barriers faced by the uninsured. This study found that the majority of the uninsured were able to gain access to SHI through full subsidization of premiums, enrollment assistance and education. However, initiating TB care and SHI enrollment concomitantly left a significant portion of the 6-month TB treatment duration without financial protection. Additionally, a quarter of the uninsured with TB were unable to gain access to SHI during treatment, primarily due to a lack of documentation. There is great need for official mechanisms to be in place that enable those without sufficient state documents to access the TB program and to address the time-sensitive nature of providing effective financial protection during treatment of an infectious disease. These findings are relevant for other high TB burden, middle-income countries who are on a similar pathway for transitioning away from donor-financed TB programs to ones supported with a higher proportion of domestic resources.

Availability of data and materials

The quantitative dataset used and analyzed during the current study are available from the corresponding author on reasonable request. Seven anonymized transcripts of interviews with the people enrolled in the pilot intervention and non-beneficiaries have been uploaded to the following URL: https://doi.org/ https://doi.org/10.5281/zenodo.7736220 .

Abbreviations

Anti antiretroviral therapy

Drug resistant tuberculosis

Drug susceptible tuberculosis

District TB Unit

Ho Chi Minh City

Health Insurance Law

Human immunodeficiency virus

International Labour Organization

Interquartile range

National Tuberculosis Program

Out of pocket

People Living with HIV

Social Health Insurance

• Tuberculosis

Universal Health Coverage

United States Dollar

Vietnamese Dong

Vietnam Social Security

World health statistics 2023: monitoring health for the SDGs, sustainable development goals. https://www.who.int/publications-detail-redirect/9789240074323 . Accessed 31 May 2023.

SDG Target 3.8 | Achieve universal health coverage, including financial risk protection, access to quality essential health-care services and access to safe, effective, quality and affordable essential medicines and vaccines for all. https://www.who.int/data/gho/data/themes/topics/indicator-groups/indicator-group-details/GHO/sdg-target-3.8-achieve-universal-health-coverage-(uhc)-including-financial-risk-protection . Accessed 29 May 2023.

Hondo D, Kim Y. Achieving universal health coverage in developing Asia and the Pacific. Asian Development Bank Institute; 2023. Report No.: 2023-15 (November). https://doi.org/10.56506/VUWA5043 . Accessed 1 Feb 2024.

Le QN, Blizzard L, Si L, Giang LT, Neil AL. The evolution of social health insurance in Vietnam and its role towards achieving universal health coverage. Health Policy OPEN. 2020;100011. http://www.sciencedirect.com/science/article/pii/S2590229620300095 . Accessed 10 Sep 2023.

Somanathan A, Tandon A, Dao HL, Hurt KL, Fuenzalida-Puelma HL. Moving toward universal coverage of social health insurance in Vietnam: assessments and options. Washington, DC: World Bank; 2014, p. 177. (Directions in Development). https://doi.org/10.1596/978-1-4648-0261-4

Hardy A. Rules and resources: negotiating the household registration system in Vietnam under reform. Sojourn J Soc Issues Southeast Asia. 2001;16(2):187–212.

Article   CAS   Google Scholar  

Bales S, Phe Goursat M, Phuong NK. Extending social health protection in Viet Nam: accelerating progress towards universal health coverage. International Labour Organization; 2021, p. 1–14. (Social Protection in action: building social protection floors for all.). https://www.social-protection.org/gimi/gess/ShowRessource.action;jsessionid=iFmxgS-hSrmJjf_1uG90JAnijskVEc8XJD9-4lubAqbFD4-JPYOm!341651707?id=57668 . Accessed 26 Aug 2022.

Bao Hiem Xa Hoi [Vietnam Social Security]. 2019. Phát triển bền vững BHYT toàn dân [Sustainable development of universal health insurance]. https://baohiemxahoi.gov.vn/congkhai/Pages/so-lieu-thong-ke-nganh.aspx?CateID=130&ItemID=12916 . Accessed 26 Aug 2022.

Vu LTH, Johns B, Bui QTT, Duong Thuy A, Nguyen Thi D, Nguyen Thi H, et al. Moving to social health insurance financing and payment for HIV/AIDS treatment in Vietnam. Health Serves Insights. 2021;14:1178632920988843. https://doi.org/10.1177/1178632920988843 .

Article   Google Scholar  

Nhung NV, Hoa NB, Anh NT, Anh LTN, Siroka A, Lonnroth K, et al. Measuring catastrophic costs due to tuberculosis in Viet Nam. Int J Tuberc Lung Dis. 2018;22(9):983–90.

Article   CAS   PubMed   Google Scholar  

Tran VT. Circular No. 36/2021/TT-BYT dated December 31, 2021 on Medical examination and treatment, and payment for medical examination and treatment costs covered by health insurance relating to tuberculosis examination and treatment. 36/2021/TT-BYT Dec 31, 2021. https://hethongphapluat.com/circular-no-36-2021-tt-byt-dated-december-31-2021-on-medical-examination-and-treatment-and-payment-for-medical-examination-and-treatment-costs-covered-by-health-insurance-relating-to-tuberculosis-examination-and-treatment.html .

The Global Fund. Country Profiles Tuberculosis- 2022- Viet Nam. The Global Fund; 2023. https://data.theglobalfund.org/location/VNM/documents . Accessed 30 May 2023.

Nguyen Thi Thu Cúc. Socialist Republic of Viet Nam: strengthening the policy and institutional framework of social health insurance. Hanoi, Vietnam: Asian Development Bank; 2021, p. 1–27. Report No.: 50139–002. https://www.adb.org/projects/documents/vie-50139-002-tacr-4 . Accessed 26 Aug 2022.

Global tuberculosis report 2022. Geneva: World Health Organization; 2022. Licence: CC BY-NC-SA 3.0 IGO. https://www.who.int/publications-detail-redirect/9789240061729 .

Vo LNQ, Forse RJ, Codlin AJ, Dang HM, Van Truong V, Nguyen LH, et al. Socio-protective effects of active case finding on catastrophic costs from tuberculosis in Ho Chi Minh City, Viet Nam: a longitudinal patient cost survey. BMC Health Serv Res. 2021;21(1):1051. https://doi.org/10.1186/s12913-021-06984-2 .

Article   PubMed   PubMed Central   Google Scholar  

Pham TAM, Forse R, Codlin AJ, Phan THY, Nguyen TT, Nguyen N, et al. Determinants of catastrophic costs among households affected by multi-drug resistant tuberculosis in Ho Chi Minh City, Viet Nam: a prospective cohort study. BMC Public Health. 2023;23(1):2372. https://doi.org/10.1186/s12889-023-17078-5 .

Creswell JW, Clark Plano VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks, California: SAGE Publications; 2017. p. 520.

Google Scholar  

George AL, Bennett A. Case studies and theory development in the social sciences. Cambridge, MA: MIT Press; 2005.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11(1):100. https://doi.org/10.1186/1471-2288-11-100 .

Thai LH, Nhat LM, Shah N, Lyss S, Ackers M. Sensitivity, completeness and agreement of the tuberculosis electronic system in Ho Chi Minh City. Viet Nam Public Health Act. 2017;7(4):294–8.

IMPACT TB Overview | IMPACT TB. https://impacttbproject.org/impact-tb-overview/ . Accessed 2 Jun 2023.

Fact Sheet: USAID Erase TB Project 2020. 2020. https://www.usaid.gov/vietnam/documents/fact-sheet-erase-tb-project . Accessed 24 Sep 2024.

Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. 2015;42(5):533–44. https://doi.org/10.1007/s10488-013-0528-y .

Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res. 2016;26(13):1753–60. https://doi.org/10.1177/1049732315617444 .

Article   PubMed   Google Scholar  

Forse R, Nguyen TT, Dam T, Vo LNQ, Codlin AJ, Caws M, et al. A qualitative assessment on the acceptability of providing cash transfers and social health insurance for tuberculosis-affected families in Ho Chi Minh City, Vietnam. PLOS Glob Public Health. 2023;3(12): e0002439.

Ritchie J, Lewis J, Nicholls CM, Ormston R. Qualitative research practice: a guide for social science students and researchers. Second edn. SAGE Publications Ltd; 2013. p. 456.

Behera DK, Dash U, Sahu SK. Exploring the possible sources of fiscal space for health in India: insights from political regimes. Health Res Policy Syst. 2022;20(1):32. https://doi.org/10.1186/s12961-022-00831-4 .

Draft Report: Key Messages from the TB National Strategic Plan Mid Term Review (MTR) for Vietnam, between 8th to 18th January 2023. Hanoi, Vietnam: Vietnam’s National TB Program; 2023 Jan p. 1.

Matsumoto S, Nguyen HDT, Nguyen DT, Van Tran G, Tanuma J, Mizushima D, et al. The patient voice: a survey of worries and anxieties during health system transition in HIV services in Vietnam. BMC Int Health Hum Rights. 2020;20:1.

Wagstaff A, Nguyen HTH, Dao H, Bales S. Encouraging health insurance for the informal sector: a cluster randomized experiment in Vietnam. Health Econ. 2016;25(6):663–74. https://doi.org/10.1002/hec.3293 .

Hoa NB, Nhung NV. National tuberculosis patients cost survey: research findings lead to change in policy and practice. Viet Nam Public Health Act. 2019;9(2):50–2.

Wiemers A, Sidney-Annerstedt K, Forse R, Phan THY, Codlin A, Vo LNQ, et al. EP-13-729 COVID-19 lockdowns resulting in household income decline for people with TB in Ho Chi Minh City: interrupted time series analysis of household income changes. In: Proceedings of the union world conference on lung health 2022. Virtual event, 8–11 November 2022: the international journal of tuberculosis and lung disease (IJTLD); 2022. p. 191.

Lönnroth K, Jaramillo E, Williams BG, Dye C, Raviglione M. Drivers of tuberculosis epidemics: the role of risk factors and social determinants. Soc Sci Med. 2009;68(12):2240–6.

Download references

Acknowledgements

The authors would like to acknowledge the contributions of Hoang Thi My Linh, Chu Thi Hoang Anh, Nguyen Khac Cuong, Nham Thi Yen Ngoc and Tran Thai Hiep for conducting qualitative interviews and assisting with SHI enrollment activities. Special thanks to Dr. Kerri Viney for providing insightful comments on an early draft of this manuscript; they greatly strengthened the final version. This work was graciously supported by the staff of Vietnam’s National TB Program, the Hanoi Lung Hospital, Pham Ngoc Thach Provincial TB Hospital and 10 District TB Units. Lastly, we would like to thank the interview participants who shared their time and insights.

Open access funding provided by Karolinska Institute. The European Commission's Horizon 2020 program supported the provision of SHI and all data collection in 2019 through the IMPACT-TB study under grant agreement number 733174. For the period of 2020–2022, support to implement the pilot and conduct the evaluation was made possible by the generous support of the American people through the USAID under award number 72044020FA00001. TW was supported by grants from: the Wellcome Trust, UK ( Seed Award, grant number 209075/Z/17/Z); the Department of Health and Social Care (DHSC), the Foreign, Commonwealth & Development Office (FCDO), the Medical Research Council (MRC) and Wellcome, UK (Joint Global Health Trials, MR/V004832/1); the Medical Research Council (Public Health Intervention Development Award “PHIND”, APP2293); and the Medical Research Foundation (Dorothy Temple Cross International Collaboration Research Grant, MRF-131–0006-RG-KHOS-C0942). KSA was supported by the ASPECT Trial funded the Swedish Research Council (2022-00727). The contents of this study are the responsibility of the listed authors, and do not necessarily reflect the views of USAID or the United States Government.

Author information

Authors and affiliations.

Friends for International TB Relief, Hanoi, Vietnam

Rachel Forse, Thanh Thi Nguyen, Luan N. Q. Vo & Andrew J. Codlin

Department of Global Public Health, WHO Collaboration Centre on Tuberculosis and Social Medicine, Karolinska Institutet, Stockholm, Sweden

Rachel Forse, Clara Akie Yoshino, Luan N. Q. Vo, Andrew J. Codlin, Tom Wingfield, Knut Lönnroth & Kristi Sidney-Annerstedt

Centre for Development of Community Health Initiatives, Hanoi, Vietnam

Thi Hoang Yen Phan

IRD VN, Ho Chi Minh City, Vietnam

Lan Nguyen & Chi Hoang

USAID Vietnam, Hanoi, Vietnam

Lopa Basu & Minh Pham

National Lung Hospital, Hanoi, Vietnam

Hoa Binh Nguyen & Luong Van Dinh

Centre for TB Research, Department of Clinical Sciences, Liverpool School of Tropical Medicine, Liverpool, UK

Maxine Caws & Tom Wingfield

Birat Nepal Medical Trust, Kathmandu, Nepal

Maxine Caws

You can also search for this author in PubMed   Google Scholar

Contributions

This study was conceived of by RF, KSA, TTN, THYP, CAY, AJC, LNQV. The study was administered by RF, YP, TTN, AJC. Support from Vietnam’s National TB program was provided by HBN and LVD. The methodology was developed by RJ, CAY, KV, KL, KSA. The analysis was carried out by RF, CAY, TTN, and THYP. LNQV, AJC, TW, LN, CH, LB, MP, HBN, LVD, MC, KV, KL, and KSA supported the interpretation of findings. The first manuscript was written by RF. All co-authors reviewed and commented on the initial manuscript. The final manuscript was approved and reviewed by all authors.

Corresponding author

Correspondence to Rachel Forse .

Ethics declarations

Ethics approval and consent to participate.

All study procedures were conducted in strict adherence to the Declaration of Helsinki. Ethical approvals were granted by the National Lung Hospital Institutional Review Board (114/19/CT-HĐKH-ĐĐ), the Pham Ngoc Thach Hospital Institutional Review Board (1225/PNT-HĐĐĐ) and Ha Noi University of Public Health Institutional Review Board (300/2020/YTCC-HD3). All participants provided written informed consent and individual-level data were pseudonymized prior to analysis.

Consent for publication

Informed written consent was obtained for all individuals who the study attempted to enroll in SHI, as part of the pilot intervention. It was also obtained for all individuals who participated in the qualitative interviews.

Competing interests

Ten of the authors received salary support from one of the funding agencies to implement the pilot interventions and their evaluation. Two of the authors were employed by United States Agency for International Development (USAID), which funded one of the two pilot interventions. They played no role in the design or implementation of the pilot interventions or their evaluation, but during the development of the manuscript, they provided their insights about the context of the results and Vietnam’s health financing transition as experts in the field.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Mapping of procedures and costs for first-time enrollment into Vietnam's social health insurance scheme.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Forse, R., Yoshino, C.A., Nguyen, T.T. et al. Towards universal health coverage in Vietnam: a mixed-method case study of enrolling people with tuberculosis into social health insurance. Health Res Policy Sys 22 , 40 (2024). https://doi.org/10.1186/s12961-024-01132-8

Download citation

Received : 11 August 2023

Accepted : 13 March 2024

Published : 02 April 2024

DOI : https://doi.org/10.1186/s12961-024-01132-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social health insurance
• Universal health coverage
• Health financing transition
• Financial protection

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]