comparative review of two research articles

• Utility Menu

GA4 Tracking Code

Gen ed writes, writing across the disciplines at harvard college.

• Comparative Analysis

What It Is and Why It's Useful

Comparative analysis asks writers to make an argument about the relationship between two or more texts. Beyond that, there's a lot of variation, but three overarching kinds of comparative analysis stand out:

• Coordinate (A ↔ B): In this kind of analysis, two (or more) texts are being read against each other in terms of a shared element, e.g., a memoir and a novel, both by Jesmyn Ward; two sets of data for the same experiment; a few op-ed responses to the same event; two YA books written in Chicago in the 2000s; a film adaption of a play; etc. 
• Subordinate (A  → B) or (B → A ): Using a theoretical text (as a "lens") to explain a case study or work of art (e.g., how Anthony Jack's The Privileged Poor can help explain divergent experiences among students at elite four-year private colleges who are coming from similar socio-economic backgrounds) or using a work of art or case study (i.e., as a "test" of) a theory's usefulness or limitations (e.g., using coverage of recent incidents of gun violence or legislation un the U.S. to confirm or question the currency of Carol Anderson's The Second ).
• Hybrid [A  → (B ↔ C)] or [(B ↔ C) → A] , i.e., using coordinate and subordinate analysis together. For example, using Jack to compare or contrast the experiences of students at elite four-year institutions with students at state universities and/or community colleges; or looking at gun culture in other countries and/or other timeframes to contextualize or generalize Anderson's main points about the role of the Second Amendment in U.S. history.

"In the wild," these three kinds of comparative analysis represent increasingly complex—and scholarly—modes of comparison. Students can of course compare two poems in terms of imagery or two data sets in terms of methods, but in each case the analysis will eventually be richer if the students have had a chance to encounter other people's ideas about how imagery or methods work. At that point, we're getting into a hybrid kind of reading (or even into research essays), especially if we start introducing different approaches to imagery or methods that are themselves being compared along with a couple (or few) poems or data sets.

Why It's Useful

In the context of a particular course, each kind of comparative analysis has its place and can be a useful step up from single-source analysis. Intellectually, comparative analysis helps overcome the "n of 1" problem that can face single-source analysis. That is, a writer drawing broad conclusions about the influence of the Iranian New Wave based on one film is relying entirely—and almost certainly too much—on that film to support those findings. In the context of even just one more film, though, the analysis is suddenly more likely to arrive at one of the best features of any comparative approach: both films will be more richly experienced than they would have been in isolation, and the themes or questions in terms of which they're being explored (here the general question of the influence of the Iranian New Wave) will arrive at conclusions that are less at-risk of oversimplification.

For scholars working in comparative fields or through comparative approaches, these features of comparative analysis animate their work. To borrow from a stock example in Western epistemology, our concept of "green" isn't based on a single encounter with something we intuit or are told is "green." Not at all. Our concept of "green" is derived from a complex set of experiences of what others say is green or what's labeled green or what seems to be something that's neither blue nor yellow but kind of both, etc. Comparative analysis essays offer us the chance to engage with that process—even if only enough to help us see where a more in-depth exploration with a higher and/or more diverse "n" might lead—and in that sense, from the standpoint of the subject matter students are exploring through writing as well the complexity of the genre of writing they're using to explore it—comparative analysis forms a bridge of sorts between single-source analysis and research essays.

Typical learning objectives for single-sources essays: formulate analytical questions and an arguable thesis, establish stakes of an argument, summarize sources accurately, choose evidence effectively, analyze evidence effectively, define key terms, organize argument logically, acknowledge and respond to counterargument, cite sources properly, and present ideas in clear prose.

Common types of comparative analysis essays and related types: two works in the same genre, two works from the same period (but in different places or in different cultures), a work adapted into a different genre or medium, two theories treating the same topic; a theory and a case study or other object, etc.

How to Teach It: Framing + Practice

Framing multi-source writing assignments (comparative analysis, research essays, multi-modal projects) is likely to overlap a great deal with "Why It's Useful" (see above), because the range of reasons why we might use these kinds of writing in academic or non-academic settings is itself the reason why they so often appear later in courses. In many courses, they're the best vehicles for exploring the complex questions that arise once we've been introduced to the course's main themes, core content, leading protagonists, and central debates.

For comparative analysis in particular, it's helpful to frame assignment's process and how it will help students successfully navigate the challenges and pitfalls presented by the genre. Ideally, this will mean students have time to identify what each text seems to be doing, take note of apparent points of connection between different texts, and start to imagine how those points of connection (or the absence thereof)

• complicates or upends their own expectations or assumptions about the texts
• complicates or refutes the expectations or assumptions about the texts presented by a scholar
• confirms and/or nuances expectations and assumptions they themselves hold or scholars have presented
• presents entirely unforeseen ways of understanding the texts

—and all with implications for the texts themselves or for the axes along which the comparative analysis took place. If students know that this is where their ideas will be heading, they'll be ready to develop those ideas and engage with the challenges that comparative analysis presents in terms of structure (See "Tips" and "Common Pitfalls" below for more on these elements of framing).

Like single-source analyses, comparative essays have several moving parts, and giving students practice here means adapting the sample sequence laid out at the " Formative Writing Assignments " page. Three areas that have already been mentioned above are worth noting:

• Gathering evidence : Depending on what your assignment is asking students to compare (or in terms of what), students will benefit greatly from structured opportunities to create inventories or data sets of the motifs, examples, trajectories, etc., shared (or not shared) by the texts they'll be comparing. See the sample exercises below for a basic example of what this might look like.
• Why it Matters: Moving beyond "x is like y but also different" or even "x is more like y than we might think at first" is what moves an essay from being "compare/contrast" to being a comparative analysis . It's also a move that can be hard to make and that will often evolve over the course of an assignment. A great way to get feedback from students about where they're at on this front? Ask them to start considering early on why their argument "matters" to different kinds of imagined audiences (while they're just gathering evidence) and again as they develop their thesis and again as they're drafting their essays. ( Cover letters , for example, are a great place to ask writers to imagine how a reader might be affected by reading an their argument.)
• Structure: Having two texts on stage at the same time can suddenly feel a lot more complicated for any writer who's used to having just one at a time. Giving students a sense of what the most common patterns (AAA / BBB, ABABAB, etc.) are likely to be can help them imagine, even if provisionally, how their argument might unfold over a series of pages. See "Tips" and "Common Pitfalls" below for more information on this front.

Sample Exercises and Links to Other Resources

• Common Pitfalls
• Advice on Timing
• Try to keep students from thinking of a proposed thesis as a commitment. Instead, help them see it as more of a hypothesis that has emerged out of readings and discussion and analytical questions and that they'll now test through an experiment, namely, writing their essay. When students see writing as part of the process of inquiry—rather than just the result—and when that process is committed to acknowledging and adapting itself to evidence, it makes writing assignments more scientific, more ethical, and more authentic. 
• Have students create an inventory of touch points between the two texts early in the process.
• Ask students to make the case—early on and at points throughout the process—for the significance of the claim they're making about the relationship between the texts they're comparing.
• For coordinate kinds of comparative analysis, a common pitfall is tied to thesis and evidence. Basically, it's a thesis that tells the reader that there are "similarities and differences" between two texts, without telling the reader why it matters that these two texts have or don't have these particular features in common. This kind of thesis is stuck at the level of description or positivism, and it's not uncommon when a writer is grappling with the complexity that can in fact accompany the "taking inventory" stage of comparative analysis. The solution is to make the "taking inventory" stage part of the process of the assignment. When this stage comes before students have formulated a thesis, that formulation is then able to emerge out of a comparative data set, rather than the data set emerging in terms of their thesis (which can lead to confirmation bias, or frequency illusion, or—just for the sake of streamlining the process of gathering evidence—cherry picking). 
• For subordinate kinds of comparative analysis , a common pitfall is tied to how much weight is given to each source. Having students apply a theory (in a "lens" essay) or weigh the pros and cons of a theory against case studies (in a "test a theory") essay can be a great way to help them explore the assumptions, implications, and real-world usefulness of theoretical approaches. The pitfall of these approaches is that they can quickly lead to the same biases we saw here above. Making sure that students know they should engage with counterevidence and counterargument, and that "lens" / "test a theory" approaches often balance each other out in any real-world application of theory is a good way to get out in front of this pitfall.
• For any kind of comparative analysis, a common pitfall is structure. Every comparative analysis asks writers to move back and forth between texts, and that can pose a number of challenges, including: what pattern the back and forth should follow and how to use transitions and other signposting to make sure readers can follow the overarching argument as the back and forth is taking place. Here's some advice from an experienced writing instructor to students about how to think about these considerations:

a quick note on STRUCTURE

     Most of us have encountered the question of whether to adopt what we might term the “A→A→A→B→B→B” structure or the “A→B→A→B→A→B” structure.  Do we make all of our points about text A before moving on to text B?  Or do we go back and forth between A and B as the essay proceeds?  As always, the answers to our questions about structure depend on our goals in the essay as a whole.  In a “similarities in spite of differences” essay, for instance, readers will need to encounter the differences between A and B before we offer them the similarities (A d →B d →A s →B s ).  If, rather than subordinating differences to similarities you are subordinating text A to text B (using A as a point of comparison that reveals B’s originality, say), you may be well served by the “A→A→A→B→B→B” structure.  

     Ultimately, you need to ask yourself how many “A→B” moves you have in you.  Is each one identical?  If so, you may wish to make the transition from A to B only once (“A→A→A→B→B→B”), because if each “A→B” move is identical, the “A→B→A→B→A→B” structure will appear to involve nothing more than directionless oscillation and repetition.  If each is increasingly complex, however—if each AB pair yields a new and progressively more complex idea about your subject—you may be well served by the “A→B→A→B→A→B” structure, because in this case it will be visible to readers as a progressively developing argument.

As we discussed in "Advice on Timing" at the page on single-source analysis, that timeline itself roughly follows the "Sample Sequence of Formative Assignments for a 'Typical' Essay" outlined under " Formative Writing Assignments, " and it spans about 5–6 steps or 2–4 weeks. 

Comparative analysis assignments have a lot of the same DNA as single-source essays, but they potentially bring more reading into play and ask students to engage in more complicated acts of analysis and synthesis during the drafting stages. With that in mind, closer to 4 weeks is probably a good baseline for many single-source analysis assignments. For sections that meet once per week, the timeline will either probably need to expand—ideally—a little past the 4-week side of things, or some of the steps will need to be combined or done asynchronously.

What It Can Build Up To

Comparative analyses can build up to other kinds of writing in a number of ways. For example:

• They can build toward other kinds of comparative analysis, e.g., student can be asked to choose an additional source to complicate their conclusions from a previous analysis, or they can be asked to revisit an analysis using a different axis of comparison, such as race instead of class. (These approaches are akin to moving from a coordinate or subordinate analysis to more of a hybrid approach.)
• They can scaffold up to research essays, which in many instances are an extension of a "hybrid comparative analysis."
• Like single-source analysis, in a course where students will take a "deep dive" into a source or topic for their capstone, they can allow students to "try on" a theoretical approach or genre or time period to see if it's indeed something they want to research more fully.
• DIY Guides for Analytical Writing Assignments

• Types of Assignments
• Unpacking the Elements of Writing Prompts
• Formative Writing Assignments
• Single-Source Analysis
• Research Essays
• Multi-Modal or Creative Projects
• Giving Feedback to Students

Assignment Decoder

To read this content please select one of the options below:

Please note you do not have access to teaching notes, an introduction to comparing comparative methodologies: a framework for understanding pitfalls and operationalizing promises.

Comparative Sciences: Interdisciplinary Approaches

ISBN : 978-1-78350-455-8 , eISBN : 978-1-78350-456-5

Publication date: 7 May 2015

Systematic, consistent, and holistic reflection on comparative methodologies across different disciplines and fields is rare. This chapter, however, develops a framework for both understanding and operationalizing comparative research. First, the basic characteristics of comparison and how it is used in social science research is described. Then, the benefits of comparing for identifying similarities versus differences and the contexts that determine the appropriateness of comparison are discussed. Next, several questions are posed that serve as guides in the operationalization of both the promises and the pitfalls of comparison. Finally, these questions are used to frame both conceptual and practical approaches to inter- as well as intra-disciplinary comparative research.

• Comparative methods
• Comparative science
• Comparative research
• Comparative reflection
• Methodological nationalism

Wiseman, A.W. and Popov, N. (2015), "An Introduction to Comparing Comparative Methodologies: A Framework for Understanding Pitfalls and Operationalizing Promises", Comparative Sciences: Interdisciplinary Approaches ( International Perspectives on Education and Society, Vol. 26 ), Emerald Group Publishing Limited, Leeds, pp. 1-11. https://doi.org/10.1108/S1479-367920140000026001

Emerald Group Publishing Limited

Copyright © 2015 Emerald Group Publishing Limited

All feedback is valuable

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

• Open access
• Published: 07 May 2021

The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions

• Benjamin Hanckel 1 ,
• Mark Petticrew 2 ,
• James Thomas 3 &
• Judith Green 4  

BMC Public Health volume  21 , Article number:  877 ( 2021 ) Cite this article

24k Accesses

50 Citations

33 Altmetric

Metrics details

Qualitative Comparative Analysis (QCA) is a method for identifying the configurations of conditions that lead to specific outcomes. Given its potential for providing evidence of causality in complex systems, QCA is increasingly used in evaluative research to examine the uptake or impacts of public health interventions. We map this emerging field, assessing the strengths and weaknesses of QCA approaches identified in published studies, and identify implications for future research and reporting.

PubMed, Scopus and Web of Science were systematically searched for peer-reviewed studies published in English up to December 2019 that had used QCA methods to identify the conditions associated with the uptake and/or effectiveness of interventions for public health. Data relating to the interventions studied (settings/level of intervention/populations), methods (type of QCA, case level, source of data, other methods used) and reported strengths and weaknesses of QCA were extracted and synthesised narratively.

The search identified 1384 papers, of which 27 (describing 26 studies) met the inclusion criteria. Interventions evaluated ranged across: nutrition/obesity ( n  = 8); physical activity ( n  = 4); health inequalities ( n  = 3); mental health ( n  = 2); community engagement ( n  = 3); chronic condition management ( n  = 3); vaccine adoption or implementation ( n  = 2); programme implementation ( n  = 3); breastfeeding ( n  = 2), and general population health ( n  = 1). The majority of studies ( n  = 24) were of interventions solely or predominantly in high income countries. Key strengths reported were that QCA provides a method for addressing causal complexity; and that it provides a systematic approach for understanding the mechanisms at work in implementation across contexts. Weaknesses reported related to data availability limitations, especially on ineffective interventions. The majority of papers demonstrated good knowledge of cases, and justification of case selection, but other criteria of methodological quality were less comprehensively met.

QCA is a promising approach for addressing the role of context in complex interventions, and for identifying causal configurations of conditions that predict implementation and/or outcomes when there is sufficiently detailed understanding of a series of comparable cases. As the use of QCA in evaluative health research increases, there may be a need to develop advice for public health researchers and journals on minimum criteria for quality and reporting.

Peer Review reports

Interest in the use of Qualitative Comparative Analysis (QCA) arises in part from growing recognition of the need to broaden methodological capacity to address causality in complex systems [ 1 , 2 , 3 ]. Guidance for researchers for evaluating complex interventions suggests process evaluations [ 4 , 5 ] can provide evidence on the mechanisms of change, and the ways in which context affects outcomes. However, this does not address the more fundamental problems with trial and quasi-experimental designs arising from system complexity [ 6 ]. As Byrne notes, the key characteristic of complex systems is ‘emergence’ [ 7 ]: that is, effects may accrue from combinations of components, in contingent ways, which cannot be reduced to any one level. Asking about ‘what works’ in complex systems is not to ask a simple question about whether an intervention has particular effects, but rather to ask: “how the intervention works in relation to all existing components of the system and to other systems and their sub-systems that intersect with the system of interest” [ 7 ]. Public health interventions are typically attempts to effect change in systems that are themselves dynamic; approaches to evaluation are needed that can deal with emergence [ 8 ]. In short, understanding the uptake and impact of interventions requires methods that can account for the complex interplay of intervention conditions and system contexts.

To build a useful evidence base for public health, evaluations thus need to assess not just whether a particular intervention (or component) causes specific change in one variable, in controlled circumstances, but whether those interventions shift systems, and how specific conditions of interventions and setting contexts interact to lead to anticipated outcomes. There have been a number of calls for the development of methods in intervention research to address these issues of complex causation [ 9 , 10 , 11 ], including calls for the greater use of case studies to provide evidence on the important elements of context [ 12 , 13 ]. One approach for addressing causality in complex systems is Qualitative Comparative Analysis (QCA): a systematic way of comparing the outcomes of different combinations of system components and elements of context (‘conditions’) across a series of cases.

The potential of qualitative comparative analysis

QCA is an approach developed by Charles Ragin [ 14 , 15 ], originating in comparative politics and macrosociology to address questions of comparative historical development. Using set theory, QCA methods explore the relationships between ‘conditions’ and ‘outcomes’ by identifying configurations of necessary and sufficient conditions for an outcome. The underlying logic is different from probabilistic reasoning, as the causal relationships identified are not inferred from the (statistical) likelihood of them being found by chance, but rather from comparing sets of conditions and their relationship to outcomes. It is thus more akin to the generative conceptualisations of causality in realist evaluation approaches [ 16 ]. QCA is a non-additive and non-linear method that emphasises diversity, acknowledging that different paths can lead to the same outcome. For evaluative research in complex systems [ 17 ], QCA therefore offers a number of benefits, including: that QCA can identify more than one causal pathway to an outcome (equifinality); that it accounts for conjectural causation (where the presence or absence of conditions in relation to other conditions might be key); and that it is asymmetric with respect to the success or failure of outcomes. That is, that specific factors explain success does not imply that their absence leads to failure (causal asymmetry).

QCA was designed, and is typically used, to compare data from a medium N (10–50) series of cases that include those with and those without the (dichotomised) outcome. Conditions can be dichotomised in ‘crisp sets’ (csQCA) or represented in ‘fuzzy sets’ (fsQCA), where set membership is calibrated (either continuously or with cut offs) between two extremes representing fully in (1) or fully out (0) of the set. A third version, multi-value QCA (mvQCA), infrequently used, represents conditions as ‘multi-value sets’, with multinomial membership [ 18 ]. In calibrating set membership, the researcher specifies the critical qualitative anchors that capture differences in kind (full membership and full non-membership), as well as differences in degree in fuzzy sets (partial membership) [ 15 , 19 ]. Data on outcomes and conditions can come from primary or secondary qualitative and/or quantitative sources. Once data are assembled and coded, truth tables are constructed which “list the logically possible combinations of causal conditions” [ 15 ], collating the number of cases where those configurations occur to see if they share the same outcome. Analysis of these truth tables assesses first whether any conditions are individually necessary or sufficient to predict the outcome, and then whether any configurations of conditions are necessary or sufficient. Necessary conditions are assessed by examining causal conditions shared by cases with the same outcome, whilst identifying sufficient conditions (or combinations of conditions) requires examining cases with the same causal conditions to identify if they have the same outcome [ 15 ]. However, as Legewie argues, the presence of a condition, or a combination of conditions in actual datasets, are likely to be “‘quasi-necessary’ or ‘quasi-sufficient’ in that the causal relation holds in a great majority of cases, but some cases deviate from this pattern” [ 20 ]. Following reduction of the complexity of the model, the final model is tested for coverage (the degree to which a configuration accounts for instances of an outcome in the empirical cases; the proportion of cases belonging to a particular configuration) and consistency (the degree to which the cases sharing a combination of conditions align with a proposed subset relation). The result is an analysis of complex causation, “defined as a situation in which an outcome may follow from several different combinations of causal conditions” [ 15 ] illuminating the ‘causal recipes’, the causally relevant conditions or configuration of conditions that produce the outcome of interest.

QCA, then, has promise for addressing questions of complex causation, and recent calls for the greater use of QCA methods have come from a range of fields related to public health, including health research [ 17 ], studies of social interventions [ 7 ], and policy evaluation [ 21 , 22 ]. In making arguments for the use of QCA across these fields, researchers have also indicated some of the considerations that must be taken into account to ensure robust and credible analyses. There is a need, for instance, to ensure that ‘contradictions’, where cases with the same configurations show different outcomes, are resolved and reported [ 15 , 23 , 24 ]. Additionally, researchers must consider the ratio of cases to conditions, and limit the number of conditions to cases to ensure the validity of models [ 25 ]. Marx and Dusa, examining crisp set QCA, have provided some guidance to the ‘ceiling’ number of conditions which can be included relative to the number of cases to increase the probability of models being valid (that is, with a low probability of being generated through random data) [ 26 ].

There is now a growing body of published research in public health and related fields drawing on QCA methods. This is therefore a timely point to map the field and assess the potential of QCA as a method for contributing to the evidence base for what works in improving public health. To inform future methodological development of robust methods for addressing complexity in the evaluation of public health interventions, we undertook a systematic review to map existing evidence, identify gaps in, and strengths and weakness of, the QCA literature to date, and identify the implications of these for conducting and reporting future QCA studies for public health evaluation. We aimed to address the following specific questions [ 27 ]:

1. How is QCA used for public health evaluation? What populations, settings, methods used in source case studies, unit/s and level of analysis (‘cases’), and ‘conditions’ have been included in QCA studies?

2. What strengths and weaknesses have been identified by researchers who have used QCA to understand complex causation in public health evaluation research?

3. What are the existing gaps in, and strengths and weakness of, the QCA literature in public health evaluation, and what implications do these have for future research and reporting of QCA studies for public health?

This systematic review was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 29 April 2019 ( CRD42019131910 ). A protocol was prepared in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement [ 28 ], and published in 2019 [ 27 ], where the methods are explained in detail. EPPI-Reviewer 4 was used to manage the process and undertake screening of abstracts [ 29 ].

Search strategy

We searched for peer-reviewed published papers in English, which used QCA methods to examine causal complexity in evaluating the implementation, uptake and/or effects of a public health intervention, in any region of the world, for any population. ‘Public health interventions’ were defined as those which aim to promote or protect health, or prevent ill health, in the population. No date exclusions were made, and papers published up to December 2019 were included.

Search strategies used the following phrases “Qualitative Comparative Analysis” and “QCA”, which were combined with the keywords “health”, “public health”, “intervention”, and “wellbeing”. See Additional file  1 for an example. Searches were undertaken on the following databases: PubMed, Web of Science, and Scopus. Additional searches were undertaken on Microsoft Academic and Google Scholar in December 2019, where the first pages of results were checked for studies that may have been missed in the initial search. No additional studies were identified. The list of included studies was sent to experts in QCA methods in health and related fields, including authors of included studies and/or those who had published on QCA methodology. This generated no additional studies within scope, but a suggestion to check the COMPASSS (Comparative Methods for Systematic Cross-Case Analysis) database; this was searched, identifying one further study that met the inclusion criteria [ 30 ]. COMPASSS ( https://compasss.org/ ) collates publications of studies using comparative case analysis.

We excluded studies where no intervention was evaluated, which included studies that used QCA to examine public health infrastructure (i.e. staff training) without a specific health outcome, and papers that report on prevalence of health issues (i.e. prevalence of child mortality). We also excluded studies of health systems or services interventions where there was no public health outcome.

After retrieval, and removal of duplicates, titles and abstracts were screened by one of two authors (BH or JG). Double screening of all records was assisted by EPPI Reviewer 4’s machine learning function. Of the 1384 papers identified after duplicates were removed, we excluded 820 after review of titles and abstracts (Fig.  1 ). The excluded studies included: a large number of papers relating to ‘quantitative coronary angioplasty’ and some which referred to the Queensland Criminal Code (both of which are also abbreviated to ‘QCA’); papers that reported methodological issues but not empirical studies; protocols; and papers that used the phrase ‘qualitative comparative analysis’ to refer to qualitative studies that compared different sub-populations or cases within the study, but did not include formal QCA methods.

Flow Diagram

Full texts of the 51 remaining studies were screened by BH and JG for inclusion, with 10 papers double coded by both authors, with complete agreement. Uncertain inclusions were checked by the third author (MP). Of the full texts, 24 were excluded because: they did not report a public health intervention ( n  = 18); had used a methodology inspired by QCA, but had not undertaken a QCA ( n  = 2); were protocols or methodological papers only ( n  = 2); or were not published in peer-reviewed journals ( n  = 2) (see Fig.  1 ).

Data were extracted manually from the 27 remaining full texts by BH and JG. Two papers relating to the same research question and dataset were combined, such that analysis was by study ( n  = 26) not by paper. We retrieved data relating to: publication (journal, first author country affiliation, funding reported); the study setting (country/region setting, population targeted by the intervention(s)); intervention(s) studied; methods (aims, rationale for using QCA, crisp or fuzzy set QCA, other analysis methods used); data sources drawn on for cases (source [primary data, secondary data, published analyses], qualitative/quantitative data, level of analysis, number of cases, final causal conditions included in the analysis); outcome explained; and claims made about strengths and weaknesses of using QCA (see Table  1 ). Data were synthesised narratively, using thematic synthesis methods [ 31 , 32 ], with interventions categorised by public health domain and level of intervention.

Quality assessment

There are no reporting guidelines for QCA studies in public health, but there are a number of discussions of best practice in the methodological literature [ 25 , 26 , 33 , 34 ]. These discussions suggest several criteria for strengthening QCA methods that we used as indicators of methodological and/or reporting quality: evidence of familiarity of cases; justification for selection of cases; discussion and justification of set membership score calibration; reporting of truth tables; reporting and justification of solution formula; and reporting of consistency and coverage measures. For studies using csQCA, and claiming an explanatory analysis, we additionally identified whether the number of cases was sufficient for the number of conditions included in the model, using a pragmatic cut-off in line with Marx & Dusa’s guideline thresholds, which indicate how many cases are sufficient for given numbers of conditions to reject a 10% probability that models could be generated with random data [ 26 ].

Overview of scope of QCA research in public health

Twenty-seven papers reporting 26 studies were included in the review (Table  1 ). The earliest was published in 2005, and 17 were published after 2015. The majority ( n  = 19) were published in public health/health promotion journals, with the remainder published in other health science ( n  = 3) or in social science/management journals ( n  = 4). The public health domain(s) addressed by each study were broadly coded by the main area of focus. They included nutrition/obesity ( n  = 8); physical activity (PA) (n = 4); health inequalities ( n  = 3); mental health ( n  = 2); community engagement ( n  = 3); chronic condition management ( n  = 3); vaccine adoption or implementation (n = 2); programme implementation ( n  = 3); breastfeeding ( n  = 2); or general population health ( n  = 1). The majority ( n  = 24) of studies were conducted solely or predominantly in high-income countries (systematic reviews in general searched global sources, but commented that the overwhelming majority of studies were from high-income countries). Country settings included: any ( n  = 6); OECD countries ( n  = 3); USA ( n  = 6); UK ( n  = 6) and one each from Nepal, Austria, Belgium, Netherlands and Africa. These largely reflected the first author’s country affiliations in the UK ( n  = 13); USA ( n  = 9); and one each from South Africa, Austria, Belgium, and the Netherlands. All three studies primarily addressing health inequalities [ 35 , 36 , 37 ] were from the UK.

Eight of the interventions evaluated were individual-level behaviour change interventions (e.g. weight management interventions, case management, self-management for chronic conditions); eight evaluated policy/funding interventions; five explored settings-based health promotion/behaviour change interventions (e.g. schools-based physical activity intervention, store-based food choice interventions); three evaluated community empowerment/engagement interventions, and two studies evaluated networks and their impact on health outcomes.

Methods and data sets used

Fifteen studies used crisp sets (csQCA), 11 used fuzzy sets (fsQCA). No study used mvQCA. Eleven studies included additional analyses of the datasets drawn on for the QCA, including six that used qualitative approaches (narrative synthesis, case comparisons), typically to identify cases or conditions for populating the QCA; and four reporting additional statistical analyses (meta-regression, linear regression) to either identify differences overall between cases prior to conducting a QCA (e.g. [ 38 ]) or to explore correlations in more detail (e.g. [ 39 ]). One study used an additional Boolean configurational technique to reduce the number of conditions in the QCA analysis [ 40 ]. No studies reported aiming to compare the findings from the QCA with those from other techniques for evaluating the uptake or effectiveness of interventions, although some [ 41 , 42 ] were explicitly using the study to showcase the possibilities of QCA compared with other approaches in general. Twelve studies drew on primary data collected specifically for the study, with five of those additionally drawing on secondary data sets; five drew only on secondary data sets, and nine used data from systematic reviews of published research. Seven studies drew primarily on qualitative data, generally derived from interviews or observations.

Many studies were undertaken in the context of one or more trials, which provided evidence of effect. Within single trials, this was generally for a process evaluation, with cases being trial sites. Fernald et al’s study, for instance, was in the context of a trial of a programme to support primary care teams in identifying and implementing self-management support tools for their patients, which measured patient and health care provider level outcomes [ 43 ]. The QCA reported here used qualitative data from the trial to identify a set of necessary conditions for health care provider practices to implement the tools successfully. In studies drawing on data from systematic reviews, cases were always at the level of intervention or intervention component, with data included from multiple trials. Harris et al., for instance, undertook a mixed-methods systematic review of school-based self-management interventions for asthma, using meta-analysis methods to identify effective interventions and QCA methods to identify which intervention features were aligned with success [ 44 ].

The largest number of studies ( n  = 10), including all the systematic reviews, analysed cases at the level of the intervention, or a component of the intervention; seven analysed organisational level cases (e.g. school class, network, primary care practice); five analysed sub-national region level cases (e.g. state, local authority area), and two each analysed country or individual level cases. Sample sizes ranged from 10 to 131, with no study having small N (< 10) sample sizes, four having large N (> 50) sample sizes, and the majority (22) being medium N studies (in the range 10–50).

Rationale for using QCA

Most papers reported a rationale for using QCA that mentioned ‘complexity’ or ‘context’, including: noting that QCA is appropriate for addressing causal complexity or multiple pathways to outcome [ 37 , 43 , 45 , 46 , 47 , 48 , 49 , 50 , 51 ]; noting the appropriateness of the method for providing evidence on how context impacts on interventions [ 41 , 50 ]; or the need for a method that addressed causal asymmetry [ 52 ]. Three stated that the QCA was an ‘exploratory’ analysis [ 53 , 54 , 55 ]. In addition to the empirical aims, several papers (e.g. [ 42 , 48 ]) sought to demonstrate the utility of QCA, or to develop QCA methods for health research (e.g. [ 47 ]).

Reported strengths and weaknesses of approach

There was a general agreement about the strengths of QCA. Specifically, that it was a useful tool to address complex causality, providing a systematic approach to understand the mechanisms at work in implementation across contexts [ 38 , 39 , 43 , 45 , 46 , 47 , 55 , 56 , 57 ], particularly as they relate to (in) effective intervention implementation [ 44 , 51 ] and the evaluation of interventions [ 58 ], or “where it is not possible to identify linearity between variables of interest and outcomes” [ 49 ]. Authors highlighted the strengths of QCA as providing possibilities for examining complex policy problems [ 37 , 59 ]; for testing existing as well as new theory [ 52 ]; and for identifying aspects of interventions which had not been previously perceived as critical [ 41 ] or which may have been missed when drawing on statistical methods that use, for instance, linear additive models [ 42 ]. The strengths of QCA in terms of providing useful evidence for policy were flagged in a number of studies, particularly where the causal recipes suggested that conventional assumptions about effectiveness were not confirmed. Blackman et al., for instance, in a series of studies exploring why unequal health outcomes had narrowed in some areas of the UK and not others, identified poorer outcomes in settings with ‘better’ contracting [ 35 , 36 , 37 ]; Harting found, contrary to theoretical assumptions about the necessary conditions for successful implementation of public health interventions, that a multisectoral network was not a necessary condition [ 30 ].

Weaknesses reported included the limitations of QCA in general for addressing complexity, as well as specific limitations with either the csQCA or the fsQCA methods employed. One general concern discussed across a number of studies was the problem of limited empirical diversity, which resulted in: limitations in the possible number of conditions included in each study, particularly with small N studies [ 58 ]; missing data on important conditions [ 43 ]; or limited reported diversity (where, for instance, data were drawn from systematic reviews, reflecting publication biases which limit reporting of ineffective interventions) [ 41 ]. Reported methodological limitations in small and intermediate N studies included concerns about the potential that case selection could bias findings [ 37 ].

In terms of potential for addressing causal complexity, the limitations of QCA for identifying unintended consequences, tipping points, and/or feedback loops in complex adaptive systems were noted [ 60 ], as were the potential limitations (especially in csQCA studies) of reducing complex conditions, drawn from detailed qualitative understanding, to binary conditions [ 35 ]. The impossibility of doing this was a rationale for using fsQCA in one study [ 57 ], where detailed knowledge of conditions is needed to make theoretically justified calibration decisions. However, others [ 47 ] make the case that csQCA provides more appropriate findings for policy: dichotomisation forces a focus on meaningful distinctions, including those related to decisions that practitioners/policy makers can action. There is, then, a potential trade-off in providing ‘interpretable results’, but ones which preclude potential for utilising more detailed information [ 45 ]. That QCA does not deal with probabilistic causation was noted [ 47 ].

Quality of published studies

Assessment of ‘familiarity with cases’ was made subjectively on the basis of study authors’ reports of their knowledge of the settings (empirical or theoretical) and the descriptions they provided in the published paper: overall, 14 were judged as sufficient, and 12 less than sufficient. Studies which included primary data were more likely to be judged as demonstrating familiarity ( n  = 10) than those drawing on secondary sources or systematic reviews, of which only two were judged as demonstrating familiarity. All studies justified how the selection of cases had been made; for those not using the full available population of cases, this was in general (appropriately) done theoretically: following previous research [ 52 ]; purposively to include a range of positive and negative outcomes [ 41 ]; or to include a diversity of cases [ 58 ]. In identifying conditions leading to effective/not effective interventions, one purposive strategy was to include a specified percentage or number of the most effective and least effective interventions (e.g. [ 36 , 40 , 51 , 52 ]). Discussion of calibration of set membership scores was judged adequate in 15 cases, and inadequate in 11; 10 reported raw data matrices in the paper or supplementary material; 21 reported truth tables in the paper or supplementary material. The majority ( n  = 21) reported at least some detail on the coverage (the number of cases with a particular configuration) and consistency (the percentage of similar causal configurations which result in the same outcome). The majority ( n  = 21) included truth tables (or explicitly provided details of how to obtain them); fewer ( n  = 10) included raw data. Only five studies met all six of these quality criteria (evidence of familiarity with cases, justification of case selection, discussion of calibration, reporting truth tables, reporting raw data matrices, reporting coverage and consistency); a further six met at least five of them.

Of the csQCA studies which were not reporting an exploratory analysis, four appeared to have insufficient cases for the large number of conditions entered into at least one of the models reported, with a consequent risk to the validity of the QCA models [ 26 ].

QCA has been widely used in public health research over the last decade to advance understanding of causal inference in complex systems. In this review of published evidence to date, we have identified studies using QCA to examine the configurations of conditions that lead to particular outcomes across contexts. As noted by most study authors, QCA methods have promised advantages over probabilistic statistical techniques for examining causation where systems and/or interventions are complex, providing public health researchers with a method to test the multiple pathways (configurations of conditions), and necessary and sufficient conditions that lead to desired health outcomes.

The origins of QCA approaches are in comparative policy studies. Rihoux et al’s review of peer-reviewed journal articles using QCA methods published up to 2011 found the majority of published examples were from political science and sociology, with fewer than 5% of the 313 studies they identified coming from health sciences [ 61 ]. They also reported few examples of the method being used in policy evaluation and implementation studies [ 62 ]. In the decade since their review of the field [ 61 ], there has been an emerging body of evaluative work in health: we identified 26 studies in the field of public health alone, with the majority published in public health journals. Across these studies, QCA has been used for evaluative questions in a range of settings and public health domains to identify the conditions under which interventions are implemented and/or have evidence of effect for improving population health. All studies included a series of cases that included some with and some without the outcome of interest (such as behaviour change, successful programme implementation, or good vaccination uptake). The dominance of high-income countries in both intervention settings and author affiliations is disappointing, but reflects the disproportionate location of public health research in the global north more generally [ 63 ].

The largest single group of studies included were systematic reviews, using QCA to compare interventions (or intervention components) to identify successful (and non-successful) configurations of conditions across contexts. Here, the value of QCA lies in its potential for synthesis with quantitative meta-synthesis methods to identify the particular conditions or contexts in which interventions or components are effective. As Parrott et al. note, for instance, their meta-analysis could identify probabilistic effects of weight management programmes, and the QCA analysis enabled them to address the “role that the context of the [paediatric weight management] intervention has in influencing how, when, and for whom an intervention mix will be successful” [ 50 ]. However, using QCA to identify configurations of conditions that lead to effective or non- effective interventions across particular areas of population health is an application that does move away in some significant respects from the origins of the method. First, researchers drawing on evidence from systematic reviews for their data are reliant largely on published evidence for information on conditions (such as the organisational contexts in which interventions were implemented, or the types of behaviour change theory utilised). Although guidance for describing interventions [ 64 ] advises key aspects of context are included in reports, this may not include data on the full range of conditions that might be causally important, and review research teams may have limited knowledge of these ‘cases’ themselves. Second, less successful interventions are less likely to be published, potentially limiting the diversity of cases, particularly of cases with unsuccessful outcomes. A strength of QCA is the separate analysis of conditions leading to positive and negative outcomes: this is precluded where there is insufficient evidence on negative outcomes [ 50 ]. Third, when including a range of types of intervention, it can be unclear whether the cases included are truly comparable. A QCA study requires a high degree of theoretical and pragmatic case knowledge on the part of the researcher to calibrate conditions to qualitative anchors: it is reliant on deep understanding of complex contexts, and a familiarity with how conditions interact within and across contexts. Perhaps surprising is that only seven of the studies included here clearly drew on qualitative data, given that QCA is primarily seen as a method that requires thick, detailed knowledge of cases, particularly when the aim is to understand complex causation [ 8 ]. Whilst research teams conducting QCA in the context of systematic reviews may have detailed understanding in general of interventions within their spheres of expertise, they are unlikely to have this for the whole range of cases, particularly where a diverse set of contexts (countries, organisational settings) are included. Making a theoretical case for the valid comparability of such a case series is crucial. There may, then, be limitations in the portability of QCA methods for conducting studies entirely reliant on data from published evidence.

QCA was developed for small and medium N series of cases, and (as in the field more broadly, [ 61 ]), the samples in our studies predominantly had between 10 and 50 cases. However, there is increasing interest in the method as an alternative or complementary technique to regression-oriented statistical methods for larger samples [ 65 ], such as from surveys, where detailed knowledge of cases is likely to be replaced by theoretical knowledge of relationships between conditions (see [ 23 ]). The two larger N (> 100 cases) studies in our sample were an individual level analysis of survey data [ 46 , 47 ] and an analysis of intervention arms from a systematic review [ 50 ]. Larger sample sizes allow more conditions to be included in the analysis [ 23 , 26 ], although for evaluative research, where the aim is developing a causal explanation, rather than simply exploring patterns, there remains a limit to the number of conditions that can be included. As the number of conditions included increases, so too does the number of possible configurations, increasing the chance of unique combinations and of generating spurious solutions with a high level of consistency. As a rule of thumb, once the number of conditions exceeds 6–8 (with up to 50 cases) or 10 (for larger samples), the credibility of solutions may be severely compromised [ 23 ].

Strengths and weaknesses of the study

A systematic review has the potential advantages of transparency and rigour and, if not exhaustive, our search is likely to be representative of the body of research using QCA for evaluative public health research up to 2020. However, a limitation is the inevitable difficulty in operationalising a ‘public health’ intervention. Exclusions on scope are not straightforward, given that most social, environmental and political conditions impact on public health, and arguably a greater range of policy and social interventions (such as fiscal or trade policies) that have been the subject of QCA analyses could have been included, or a greater range of more clinical interventions. However, to enable a manageable number of papers to review, and restrict our focus to those papers that were most directly applicable to (and likely to be read by) those in public health policy and practice, we operationalised ‘public health interventions’ as those which were likely to be directly impacting on population health outcomes, or on behaviours (such as increased physical activity) where there was good evidence for causal relationships with public health outcomes, and where the primary research question of the study examined the conditions leading to those outcomes. This review has, of necessity, therefore excluded a considerable body of evidence likely to be useful for public health practice in terms of planning interventions, such as studies on how to better target smoking cessation [ 66 ] or foster social networks [ 67 ] where the primary research question was on conditions leading to these outcomes, rather than on conditions for outcomes of specific interventions. Similarly, there are growing number of descriptive epidemiological studies using QCA to explore factors predicting outcomes across such diverse areas as lupus and quality of life [ 68 ]; length of hospital stay [ 69 ]; constellations of factors predicting injury [ 70 ]; or the role of austerity, crisis and recession in predicting public health outcomes [ 71 ]. Whilst there is undoubtedly useful information to be derived from studying the conditions that lead to particular public health problems, these studies were not directly evaluating interventions, so they were also excluded.

Restricting our search to publications in English and to peer reviewed publications may have missed bodies of work from many regions, and has excluded research from non-governmental organisations using QCA methods in evaluation. As this is a rapidly evolving field, with relatively recent uptake in public health (all our included studies were after 2005), our studies may not reflect the most recent advances in the area.

Implications for conducting and reporting QCA studies

This systematic review has reviewed studies that deployed an emergent methodology, which has no reporting guidelines and has had, to date, a relatively low level of awareness among many potential evidence users in public health. For this reason, many of the studies reviewed were relatively detailed on the methods used, and the rationale for utilising QCA.

We did not assess quality directly, but used indicators of good practice discussed in QCA methodological literature, largely written for policy studies scholars, and often post-dating the publication dates of studies included in this review. It is also worth noting that, given the relatively recent development of QCA methods, methodological debate is still thriving on issues such as the reliability of causal inferences [ 72 ], alongside more general critiques of the usefulness of the method for policy decisions (see, for instance, [ 73 ]). The authors of studies included in this review also commented directly on methodological development: for instance, Thomas et al. suggests that QCA may benefit from methods development for sensitivity analyses around calibration decisions [ 42 ].

However, we selected quality criteria that, we argue, are relevant for public health research> Justifying the selection of cases, discussing and justifying the calibration of set membership, making data sets available, and reporting truth tables, consistency and coverage are all good practice in line with the usual requirements of transparency and credibility in methods. When QCA studies aim to provide explanation of outcomes (rather than exploring configurations), it is also vital that they are reported in ways that enhance the credibility of claims made, including justifying the number of conditions included relative to cases. Few of the studies published to date met all these criteria, at least in the papers included here (although additional material may have been provided in other publications). To improve the future discoverability and uptake up of QCA methods in public health, and to strengthen the credibility of findings from these methods, we therefore suggest the following criteria should be considered by authors and reviewers for reporting QCA studies which aim to provide causal evidence about the configurations of conditions that lead to implementation or outcomes:

The paper title and abstract state the QCA design;

The sampling unit for the ‘case’ is clearly defined (e.g.: patient, specified geographical population, ward, hospital, network, policy, country);

The population from which the cases have been selected is defined (e.g.: all patients in a country with X condition, districts in X country, tertiary hospitals, all hospitals in X country, all health promotion networks in X province, European policies on smoking in outdoor places, OECD countries);

The rationale for selection of cases from the population is justified (e.g.: whole population, random selection, purposive sample);

There are sufficient cases to provide credible coverage across the number of conditions included in the model, and the rationale for the number of conditions included is stated;

Cases are comparable;

There is a clear justification for how choices of relevant conditions (or ‘aspects of context’) have been made;

There is sufficient transparency for replicability: in line with open science expectations, datasets should be available where possible; truth tables should be reported in publications, and reports of coverage and consistency provided.

Implications for future research

In reviewing methods for evaluating natural experiments, Craig et al. focus on statistical techniques for enhancing causal inference, noting only that what they call ‘qualitative’ techniques (the cited references for these are all QCA studies) require “further studies … to establish their validity and usefulness” [ 2 ]. The studies included in this review have demonstrated that QCA is a feasible method when there are sufficient (comparable) cases for identifying configurations of conditions under which interventions are effective (or not), or are implemented (or not). Given ongoing concerns in public health about how best to evaluate interventions across complex contexts and systems, this is promising. This review has also demonstrated the value of adding QCA methods to the tool box of techniques for evaluating interventions such as public policies, health promotion programmes, and organisational changes - whether they are implemented in a randomised way or not. Many of the studies in this review have clearly generated useful evidence: whether this evidence has had more or less impact, in terms of influencing practice and policy, or is more valid, than evidence generated by other methods is not known. Validating the findings of a QCA study is perhaps as challenging as validating the findings from any other design, given the absence of any gold standard comparators. Comparisons of the findings of QCA with those from other methods are also typically constrained by the rather different research questions asked, and the different purposes of the analysis. In our review, QCA were typically used alongside other methods to address different questions, rather than to compare methods. However, as the field develops, follow up studies, which evaluate outcomes of interventions designed in line with conditions identified as causal in prior QCAs, might be useful for contributing to validation.

This review was limited to public health evaluation research: other domains that would be useful to map include health systems/services interventions and studies used to design or target interventions. There is also an opportunity to broaden the scope of the field, particularly for addressing some of the more intractable challenges for public health research. Given the limitations in the evidence base on what works to address inequalities in health, for instance [ 74 ], QCA has potential here, to help identify the conditions under which interventions do or do not exacerbate unequal outcomes, or the conditions that lead to differential uptake or impacts across sub-population groups. It is perhaps surprising that relatively few of the studies in this review included cases at the level of country or region, the traditional level for QCA studies. There may be scope for developing international comparisons for public health policy, and using QCA methods at the case level (nation, sub-national region) of classic policy studies in the field. In the light of debate around COVID-19 pandemic response effectiveness, comparative studies across jurisdictions might shed light on issues such as differential population responses to vaccine uptake or mask use, for example, and these might in turn be considered as conditions in causal configurations leading to differential morbidity or mortality outcomes.

When should be QCA be considered?

Public health evaluations typically assess the efficacy, effectiveness or cost-effectiveness of interventions and the processes and mechanisms through which they effect change. There is no perfect evaluation design for achieving these aims. As in other fields, the choice of design will in part depend on the availability of counterfactuals, the extent to which the investigator can control the intervention, and the range of potential cases and contexts [ 75 ], as well as political considerations, such as the credibility of the approach with key stakeholders [ 76 ]. There are inevitably ‘horses for courses’ [ 77 ]. The evidence from this review suggests that QCA evaluation approaches are feasible when there is a sufficient number of comparable cases with and without the outcome of interest, and when the investigators have, or can generate, sufficiently in-depth understanding of those cases to make sense of connections between conditions, and to make credible decisions about the calibration of set membership. QCA may be particularly relevant for understanding multiple causation (that is, where different configurations might lead to the same outcome), and for understanding the conditions associated with both lack of effect and effect. As a stand-alone approach, QCA might be particularly valuable for national and regional comparative studies of the impact of policies on public health outcomes. Alongside cluster randomised trials of interventions, or alongside systematic reviews, QCA approaches are especially useful for identifying core combinations of causal conditions for success and lack of success in implementation and outcome.

Conclusions

QCA is a relatively new approach for public health research, with promise for contributing to much-needed methodological development for addressing causation in complex systems. This review has demonstrated the large range of evaluation questions that have been addressed to date using QCA, including contributions to process evaluations of trials and for exploring the conditions leading to effectiveness (or not) in systematic reviews of interventions. There is potential for QCA to be more widely used in evaluative research, to identify the conditions under which interventions across contexts are implemented or not, and the configurations of conditions associated with effect or lack of evidence of effect. However, QCA will not be appropriate for all evaluations, and cannot be the only answer to addressing complex causality. For explanatory questions, the approach is most appropriate when there is a series of enough comparable cases with and without the outcome of interest, and where the researchers have detailed understanding of those cases, and conditions. To improve the credibility of findings from QCA for public health evidence users, we recommend that studies are reported with the usual attention to methodological transparency and data availability, with key details that allow readers to judge the credibility of causal configurations reported. If the use of QCA continues to expand, it may be useful to develop more comprehensive consensus guidelines for conduct and reporting.

Availability of data and materials

Full search strategies and extraction forms are available by request from the first author.

Abbreviations

Comparative Methods for Systematic Cross-Case Analysis

crisp set QCA

fuzzy set QCA

multi-value QCA

Medical Research Council

• Qualitative Comparative Analysis

randomised control trial

Physical Activity

Green J, Roberts H, Petticrew M, Steinbach R, Goodman A, Jones A, et al. Integrating quasi-experimental and inductive designs in evaluation: a case study of the impact of free bus travel on public health. Evaluation. 2015;21(4):391–406. https://doi.org/10.1177/1356389015605205 .

Article   Google Scholar  

Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017;38(1):39–56. https://doi.org/10.1146/annurev-publhealth-031816-044327 .

Article   PubMed   PubMed Central   Google Scholar  

Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3. https://doi.org/10.1136/bmj.39569.510521.AD .

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350(mar19 6):h1258. https://doi.org/10.1136/bmj.h1258 .

Pattyn V, Álamos-Concha P, Cambré B, Rihoux B, Schalembier B. Policy effectiveness through Configurational and mechanistic lenses: lessons for concept development. J Comp Policy Anal Res Pract. 2020;0:1–18.

Google Scholar  

Byrne D. Evaluating complex social interventions in a complex world. Evaluation. 2013;19(3):217–28. https://doi.org/10.1177/1356389013495617 .

Gerrits L, Pagliarin S. Social and causal complexity in qualitative comparative analysis (QCA): strategies to account for emergence. Int J Soc Res Methodol 2020;0:1–14, doi: https://doi.org/10.1080/13645579.2020.1799636 .

Grant RL, Hood R. Complex systems, explanation and policy: implications of the crisis of replication for public health research. Crit Public Health. 2017;27(5):525–32. https://doi.org/10.1080/09581596.2017.1282603 .

Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4. https://doi.org/10.1016/S0140-6736(17)31267-9 .

Article   PubMed   Google Scholar  

Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95. https://doi.org/10.1186/s12916-018-1089-4 .

Craig P, Di Ruggiero E, Frohlich KL, Mykhalovskiy E and White M, on behalf of the Canadian Institutes of Health Research (CIHR)–National Institute for Health Research (NIHR) Context Guidance Authors Group. Taking account of context in population health intervention research: guidance for producers, users and funders of research. Southampton: NIHR Evaluation, Trials and Studies Coordinating Centre; 2018.

Paparini S, Green J, Papoutsi C, Murdoch J, Petticrew M, Greenhalgh T, et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med. 2020;18(1):301. https://doi.org/10.1186/s12916-020-01777-6 .

Ragin. The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Berkeley: University of California Press; 1987.

Ragin C. Redesigning social inquiry: fuzzy sets and beyond - Charles C: Ragin - Google Books. The University of Chicago Press; 2008. https://doi.org/10.7208/chicago/9780226702797.001.0001 .

Book   Google Scholar  

Befani B, Ledermann S, Sager F. Realistic evaluation and QCA: conceptual parallels and an empirical application. Evaluation. 2007;13(2):171–92. https://doi.org/10.1177/1356389007075222 .

Kane H, Lewis MA, Williams PA, Kahwati LC. Using qualitative comparative analysis to understand and quantify translation and implementation. Transl Behav Med. 2014;4(2):201–8. https://doi.org/10.1007/s13142-014-0251-6 .

Cronqvist L, Berg-Schlosser D. Chapter 4: Multi-Value QCA (mvQCA). In: Rihoux B, Ragin C, editors. Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. 2455 Teller Road, Thousand Oaks California 91320 United States: SAGE Publications, Inc.; 2009. p. 69–86. doi: https://doi.org/10.4135/9781452226569 .

Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 1999;34(5 Pt 2):1225–39.

CAS   PubMed   PubMed Central   Google Scholar  

Legewie N. An introduction to applied data analysis with qualitative comparative analysis (QCA). Forum Qual Soc Res. 2013;14.  https://doi.org/10.17169/fqs-14.3.1961 .

Varone F, Rihoux B, Marx A. A new method for policy evaluation? In: Rihoux B, Grimm H, editors. Innovative comparative methods for policy analysis: beyond the quantitative-qualitative divide. Boston: Springer US; 2006. p. 213–36. https://doi.org/10.1007/0-387-28829-5_10 .

Chapter   Google Scholar  

Gerrits L, Verweij S. The evaluation of complex infrastructure projects: a guide to qualitative comparative analysis. Cheltenham: Edward Elgar Pub; 2018. https://doi.org/10.4337/9781783478422 .

Greckhamer T, Misangyi VF, Fiss PC. The two QCAs: from a small-N to a large-N set theoretic approach. In: Configurational Theory and Methods in Organizational Research. Emerald Group Publishing Ltd.; 2013. p. 49–75. https://pennstate.pure.elsevier.com/en/publications/the-two-qcas-from-a-small-n-to-a-large-n-set-theoretic-approach . Accessed 16 Apr 2021.

Rihoux B, Ragin CC. Configurational comparative methods: qualitative comparative analysis (QCA) and related techniques. SAGE; 2009, doi: https://doi.org/10.4135/9781452226569 .

Marx A. Crisp-set qualitative comparative analysis (csQCA) and model specification: benchmarks for future csQCA applications. Int J Mult Res Approaches. 2010;4(2):138–58. https://doi.org/10.5172/mra.2010.4.2.138 .

Marx A, Dusa A. Crisp-set qualitative comparative analysis (csQCA), contradictions and consistency benchmarks for model specification. Methodol Innov Online. 2011;6(2):103–48. https://doi.org/10.4256/mio.2010.0037 .

Hanckel B, Petticrew M, Thomas J, Green J. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev. 2019;8(1):252. https://doi.org/10.1186/s13643-019-1159-5 .

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349(1):g7647. https://doi.org/10.1136/bmj.g7647 .

EPPI-Reviewer 4.0: Software for research synthesis. UK: University College London; 2010.

Harting J, Peters D, Grêaux K, van Assema P, Verweij S, Stronks K, et al. Implementing multiple intervention strategies in Dutch public health-related policy networks. Health Promot Int. 2019;34(2):193–203. https://doi.org/10.1093/heapro/dax067 .

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45. https://doi.org/10.1186/1471-2288-8-45 .

Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews: a product from the ESRC methods Programme. 2006.

Wagemann C, Schneider CQ. Qualitative comparative analysis (QCA) and fuzzy-sets: agenda for a research approach and a data analysis technique. Comp Sociol. 2010;9:376–96.

Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis: Cambridge University Press; 2012. https://doi.org/10.1017/CBO9781139004244 .

Blackman T, Dunstan K. Qualitative comparative analysis and health inequalities: investigating reasons for differential Progress with narrowing local gaps in mortality. J Soc Policy. 2010;39(3):359–73. https://doi.org/10.1017/S0047279409990675 .

Blackman T, Wistow J, Byrne D. A Qualitative Comparative Analysis of factors associated with trends in narrowing health inequalities in England. Soc Sci Med 1982. 2011;72:1965–74.

Blackman T, Wistow J, Byrne D. Using qualitative comparative analysis to understand complex policy problems. Evaluation. 2013;19(2):126–40. https://doi.org/10.1177/1356389013484203 .

Glatman-Freedman A, Cohen M-L, Nichols KA, Porges RF, Saludes IR, Steffens K, et al. Factors affecting the introduction of new vaccines to poor nations: a comparative study of the haemophilus influenzae type B and hepatitis B vaccines. PLoS One. 2010;5(11):e13802. https://doi.org/10.1371/journal.pone.0013802 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ford EW, Duncan WJ, Ginter PM. Health departments’ implementation of public health’s core functions: an assessment of health impacts. Public Health. 2005;119(1):11–21. https://doi.org/10.1016/j.puhe.2004.03.002 .

Article   CAS   PubMed   Google Scholar  

Lucidarme S, Cardon G, Willem A. A comparative study of health promotion networks: configurations of determinants for network effectiveness. Public Manag Rev. 2016;18(8):1163–217. https://doi.org/10.1080/14719037.2015.1088567 .

Melendez-Torres GJ, Sutcliffe K, Burchett HED, Rees R, Richardson M, Thomas J. Weight management programmes: re-analysis of a systematic review to identify pathways to effectiveness. Health Expect Int J Public Particip Health Care Health Policy. 2018;21:574–84.

CAS   Google Scholar  

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3(1):67. https://doi.org/10.1186/2046-4053-3-67 .

Fernald DH, Simpson MJ, Nease DE, Hahn DL, Hoffmann AE, Michaels LC, et al. Implementing community-created self-management support tools in primary care practices: multimethod analysis from the INSTTEPP study. J Patient-Centered Res Rev. 2018;5(4):267–75. https://doi.org/10.17294/2330-0698.1634 .

Harris K, Kneale D, Lasserson TJ, McDonald VM, Grigg J, Thomas J. School-based self-management interventions for asthma in children and adolescents: a mixed methods systematic review. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD011651.pub2 .

Kahwati LC, Lewis MA, Kane H, Williams PA, Nerz P, Jones KR, et al. Best practices in the veterans health Administration’s MOVE! Weight management program. Am J Prev Med. 2011;41(5):457–64. https://doi.org/10.1016/j.amepre.2011.06.047 .

Warren J, Wistow J, Bambra C. Applying qualitative comparative analysis (QCA) to evaluate a public health policy initiative in the north east of England. Polic Soc. 2013;32(4):289–301. https://doi.org/10.1016/j.polsoc.2013.10.002 .

Warren J, Wistow J, Bambra C. Applying qualitative comparative analysis (QCA) in public health: a case study of a health improvement service for long-term incapacity benefit recipients. J Public Health. 2014;36(1):126–33. https://doi.org/10.1093/pubmed/fdt047 .

Article   CAS   Google Scholar  

Brunton G, O’Mara-Eves A, Thomas J. The “active ingredients” for successful community engagement with disadvantaged expectant and new mothers: a qualitative comparative analysis. J Adv Nurs. 2014;70(12):2847–60. https://doi.org/10.1111/jan.12441 .

McGowan VJ, Wistow J, Lewis SJ, Popay J, Bambra C. Pathways to mental health improvement in a community-led area-based empowerment initiative: evidence from the big local ‘communities in control’ study. England J Public Health. 2019;41(4):850–7. https://doi.org/10.1093/pubmed/fdy192 .

Parrott JS, Henry B, Thompson KL, Ziegler J, Handu D. Managing Complexity in Evidence Analysis: A Worked Example in Pediatric Weight Management. J Acad Nutr Diet. 2018;118:1526–1542.e3.

Kien C, Grillich L, Nussbaumer-Streit B, Schoberberger R. Pathways leading to success and non-success: a process evaluation of a cluster randomized physical activity health promotion program applying fuzzy-set qualitative comparative analysis. BMC Public Health. 2018;18(1):1386. https://doi.org/10.1186/s12889-018-6284-x .

Lubold AM. The effect of family policies and public health initiatives on breastfeeding initiation among 18 high-income countries: a qualitative comparative analysis research design. Int Breastfeed J. 2017;12(1):34. https://doi.org/10.1186/s13006-017-0122-0 .

Bianchi F, Garnett E, Dorsel C, Aveyard P, Jebb SA. Restructuring physical micro-environments to reduce the demand for meat: a systematic review and qualitative comparative analysis. Lancet Planet Health. 2018;2(9):e384–97. https://doi.org/10.1016/S2542-5196(18)30188-8 .

Bianchi F, Dorsel C, Garnett E, Aveyard P, Jebb SA. Interventions targeting conscious determinants of human behaviour to reduce the demand for meat: a systematic review with qualitative comparative analysis. Int J Behav Nutr Phys Act. 2018;15(1):102. https://doi.org/10.1186/s12966-018-0729-6 .

Hartmann-Boyce J, Bianchi F, Piernas C, Payne Riches S, Frie K, Nourse R, et al. Grocery store interventions to change food purchasing behaviors: a systematic review of randomized controlled trials. Am J Clin Nutr. 2018;107(6):1004–16. https://doi.org/10.1093/ajcn/nqy045 .

Burchett HED, Sutcliffe K, Melendez-Torres GJ, Rees R, Thomas J. Lifestyle weight management programmes for children: a systematic review using qualitative comparative analysis to identify critical pathways to effectiveness. Prev Med. 2018;106:1–12. https://doi.org/10.1016/j.ypmed.2017.08.025 .

Chiappone A. Technical assistance and changes in nutrition and physical activity practices in the National Early Care and education learning Collaboratives project, 2015–2016. Prev Chronic Dis. 2018;15. https://doi.org/10.5888/pcd15.170239 .

Kane H, Hinnant L, Day K, Council M, Tzeng J, Soler R, et al. Pathways to program success: a qualitative comparative analysis (QCA) of communities putting prevention to work case study programs. J Public Health Manag Pract JPHMP. 2017;23(2):104–11. https://doi.org/10.1097/PHH.0000000000000449 .

Roberts MC, Murphy T, Moss JL, Wheldon CW, Psek W. A qualitative comparative analysis of combined state health policies related to human papillomavirus vaccine uptake in the United States. Am J Public Health. 2018;108(4):493–9. https://doi.org/10.2105/AJPH.2017.304263 .

Breuer E, Subba P, Luitel N, Jordans M, Silva MD, Marchal B, et al. Using qualitative comparative analysis and theory of change to unravel the effects of a mental health intervention on service utilisation in Nepal. BMJ Glob Health. 2018;3(6):e001023. https://doi.org/10.1136/bmjgh-2018-001023 .

Rihoux B, Álamos-Concha P, Bol D, Marx A, Rezsöhazy I. From niche to mainstream method? A comprehensive mapping of QCA applications in journal articles from 1984 to 2011. Polit Res Q. 2013;66:175–84.

Rihoux B, Rezsöhazy I, Bol D. Qualitative comparative analysis (QCA) in public policy analysis: an extensive review. Ger Policy Stud. 2011;7:9–82.

Plancikova D, Duric P, O’May F. High-income countries remain overrepresented in highly ranked public health journals: a descriptive analysis of research settings and authorship affiliations. Crit Public Health 2020;0:1–7, DOI: https://doi.org/10.1080/09581596.2020.1722313 .

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348(mar07 3):g1687. https://doi.org/10.1136/bmj.g1687 .

Fiss PC, Sharapov D, Cronqvist L. Opposites attract? Opportunities and challenges for integrating large-N QCA and econometric analysis. Polit Res Q. 2013;66:191–8.

Blackman T. Can smoking cessation services be better targeted to tackle health inequalities? Evidence from a cross-sectional study. Health Educ J. 2008;67(2):91–101. https://doi.org/10.1177/0017896908089388 .

Haynes P, Banks L, Hill M. Social networks amongst older people in OECD countries: a qualitative comparative analysis. J Int Comp Soc Policy. 2013;29(1):15–27. https://doi.org/10.1080/21699763.2013.802988 .

Rioja EC. Valero-Moreno S, Giménez-Espert M del C, Prado-Gascó V. the relations of quality of life in patients with lupus erythematosus: regression models versus qualitative comparative analysis. J Adv Nurs. 2019;75(7):1484–92. https://doi.org/10.1111/jan.13957 .

Dy SM. Garg Pushkal, Nyberg Dorothy, Dawson Patricia B., Pronovost Peter J., Morlock Laura, et al. critical pathway effectiveness: assessing the impact of patient, hospital care, and pathway characteristics using qualitative comparative analysis. Health Serv Res. 2005;40(2):499–516. https://doi.org/10.1111/j.1475-6773.2005.0r370.x .

MELINDER KA, ANDERSSON R. The impact of structural factors on the injury rate in different European countries. Eur J Pub Health. 2001;11(3):301–8. https://doi.org/10.1093/eurpub/11.3.301 .

Saltkjel T, Holm Ingelsrud M, Dahl E, Halvorsen K. A fuzzy set approach to economic crisis, austerity and public health. Part II: How are configurations of crisis and austerity related to changes in population health across Europe? Scand J Public Health. 2017;45(18_suppl):48–55.

Baumgartner M, Thiem A. Often trusted but never (properly) tested: evaluating qualitative comparative analysis. Sociol Methods Res. 2020;49(2):279–311. https://doi.org/10.1177/0049124117701487 .

Tanner S. QCA is of questionable value for policy research. Polic Soc. 2014;33(3):287–98. https://doi.org/10.1016/j.polsoc.2014.08.003 .

Mackenbach JP. Tackling inequalities in health: the need for building a systematic evidence base. J Epidemiol Community Health. 2003;57(3):162. https://doi.org/10.1136/jech.57.3.162 .

Stern E, Stame N, Mayne J, Forss K, Davies R, Befani B. Broadening the range of designs and methods for impact evaluations. Technical report. London: DfiD; 2012.

Pattyn V. Towards appropriate impact evaluation methods. Eur J Dev Res. 2019;31(2):174–9. https://doi.org/10.1057/s41287-019-00202-w .

Petticrew M, Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health. 2003;57(7):527–9. https://doi.org/10.1136/jech.57.7.527 .

Download references

Acknowledgements

The authors would like to thank and acknowledge the support of Sara Shaw, PI of MR/S014632/1 and the rest of the Triple C project team, the experts who were consulted on the final list of included studies, and the reviewers who provided helpful feedback on the original submission.

This study was funded by MRC: MR/S014632/1 ‘Case study, context and complex interventions (Triple C): development of guidance and publication standards to support case study research’. The funder played no part in the conduct or reporting of the study. JG is supported by a Wellcome Trust Centre grant 203109/Z/16/Z.

Author information

Authors and affiliations.

Institute for Culture and Society, Western Sydney University, Sydney, Australia

Benjamin Hanckel

Department of Public Health, Environments and Society, LSHTM, London, UK

Mark Petticrew

UCL Institute of Education, University College London, London, UK

James Thomas

Wellcome Centre for Cultures & Environments of Health, University of Exeter, Exeter, UK

Judith Green

You can also search for this author in PubMed   Google Scholar

Contributions

BH - research design, data acquisition, data extraction and coding, data interpretation, paper drafting; JT – research design, data interpretation, contributing to paper; MP – funding acquisition, research design, data interpretation, contributing to paper; JG – funding acquisition, research design, data extraction and coding, data interpretation, paper drafting. All authors approved the final version.

Corresponding author

Correspondence to Judith Green .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Competing interests

All authors declare they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Example search strategy.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hanckel, B., Petticrew, M., Thomas, J. et al. The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions. BMC Public Health 21 , 877 (2021). https://doi.org/10.1186/s12889-021-10926-2

Download citation

Received : 03 February 2021

Accepted : 22 April 2021

Published : 07 May 2021

DOI : https://doi.org/10.1186/s12889-021-10926-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public health
• Intervention
• Systematic review

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Comparative effectiveness research for the clinician researcher: a framework for making a methodological design choice

Cylie m. williams.

1 Peninsula Health, Community Health, PO Box 52, Frankston, Melbourne, Victoria 3199 Australia

2 Monash University, School of Physiotherapy, Melbourne, Australia

3 Monash Health, Allied Health Research Unit, Melbourne, Australia

Elizabeth H. Skinner

4 Western Health, Allied Health, Melbourne, Australia

Alicia M. James

Jill l. cook, steven m. mcphail.

5 Queensland University of Technology, School of Public Health and Social Work, Brisbane, Australia

Terry P. Haines

Comparative effectiveness research compares two active forms of treatment or usual care in comparison with usual care with an additional intervention element. These types of study are commonly conducted following a placebo or no active treatment trial. Research designs with a placebo or non-active treatment arm can be challenging for the clinician researcher when conducted within the healthcare environment with patients attending for treatment.

A framework for conducting comparative effectiveness research is needed, particularly for interventions for which there are no strong regulatory requirements that must be met prior to their introduction into usual care. We argue for a broader use of comparative effectiveness research to achieve translatable real-world clinical research. These types of research design also affect the rapid uptake of evidence-based clinical practice within the healthcare setting.

This framework includes questions to guide the clinician researcher into the most appropriate trial design to measure treatment effect. These questions include consideration given to current treatment provision during usual care, known treatment effectiveness, side effects of treatments, economic impact, and the setting in which the research is being undertaken.

Comparative effectiveness research compares two active forms of treatment or usual care in comparison with usual care with an additional intervention element. Comparative effectiveness research differs from study designs that have an inactive control, such as a ‘no-intervention’ or placebo group. In pharmaceutical research, trial designs in which placebo drugs are tested against the trial medication are often labeled ‘Phase III’ trials. Phase III trials aim to produce high-quality evidence of intervention efficacy and are important to identify potential side effects and benefits. Health outcome research with this study design involves the placebo being non-treatment or a ‘sham’ treatment option [ 1 ].

Traditionally, comparative effectiveness research is conducted following completion of a Phase III placebo control trial [ 2 – 4 ]. It is possible that comparative effectiveness research might not determine whether one treatment has clinical beneficence, because the comparator treatment might be harmful, irrelevant, or ineffective. This is unless the comparator treatment has already demonstrated superiority to a placebo [ 2 ]. Moreover, comparing an active treatment to an inactive control will be more likely to produce larger effect sizes than a comparison of two active treatments [ 5 ], requiring smaller sample sizes and lower costs to establish or refute the effectiveness of a treatment. Historically, then, treatments only become candidates for comparative effectiveness research to establish superiority, after a treatment has demonstrated efficacy against an inactive control.

Frequently, the provision of health interventions precedes development of the evidence base directly supporting their use [ 6 ]. Some service-provision contexts are highly regulated and high standards of evidence are required before an intervention can be provided (such as pharmacological interventions and device use). However, this is not universally the case for all services that may be provided in healthcare interventions. Despite this, there may be expectation from the individual patient and the public that individuals who present to a health service will receive some form of care deemed appropriate by treating clinicians, even in the absence of research-based evidence supporting this. This expectation may be amplified in publicly subsidized health services (as is largely the case in Canada, the UK, Australia, and many other developed nations) [ 7 – 9 ]. If a treatment is already widely employed by health professionals and is accepted by patients as a component of usual care, then it is important to consider the ethics and practicality of attempting a placebo or no-intervention control trial in this context. In this context, comparative effectiveness research could provide valuable insights to treatment effectiveness, disease pathophysiology, and economic efficiency in service delivery, with greater research feasibility than the traditional paradigm just described. Further, some authors have argued that studies with inactive control groups are used when comparative effectiveness research designs are more appropriate [ 10 ]. We propose and justify a framework for conducting research that argues for the broader use of comparative effectiveness research to achieve more feasible and translatable real-world clinical research.

This debate is important for the research community; particularly those engaged in the planning and execution of research in clinical practice settings, particularly in the provision of non-pharmacological, non-device type interventions. The ethical, preferential, and pragmatic implications from active versus inactive comparator selection in clinical trials not only influence the range of theoretical conclusions that could be drawn from a study, but also the lived experiences of patients and their treating clinical teams. The comparator selection will also have important implications for policy and practice when considering potential translation into clinical settings. It is these implications that affect the clinical researcher’s methodological design choice and justification.

The decision-making framework takes the form of a decision tree (Fig.  1 ) to determine when a comparative effectiveness study can be justified and is particularly relevant to the provision of services that do not have a tight regulatory framework governing when an intervention can be used as part of usual care. This framework is headed by Level 1 questions (demarcated by a question within an oval), which feed into decision nodes (demarcated by rectangles), which end in decision points (demarcated by diamonds). Each question is discussed with clinical examples to illustrate relevant points.

Comparative effectiveness research decision-making framework. Treatment A represents any treatment for a particular condition, which may or may not be a component of usual care to manage that condition. Treatment B is used to represent our treatment of interest. Where the response is unknown, the user should choose the NO response

Treatment A is any treatment for a particular condition that may or may not be a component of usual care to manage that condition. Treatment B is our treatment of interest. The framework results in three possible recommendations: that either (i) a study design comparing Treatment B with no active intervention could be used, or (ii) a study design comparing Treatment A, Treatment B and no active intervention should be used, or (iii) a comparative effectiveness study (Treatment A versus Treatment B) should be used.

Level 1 questions

Is the condition of interest being managed by any treatment as part of usual care either locally or internationally.

Researchers first need to identify what treatments are being offered as usual care to their target patient population to consider whether to perform a comparative effectiveness research (Treatment A versus B) or use a design comparing Treatment B with an inactive control. Usual care has been shown to vary across healthcare settings for many interventions [ 11 , 12 ]; thus, researchers should understand that usual care in their context might not be usual care universally. Consequently, researchers must consider what comprises usual care both in their local context and more broadly.

If there is no usual care treatment, then it is practical to undertake a design comparing Treatment B with no active treatment (Fig.  1 , Exit 1). If there is strong evidence of treatment effectiveness, safety, and cost-effectiveness of Treatment A that is not a component of usual care locally, this treatment should be considered for inclusion in the study. This situation can occur from delayed translation of research evidence into practice, with an estimated 17 years to implement only 14 % of research in evidence-based care [ 13 ]. In this circumstance, although it may be more feasible to use a Treatment B versus no active treatment design, the value of this research will be very limited, compared with comparative effectiveness research of Treatment A versus B. If the condition is currently being treated as part of usual care, then the researcher should consider the alternate Level 1 question for progression to Level 2.

As an example, prevention of falls is a safety priority within all healthcare sectors and most healthcare services have mitigation strategies in place. Evaluation of the effectiveness of different fall-prevention strategies within the hospital setting would most commonly require a comparative design [ 14 ]. A non-active treatment in this instance would mean withdrawal of a service that might be perceived as essential, a governmental health priority, and already integrated in the healthcare system.

Is there evidence of Treatment A’s effectiveness compared with no active intervention beyond usual care?

If there is evidence of Treatment A’s effectiveness compared with a placebo or no active treatment, then we progress to Question 3. If Treatment A has limited evidence, a comparative effectiveness research design of Treatment B versus no active treatment design can be considered. By comparing Treatment A with Treatment B, researchers would generate relevant research evidence for their local healthcare setting (is Treatment B superior to usual care or Treatment A?) and other healthcare settings that use Treatment A as their usual care. This design may be particularly useful when the local population is targeted and extrapolation of research findings is less relevant.

For example, the success of chronic disease management programs (Treatment A) run in different Aboriginal communities were highly influenced by unique characteristics and local cultures and traditions [ 15 ]. Therefore, taking Treatment A to an urban setting or non-indigenous setting with those unique characteristics will render Treatment A ineffectual. The use of Treatment A may also be particularly useful in circumstances where the condition of interest has an uncertain etiology and the competing treatments under consideration address different pathophysiological pathways. However, if Treatment A has limited use beyond the research location and there are no compelling reasons to extrapolate findings more broadly applicable, then Treatment B versus no active control design may be suitable.

The key points clinical researchers should consider are:

• The commonality of the treatment within usual care
• The success of established treatments in localized or unique population groups only
• Established effectiveness of treatments compared with placebo or no active treatment

Level 2 questions

Do the benefits of treatment a exceed the side effects when compared with no active intervention beyond usual care.

Where Treatment A is known to be effective, yet produces side effects, the severity, risk of occurrence, and duration of the side effects should be considered before it is used as a comparator for Treatment B. If the risk or potential severity of Treatment A is unacceptably high or is uncertain, and there are no other potential comparative treatments available, a study design comparing Treatment B with no active intervention should be used (Fig.  1 , Exit 2). Whether Treatment A remains a component of usual care should also be considered. If the side effects of Treatment A are considered acceptable, comparative effectiveness research may still be warranted.

The clinician researcher may also be challenged when the risk of the Treatment A and risk of Treatment B are unknown or when one is marginally more risky than the other [ 16 ]. Unknown risk comparison between the two treatments when using this framework should be considered as uncertain and the design of Treatment A versus Treatment B or Treatment B versus no intervention or a three-arm trial investigating Treatment A, B and no intervention is potentially justified (Fig.  1 , Exit 3).

A good example of risk comparison is the use of exercise programs. Walking has many health benefits, particularly for older adults, and has also demonstrated benefits in reducing falls [ 17 ]. Exercise programs inclusive of walking training have been shown to prevent falls but brisk walking programs for people at high risk of falls can increase the number of falls experienced [ 18 ]. The pragmatic approach of risk and design of comparative effectiveness research could better demonstrate the effect than a placebo (no active treatment) based trial.

• Risk of treatment side effects (including death) in the design
• Acceptable levels of risk are present for all treatments

Level 3 question

Does treatment a have a sufficient overall net benefit, when all costs and consequences or benefits are considered to deem it superior to a ‘no active intervention beyond usual care’ condition.

Simply being effective and free of unacceptable side effects is insufficient to warrant Treatment A being the standard for comparison. If the cost of providing Treatment A is so high that it renders its benefits insignificant compared with its costs, or Treatment A has been shown not to be cost-effective, or the cost-effectiveness is below acceptable thresholds, it is clear that Treatment A is not a realistic comparator. Some have advocated for a cost-effectiveness (cost-utility) threshold of $50,000 per quality-adjusted life year gained as being an appropriate threshold, though there is some disagreement about this and different societies might have different capacities to afford such a threshold [ 19 ]. Based on these considerations, one should further contemplate whether Treatment A should remain a component of usual care. If no other potential comparative treatments are available, a study design comparing Treatment B with no active intervention is recommended (Fig.  1 , Exit 4).

If Treatment A does have demonstrated efficacy, safety, and cost-effectiveness compared with no active treatment, it is unethical to pursue a study design comparing Treatment B with no active intervention, where patients providing consent are being asked to forego a safe and effective treatment that they otherwise would have received. This is an unethical approach and also unfeasible, as the recruitment rates could be very poor. However, Treatment A may be reasonable to include as a comparison if it is usually purchased by the potential participant and is made available through the trial.

The methodological design of a diabetic foot wound study illustrates the importance of health economics [ 20 ]. This study compared the outcomes of Treatment A (non-surgical sharps debridement) with Treatment B (low-frequency ultrasonic debridement). Empirical evidence supports the need for wound care and non-intervention would place the patient at risk of further wound deterioration, potentially resulting in loss of limb loss or death [ 21 ]. High consumable expenses and increased short-term time demands compared with low expense and longer term decreased time demands must also be considered. The value of information should also be considered, with the existing levels of evidence weighed up against the opportunity cost of using research funds for another purpose in the context of the probability that Treatment A is cost-effective [ 22 ].

• Economic evaluation and effect on treatment
• Understanding the health economics of treatment based on effectiveness will guide clinical practice
• Not all treatment costs are known but establishing these can guide evidence-based practice or research design

Level 4 question

Is the patient (potential participant) presenting to a health service or to a university- or research-administered clinic.

If Treatment A is not a component of usual care, one of three alternatives is being considered by the researcher: (i) conducting a comparative effectiveness study of Treatment B in addition to usual care versus usual care alone, (ii) introducing Treatment A to usual care for the purpose of the trial and then comparing it with Treatment B in addition to usual care, (iii) conducting a trial of Treatment B versus no active control. If the researcher is considering option (i), usual care should itself be considered to be Treatment A, and the researcher should return to Question 2 in our framework.

There is a recent focus on the importance of health research conducted by clinicians within health service settings as distinct from health research conducted by university-based academics within university settings [ 23 , 24 ]. People who present to health services expect to receive treatment for their complaint, unlike a person responding to a research trial advertisement, where it is clearly stated that participants might not receive active treatment. It is in these circumstances that option (ii) is most appropriate.

Using research designs (option iii) comparing Treatment B with no active control within a health service setting poses challenges to clinical staff caring for patients, as they need to consider the ethics of enrolling patients into a study who might not receive an active treatment (Fig.  1 , Exit 4). This is not to imply that the use of a non-active control is unethical. Where there is no evidence of effectiveness, this should be considered within the study design and in relation to the other framework questions about the risk and use of the treatment within usual care. Clinicians will need to establish the effectiveness, safety, and cost-effectiveness of the treatments and their impact on other health services, weighed against their concern for the patient’s well-being and the possibility that no treatment will be provided [ 25 ]. This is referred to as clinical equipoise.

Patients have a right to access publicly available health interventions, regardless of the presence of a trial. Comparing Treatment B with no active control is inappropriate, owing to usual care being withheld. However, if there is insufficient evidence that usual care is effective, or sufficient evidence that adverse events are likely, the treatment is prohibitive to implement within clinical practice, or the cost of the intervention is significant, a sham or placebo-based trial should be implemented.

Comparative effectiveness research evaluating different treatment options of heel pain within a community health service [ 26 ] highlighted the importance of the research setting. Children with heel pain who attended the health service for treatment were recruited for this study. Children and parents were asked on enrollment if they would participate if there were a potential assignment to a ‘no-intervention’ group. Of the 124 participants, only 7 % ( n  = 9) agreed that they would participate if placed into a group with no treatment [ 26 ].

• The research setting can impact the design of research
• Clinical equipoise challenges clinicians during recruitment into research in the healthcare setting
• Patients enter a healthcare service for treatment; entering a clinical trial is not the presentation motive

This framework describes and examines a decision structure for comparator selection in comparative effectiveness research based on current interventions, risk, and setting. While scientific rigor is critical, researchers in clinical contexts have additional considerations related to existing practice, patient safety, and outcomes. It is proposed that when trials are conducted in healthcare settings, a comparative effectiveness research design should be the preferred methodology to placebo-based trial design, provided that evidence for treatment options, risk, and setting have all been carefully considered.

Authors’ contributions

CMW and TPH drafted the framework and manuscript. All authors critically reviewed and revised the framework and manuscript and approved the final version of the manuscript.

Competing interests

The authors declare that they have no competing interests.

Contributor Information

Cylie M. Williams, Phone: +61 3 9784 8100, Email: ua.vog.civ.nchp@smailliweilyc .

Elizabeth H. Skinner, Email: [email protected] .

Alicia M. James, Email: ua.vog.civ.nchp@semajaicila .

Jill L. Cook, Email: [email protected] .

Steven M. McPhail, Email: [email protected] .

Terry P. Haines, Email: [email protected] .

Comparative Research: An Old Yet Unfamiliar Method

• Mashhad University of Medical Sciences

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Lorgel Maniaul
• Allysa Maninang
• Joseph Santos
• Samin Nobakht

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

Paper Information

• Previous Paper
• Paper Submission

Journal Information

• About This Journal
• Editorial Board
• Current Issue
• Author Guidelines

p-ISSN: 2162-9463    e-ISSN: 2162-8467

2014;  4(6): 148-155

doi:10.5923/j.edu.20140406.03

A Critical Review Study Conducted on Two Academic Articles Published in the Educational Field: From a Research Prospective

Ivan Hasan Murad

Department of English Language/University of Zakho-Kurdistan Region, Iraq

Copyright © 2014 Scientific & Academic Publishing. All Rights Reserved.

The primary focus of this study is to critically analyse two academic papers published in the educational field in terms of the validity and reliability of their methods of data collection and analysis, research design, and ethical implications. This is done in an attempt to demonstrate the valid procedure of conducting a research paper as a general aim for the current study. This is a desk research study conducted primarily for educational purposes. Data was collected from different resources found in the library of the University of Huddersfield in the United Kingdom. The analysis of the current research was conducted in the light of many educational resources specialized in research papers and publication. Results from the current study show that due to the lack of many standards, Brown's research is not reliable, valid and authentic, whereas Ornprapat and Saovapa's research is outstanding, valid, reliable, and authentic.

Keywords: Critical analysis, Validity, Reliability, Triangulation, Ethical consideration, Sampling

Cite this paper: Ivan Hasan Murad, A Critical Review Study Conducted on Two Academic Articles Published in the Educational Field: From a Research Prospective, Education , Vol. 4 No. 6, 2014, pp. 148-155. doi: 10.5923/j.edu.20140406.03.

Article Outline

1. introduction, 2. overview of the two articles, 2.1. article x, 2.2. article y, 3. literature review, 4. methodology, 4.1. analysis of the research design, 4.2. methods of data collection, 4.3. validity and reliability of the two articles, 4.4. ethical implications, 5. conclusions.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 05 September 2024

Comparative research on monitoring methods for nitrate nitrogen leaching in tea plantation soils

• Shenghong Zheng 1 , 2 ,
• Kang Ni 2 ,
• Hongling Chai 1 ,
• Qiuyan Ning 3 ,
• Chen Cheng 4 ,
• Huajing Kang 1 &
• Jianyun Ruan 2 , 5  

Scientific Reports volume  14 , Article number:  20747 ( 2024 ) Cite this article

Metrics details

• Plant ecology

Great concern has long been raised about nitrate leaching in cropland due to its possible environmental side effects in ground water contamination. Here we employed two common techniques to measure nitrate leaching in tea plantation soils in subtropical China. Using drainage lysimeter as a reference method, the adaptability of estimating drainage and nitrate leaching by combining the water balance equation with the suction cup technique was investigated. Results showed that the final cumulative leachate volume for the calculated and measured method was 721.43 mm and 729.92 mm respectively during the study period. However, nitrate concentration exerted great influence in the estimation of nitrate leaching from the suction cup-based method. The cumulative nitrate leaching loss from the lysimeter and suction cup-based method was 47.45 kg ha −1 and 43.58 kg ha −1 under lysimeter nitrate concentrations ranging from 7 mg L −1 to 13 mg L −1 , 156.28 kg ha −1 and 79.95 kg ha −1 under lysimeter nitrate concentrations exceeding 13 mg L −1 . Therefore, the suction cup-based method could be an alternative way of monitoring nitrate leaching loss within a range of 7–13 mg L −1 of nitrate concentrations in leachate. Besides, lower results occurred in suction cup samplers due to lack of representative samples which mainly leached via preferential flow when in strong leaching events. Thus, it is advisable to increase sampling frequency under such special conditions. The results of this experiment can serve as a reference and guidance for the application of ceramic cups in monitoring nitrogen and other nutrient-ion leaching in tea plantation soils.

Introduction

The intensive and extensive land-use activities associated with crops and animal production cause the most substantial anthropogenic source of nitrate, among which over-use of nitrogen fertilizer is one of the most contributing factors for nitrate pollution 1 , 2 . Compared with other crops, the tea plant (Camellia sinensis) requires an elevated nitrogen supply for the growth of tea shoots to enhance tea yield and quality 3 , 4 . The mean annual N application rate ranges from 281 to 745 kg ha −1 in the main tea production provinces in China. This means about 30% of the surveyed tea gardens applied excessive chemical fertilizers according to the current recommendation 5 . Meanwhile, higher N input levels increased concentrations of NO 3 − and NH 4 + in the 90–200 cm soil of the tea gardens, posing a high risk of N leaching loss in the tea gardens 6 . Thus, nitrate leaching from tea gardens should be of great concern for both scientists and producers.

In terms of monitoring methods for nitrate nitrogen leaching in agricultural soils, ceramic suction cup samplers and buried drainage lysimeters are the two most commonly employed techniques 7 . Ceramic suction cups are favored for their ease of installation and potential for repeated sampling at the same location 8 . They are deemed suitable for monitoring nitrate nitrogen leaching in non-structured soils 9 , 10 . However, ceramic suction cups are limited in their capacity to assess nitrate nitrogen concentrations only at specific soil depths and during particular sampling times 11 . This limitation makes it challenging to establish a comprehensive mass balance unless simultaneous quantification of soil water flux is undertaken. Additionally, characterized by low soil water retention and vulnerability to drought in coarse sandy soils, obtaining adequate sample volumes and capturing representative pore samples can be problematic 10 , 12 .

On the contrary, drainage lysimeters yield both the leachate volume and the nitrate nitrogen concentration, facilitating the calculation of nitrogen load passing below the defined soil layers. Other advantages embody larger sample volumes, enabling a representative sample of the soil pore network. Nonetheless, the installation and burial of drainage lysimeters traditionally introduce considerable soil disturbance, resulting in significant deviations from the original soil’s hydraulic properties and natural attributes, including the pathways for water and solute flow 13 . Strictly speaking, this approach constitutes a comprehensive method that integrates both temporal and spatial dimensions 14 . It thereby offers a more systematic and precise assessment of nitrogen leaching losses compared to other methodologies, which often capture relatively small-scale nitrogen leaching events and provide only a momentary glimpse into nitrogen leaching patterns 15 .

Due to the advantages and limitations inherent in both ceramic suction cup extraction and drainage lysimeter methodologies, these techniques are widely applied in empirical research. Several studies have also undertaken comparative analyses of their respective monitoring performance 10 , 12 , 13 . Nevertheless, extant research often relies on the ceramic suction cup approach to estimate nitrate nitrogen leaching quantities through multiplying the nitrate nitrogen concentration within the extracted solution by the measured volume obtained from the drainage lysimeter. This practice poses constraints on the application of the ceramic suction cup method, as the calculation of soil water flux becomes the key limiting factor when drainage lysimeter equipment is unavailable. Thus, it is imperative to explore alternative methods for calculating water flux and, on this basis, to conduct a comparative analysis of the two techniques. This approach is essential for promoting the practical utility and quantitative operability of the ceramic suction cup method.

Currently, there is limited research on localized nitrate nitrogen leaching in tea plantation soils, and a lack of comparative assessments of monitoring methods. In this study, we employed two methodologies, ceramic suction cup sampler and drainage lysimeters, to concurrently monitor nitrate nitrogen leaching in tea plantation soils. We put particular emphasis on the ceramic suction cup method, combined with a water balance equation, to evaluate the accuracy and efficacy of nitrate nitrogen leaching monitoring. Our objective is to provide insights and reference points for research efforts related to nitrogen leaching in tea plantation soils.

Materials and methods

Site description.

The field experiment was conducted at Tea Research Institute of Chinese Academy of Agricultural Sciences (TRI-CAAS) Experimental Station in Zhejiang province of China (29.74°N, 120.82°E). The experimental site has a typical subtropical monsoon climate, with 12.6 °C in mean annual temperature and 1200 mm yr −1 in annual total precipitation. Before the experiment, tea plants (clone variety Baiye1 and Longjing43, hereafter referred to BY1 and LJ43) were planted in rows (1.5 m between rows and 0.33 m between plants) at a density of approximately 6000 plants ha −1 and allowed to grow for 4 years in the research site. The soil at the site was acidic red soil, developed from granite parent material with a texture that is clay. Before the experiment, the surface (0–20 cm) soil properties were pH 4.47, SOC 5.71 g kg −1 , TN 0.47 g kg −1 , available potassium (AK) 20.42 g kg −1 , and low available phosphorus (AP) 1.48 g kg −1 .

Experimental design

The experiment included different nitrogen (N) treatment levels, ranging from 150 kg N ha −1 to 450 kg N ha -1 , with three replicates arranged in a randomized complete block design. Urea was used as the nitrogen fertilizer, and nitrogen fertilization was divided into spring (30% of the total), summer (20% of the total), and fall (50% of the total) applications. In addition to nitrogen, each plot received a one-time application of 90 kg ha −1 phosphorus (as P 2 O 5 ), 120 kg ha −1 potassium (as K 2 O), and 1200 kg ha −1 of organic fertilizer as a basal application. The phosphorus fertilizer used was calcium superphosphate (13% P 2 O 5 ), the potassium fertilizer was potassium sulfate (50% K 2 O), and the organic fertilizer was rapeseed cake (5% N). Fertilization was conducted during the fall season using manual trenching (10–15 cm depth). The required amount of fertilizer for each plot was evenly spread in the trench, followed by soil backfilling.

Sample collection method

Lysimeter installation and water sample collection.

Drainage lysimeters were installed in July 2015 in such a way that they were collected for a representative transect of the production bed. This involved digging pits with 1.5 m length × 1 m width × 1 m height in the middle of the tea plant rows. In case of the side-seepage of soil solution, each lysimeter pit was surrounded by a piece of plastic leather before soil backfilling. Each lysimeter was paired with two 1.5-m pipes among which one was for air passage and another was fitted with a 1.0-cm butyl rubber suction tube to allow extraction of the leachate collected at the bottom of the lysimeter by a vacuum pump. leachate was regularly removed bi-weekly by applying a partial vacuum (25–30 kpa) using a 10-L vacuum bottle placed in the vacuum line for each lysimeter. Leachate volume was determined gravimetrically and subsamples were collected from each bottle for drainage and nitrate analysis. Please refer to our previous study reported by Zheng et al. 16 for detailed information on the installation of lysimeters and the collection of water samples.

Soil solution extraction using ceramic suction cups

The soil solution extraction using the negative pressure ceramic suction cup method involved burying ceramic suction cups at a specific soil depth and connecting them to PVC pipes. Before sampling, a vacuum pump was used to create a vacuum inside the ceramic suction cup through the PVC pipe. This vacuum pressure allowed soil solution to be drawn into the ceramic suction cup, from which soil solution samples can then be extracted. In this experiment, ceramic suction cups were installed at a depth of 100 cm in the middle of tea rows. Four ceramic suction cups were placed horizontally at distances of − 0 cm, − 25 cm, − 50 cm, and − 75 cm from the tea tree roots. Before rainfall events, the ceramic suction cups were subjected to a vacuum pressure of approximately − 80 kPa to collect soil solution generated during rainfall. This sampling way was conducted simultaneously with the lysimeter method throughout the experiment.

Meteorological data were automatically collected by a weather station located about 100 m from the research site, and soil moisture was monitored using soil moisture sensors as described in our previous study reported by Zheng et al. 16 . The average temperature and rainfall during the experiment are shown in Fig.  1 . It can be observed from the figure that the total rainfall for March to December 2019 and January to June 2020 was 1374.60 mm and 1095 mm, respectively. The average daily temperature fluctuated within the range of 4.97 °C to 29.18 °C, with the highest daily average temperatures occurring in July and August and the lowest temperatures often emerging in December or January. Rainfall was most abundant from June to September, while November and December experienced lower levels of rainfall.

Total monthly precipitation and mean daily temperature by month from March 2019 to June 2020 at the research site.

Sample analysis and data processing

After filtering the collected soil solution and leachate samples, the nitrate nitrogen concentration, NO 3 − –N concentration, was determined using a UV dual-wavelength spectrophotometry method with wavelengths of 220 nm and 275 nm 17 , 18 .

For the calculation of nitrate nitrogen leaching amount (CL) from the leachate collector, it is calculated by multiplying the volume of the collected water sample by its nitrate nitrogen concentration, and the specific calculation method is as follows in Eq. ( 1 ).

where Ci is the measured NO 3 − –N concentration in the water sample, kg N L −1 , Vi is the volume of leachate collected per extraction. The numbers 1.5 and 1.0 represent the length and width of the lysimeter, m. 0.01 is the conversion factor.

For the ceramic cup method, we need to apply a water balance equation to calculate the water flux over a specific time period. After that, you can multiply it by the concentration of nitrate nitrogen in the extracting solution to obtain the nitrate nitrogen leaching amount. The specific calculation process is as follows in Eq. ( 2 ).

The cumulative nitrate nitrogen leaching amount (CLs) for the ceramic cup method can be calculated as follows:

where C ἰ and C ἰ+1 (kg N L −1 ) represent the average concentrations of nitrate nitrogen in the soil-extracting solution for two consecutive sampling times. n represents the total number of sampling events.

D represents the water flux over the time interval between the two sampling events, which can be calculated using the water balance equation as shown in Eq. ( 3 ).

where P is the precipitation (mm), I means the irrigation water quantity (mm), which is not relevant in this study and is not considered in the calculations. VR is the change in soil water storage (mm). D is the leachate flux (mm). ETc is the crop evapotranspiration (mm), calculated as ETc = kc* ET 0 , where ET 0 is the reference evapotranspiration for crops calculated from meteorological data according to FAO-56 Penman–Monteith equation 19 . The calculation of ET 0 can be simplified as follows in Eq. ( 4 ).

where ET 0 is the reference evapotranspiration (mm day −1 ), R n is the net radiation at the crop surface (MJ m −2  day −1 ), G is the soil heat flux density (MJ m −2  day −1 ), T is the mean daily air temperature at 2 m height (°C), u 2 is the wind speed at 2 m height (m s −1 ), e s is the saturation vapor pressure (kPa), e a is the actual vapor pressure (kPa), (es-ea) is the saturation vapor pressure deficit (kPa), ∆ is the slope vapor pressure curve (kPa °C −1 ), γ psychrometric constant (kPa °C −1 ), and 900 is the conversion factor.

Statistical data analysis was conducted using SPSS 22 software (SPSS Inc., New York, USA). One-way analysis of variance (ANOVA) was performed, followed by Duncan's post hoc test (p < 0.05 indicates significant differences, while p < 0.01 indicates highly significant differences). All graphs were generated using Sigmaplot 12.5 software (Systat Software Inc., Milpitas, USA).

Results and discussion

Comparison of drainage flux and leachate volume calculation.

During the experimental period from March 2019 to June 2020, 22 samples were taken both for BY1 and LJ43. The drainage flux for each sampling interval was calculated using the water balance equation. Based on the results from our previous study 16 , for BY1, Kc was set to 0.71 to calculate evapotranspiration. When the rainfall exceeded 78.02 mm, the drainage flux was fixed at the maximum value of 20.63 mm. For LJ43, Kc was set to 0.84 to calculate evapotranspiration, and when the rainfall reached or exceeded 90.98 mm, the drainage flux was fixed at the maximum value of 21.45 mm. For other rainfall levels, the drainage flux was calculated using the actual rainfall and the water balance equation. On this basis, the calculated drainage flux was compared and analyzed against the equivalent water depth calculated by converting the leachate volume extracted from the lysimeter (Lysimeter leachate). The equivalent water depth (mm) is calculated as the extracted water volume (L) divided by the lysimeter's area (1.5 m 2 in this study). The results are shown in Fig.  2 .

Correlation analysis of lysimeter leachate and calculated drainage ( a ) and comparison of cumulative leachate and cumulative calculated drainage ( b ) for the BY1 and LJ43 during the study period.

From Fig.  2 a, it can be observed that the volume data points for both methods are distributed close to the 1:1 line, indicating that the calculated drainage flux and the lysimeter leachate volume measurements are generally in good agreement. Furthermore, the total volume sums for both methods were calculated separately (Fig.  2 b). The results indicate that the cumulative calculated drainage flux for BY1 during the experimental period was 389.21 mm, slightly higher than the total lysimeter leachate volume measured at 367.77 mm. For LJ43, the total calculated drainage flux was 332.22 mm, slightly lower than the total lysimeter leachate volume of 362.15 mm. Finally, when combining all results for BY1 and LJ43, the total calculated drainage flux and the total lysimeter leachate volume were 721.43 mm and 729.92 mm, respectively, with the former only 1.16% lower than the latter. Therefore, the application of the water balance equation for soil drainage flux calculation demonstrated high accuracy and feasibility.

Comparison of soil solution and leachate nitrate nitrogen concentrations

A relationship was created with the nitrate nitrogen concentration of the lysimeter leachate during the experimental period as the x-axis and the nitrate nitrogen concentration of the soil solution extracted using the ceramic cup method as the y-axis. Additionally, a logarithmic transformation was applied to further analyze the impact of the two extraction methods on nitrate nitrogen concentration. The results are shown in Fig.  3 . It can be observed in Fig.  3 a that when the nitrate nitrogen concentration in the lysimeter leachate is less than 7 mg L −1 , all nitrate nitrogen concentrations in the soil solution extracted from the ceramic cup method are higher than those in the lysimeter leachate. Subsequently, as the nitrate nitrogen concentration in the lysimeter leachate increases from 7 mg L −1 to 13 mg L −1 , approximately half of the soil solution extracted from the ceramic cup method has a higher nitrate nitrogen concentration than the lysimeter leachate, while the other half has a lower nitrate nitrogen concentration. Then, when the nitrate nitrogen concentration in the lysimeter leachate exceeds 13 mg L −1 , all soil solution extracted using the ceramic cup method has a lower nitrate nitrogen concentration than the lysimeter leachate.

Correlation between ( a ) nitrate concentration from lysimeter and suction cup and ( b ) nitrate concentration from lysimeter and logarithmic conversion value of the ratio of nitrate concentration from lysimeter to suction cup nitrate concentration.

Further analysis was conducted by taking the ratio of the nitrate nitrogen concentrations in the lysimeter leachate and the soil solution extracted using the ceramic cup method as a real number, with a base of 2 for logarithmic transformation. The trend of this transformed value with respect to the nitrate nitrogen concentration in the lysimeter leachate is shown in Fig.  3 b. It is evident that as the nitrate nitrogen concentration in the lysimeter leachate increases, the logarithmic transformation value increases from its minimum value of − 3.51 to 1.93. The transformation value exhibits distinct trends and characteristics based on the grouping of nitrate nitrogen concentrations in the lysimeter leachate. When the lysimeter leachate concentration is less than 7 mg L −1 , the transformation value is consistently less than 0. When the lysimeter leachate concentration exceeds 13 mg L −1 , the transformation value is consistently greater than 0. However, when the lysimeter leachate concentration falls between 7 mg L −1 and 13 mg L −1 , both positive and negative transformation values coexist.

Comparison of nitrate nitrogen leaching between two methods

Similarly, a relationship was created with the nitrate nitrogen concentration of the lysimeter leachate (Lysimeter method) as the x-axis and the nitrate nitrogen concentration obtained using the ceramic cup method combined with the water balance equation (Ceramic cup method) as the y-axis. Additionally, a logarithmic transformation was applied to further analyze the impact of the two methods on nitrate nitrogen leaching. The results are shown in Fig.  4 . From Fig.  4 a, it can be observed that when the nitrate nitrogen concentration in the lysimeter leachate is less than 7 mg L −1 , almost all nitrate nitrogen leaching calculated using the ceramic cup method is higher than the nitrate nitrogen concentration in the lysimeter leachate. When the lysimeter leachate concentration falls between 7 mg L −1 and 13 mg L −1 , more than half of the nitrate nitrogen leaching calculated using the ceramic cup method is lower than the lysimeter method, while the other half is higher. Then, when the lysimeter leachate concentration exceeds 13 mg L −1 , all nitrate nitrogen concentrations calculated using the ceramic cup method are lower than the lysimeter leachate.

Correlation between ( a ) nitrate leaching from lysimeter and suction cup and ( b ) nitrate leaching from lysimeter and logarithmic conversion value of the ratio of nitrate leaching from lysimeter to suction cup nitrate leaching.

Further analysis was conducted by taking the ratio of the nitrate nitrogen concentrations in the lysimeter leachate and those calculated using the ceramic cup method as a real number, with a base of 2 for logarithmic transformation. The trend of this transformed value with respect to the nitrate nitrogen concentration in the lysimeter leachate is shown in Fig.  4 b. It is evident that the transformation value follows a trend highly similar to the concentration transformation trend mentioned above. As the nitrate nitrogen concentration in the lysimeter leachate increases, the logarithmic transformation value increases from its minimum value of − 3.51 to 1. This transformation value exhibits distinct trends and characteristics based on the grouping of nitrate nitrogen concentrations in the lysimeter leachate. When the lysimeter leachate concentration is less than 7 mg L −1 , the transformation value is consistently less than 0. When the lysimeter leachate concentration exceeds 13 mg L −1 , the transformation value is consistently greater than 0. However, when the lysimeter leachate concentration falls between 7 mg L −1 and 13 mg L −1 , both positive and negative transformation values coexist.

In addition, statistical analysis was performed on the total nitrate nitrogen leaching for each concentration group. The results indicate that when the lysimeter leachate concentration was less than 7 mg L −1 , the total nitrate nitrogen leaching obtained by the lysimeter method and the ceramic cup method is 22.24 kg ha −1 and 44.05 kg ha −1 , respectively. When the lysimeter leachate concentration fell between 7 mg L −1 and 13 mg L −1 , the total nitrate nitrogen leaching calculated by the lysimeter method and the ceramic cup method was 47.45 kg ha −1 and 43.58 kg ha −1 , respectively. When the lysimeter leachate concentration exceeded 13 mg L −1 , the total nitrate nitrogen leaching obtained by the lysimeter method and the ceramic cup method was 156.28 kg ha −1 and 79.95 kg ha −1 , respectively. In summary, there were differences in quantified nitrate nitrogen leaching losses between the two methods. If the lysimeter method was used as the standard, the ceramic cup method exhibited higher monitoring accuracy when the nitrate nitrogen concentration in the lysimeter leachate fell within the range of 7–13 mg L −1 .

Effect of rainfall on the application of the water balance model

The use of ceramic cup methods to monitor nitrate nitrogen leaching in farmland requires estimation of soil water flux through modeling. This inevitably introduces uncertainties in accurately quantifying nitrate nitrogen 20 . In this study, the application of a water balance model for quantitatively calculating soil drainage volume seemed to yield slightly lower water flux results compared to the corresponding measurements obtained through the lysimeter method, especially when rainfall was low (Fig.  2 a). One possible reason for this discrepancy could be that the water balance equation typically accounts for only the saturated flow above field capacity, neglecting unsaturated flow. However, it is reported that unsaturated flow, which occurs at lower soil moisture levels, is more common in practice, especially when rainfall is low and soil moisture levels remain relatively low 21 . Therefore, it is speculated that unsaturated flow is the primary reason for the water balance model calculating lower water flux than the lysimeter measurements under these conditions.

On the other hand, for conditions with higher rainfall intensity, when applying the water balance equation to estimate water flux, it should strictly include runoff as part of the water output, with the most accurate method being the construction of runoff tanks for precise measurement. However, this study lacked the necessary means to estimate runoff, which likely led to significant deviations in the final water flux calculations. Nevertheless, previous study reported that runoff typically occurs during heavy rainfall events and increases with higher rainfall amounts 22 , 23 , 24 and when a certain critical rainfall intensity is reached, water will be lost as runoff because the soil cannot absorb and retain it, and an eventual maximum leachate flux will occur 25 . Based on our previous study, critical rainfall amounts and maximum water leachate fluxes were determined for the tea varieties of Longjing 43 and BaiYe 1, thus mitigating the significant calculation bias arising from the absence of runoff monitoring.

Effect of soil texture on the accuracy of the suction cup-based method

The lysimeter method, being considered a relatively accurate technique for monitoring and quantifying soil nitrate nitrogen leaching, is often regarded as a true reflection of nitrate nitrogen leaching in soil 26 . This study indicated that when the nitrate nitrogen concentration in lysimeter leachate fell below 13 mg L -1 (especially within the range of 7–13 mg L −1 ), the ceramic cup method demonstrated relatively accurate monitoring results. However, when the leachate nitrate concentration exceeded 13 mg L −1 , A much lower result was obtained from the ceramic cup method compared to the lysimeter method. The reason for this may rely on the soil structure. From the perspective of soil texture, this experiment was conducted in a relatively heavy clay tea plantation, where the clay content within the top meter of soil ranged from 62.53 to 69.99% 16 . Under such soil conditions, nitrate nitrogen is likely to be transported downward through preferential flow. Preferential flow is characterized by the rapid movement of most soil water and solutes through the large and intermediate pores of the soil, bypassing the surface soil and moving downward 27 . Previous studies have found that the occurrence of preferential flow was much higher in clay soils than in sandy or loamy soils 28 , 29 , 30 . This often resulted in higher concentrations of nitrate nitrogen in leachate water 31 .

Ceramic cups, on the other side, have been reported to be unsuitable for use in clayey soils because the presence of preferential flow makes it difficult for ceramic cups to effectively collect water flowing through large pores, especially during heavy rainfall events 32 . Additionally, Barbee and Brown (1986) compared the performance of ceramic cups and lysimeters in monitoring chloride ions in soils with three different textures. The results showed that lysimeters generally provided higher and more stable monitoring results in loam and sandy loam soils, while ceramic cups were almost ineffective in clayey soils due to the rapid leaching and movement of water through large pores. Therefore, to some extent, ceramic cups were considered to be a flawed soil solution extraction technique for clayey soils. These factors need to be considered in soil nitrate nitrogen leaching studies, especially in soil types like clay, where choosing an appropriate solution extraction method is crucial for obtaining accurate data.

Conclusions

In comparison to direct measurements using lysimeters as a reference, the feasibility of the ceramic cup's negative pressure extraction estimation method was analyzed. The results demonstrated that the total calculated drainage flux and the total measured volume for lysimeter leachate were 721.43 mm and 729.92 mm, respectively, indicating that the application of the water balance equation for estimating soil drainage flux is accurate and feasible. Furthermore, through a comparative analysis of nitrate nitrogen concentrations in water samples collected by lysimeters and ceramic cups, it was observed that the ceramic cup method exhibited a certain accuracy in estimating nitrogen leaching, especially when the nitrate nitrogen concentration in lysimeter leachate fell within the range of 7–13 mg L −1 . However, under conditions of intense leaching (nitrate nitrogen concentration in lysimeter leachate exceeding 13 mg L −1 ), there was a risk of underestimation due to the potential lack of representative samples. Therefore, it is advisable to increase sampling frequency under such special circumstances.

Data availability

The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.

Abbreviations

Variety Baiye1

Variety Longjing43

Hallberg, G. R. Nitrate in groundwater in the United States. In Nitrogen Management and Groundwater Protection (ed. Follett, R. F.) 35–74 (Elsevier, 1989).

Chapter   Google Scholar  

Keeney, D. R. Sources of nitrate to ground water. CRC Crit. Rev. Environ. Contr. 16 , 257–304 (1986).

Article   CAS   Google Scholar  

Han, W. Y., Ma, L. F., Shi, Y. Z., Ruan, J. Y. & Kemmitt, S. J. Nitrogen release dynamics and transformation of slow release fertiliser products and their effects on tea yield and quality. J. Sci. Food Agric. 88 , 839–846 (2008).

Han, W., Xu, J., Wei, K., Shi, Y. & Ma, L. Estimation of N 2 O emission from tea garden soils, their adjacent vegetable garden and forest soils in eastern China. Environ. Earth Sci. 70 , 2495–2500 (2013).

Article   ADS   CAS   Google Scholar  

Ni, K. et al. Fertilization status and reduction potential in tea gardens of China. J. Plant Nutr. Fertil. 25 , 421–432 (2019).

Google Scholar  

Yan, P. et al. Tea planting affects soil acidification and nitrogen and phosphorus distribution in soil. Agric. Ecosyst. Environ. 254 , 20–25 (2018).

Wey, H., Hunkeler, D., Bischoff, W. A. & Bünemann, E. K. Field-scale monitoring of nitrate leaching in agriculture: Assessment of three methods. Environ. Monit. Assess. 194 , 1–20 (2022).

Article   Google Scholar  

Creasey, C. L. & Dreiss, S. J. Porous cup samplers: Cleaning procedures and potential sample bias from trace element contamination. Soil Sci. 145 , 93–101 (1988).

Webster, C. P., Shepherd, M. A., Goulding, K. W. T. & Lord, E. Comparisons of methods for measuring the leaching of mineral nitrogen from arable land. J. Soil Sci. 44 , 49–62 (1993).

Barbee, G. C. & Brown, K. W. Comparison between suction and free-drainage soil solution samplers. Soil Sci. 141 , 149–154 (1986).

Wolf, K. A., Pullens, J. W. & Børgesen, C. D. Optimized number of suction cups required to predict annual nitrate leaching under varying conditions in Denmark. J. Environ. Manag. 328 , 116964 (2023).

Lord, E. I. & Shepherd, M. A. Developments in the use of porous ceramic cups for measuring nitrate leaching. J. Soil Sci. 44 , 435–449 (1993).

Wang, Q. et al. Comparison of lysimeters and porous ceramic cups for measuring nitrate leaching in different soil types. New Zealand J. Agric. Res. 55 , 333–345 (2012).

Brown, S. et al. Assessing variability of soil water balance components measured at a new lysimeter facility dedicated to the study of soil ecosystem services. J. Hydrol. 603 , 127037 (2021).

Zotarelli, L., Scholberg, J. M., Dukes, M. D. & Muñoz-Carpena, R. Monitoring of nitrate leaching in sandy soils: Comparison of three methods. J. Environ. Qual. 36 , 953–962 (2007).

Article   CAS   PubMed   Google Scholar  

Zheng, S. et al. Estimation of evapotranspiration and crop coefficient of rain-fed tea plants under a subtropical climate. Agronomy 11 , 2332 (2021).

Norman, R. J., Edberg, J. C. & Stucki, J. W. Determination of nitrate in soil extracts by dual-wavelength ultraviolet spectrophotometry. Soil Sci. Soc. Am. J. 49 , 1182–1185 (1985).

Goldman, E. & Jacobs, R. Determination of Nitrates by Ultraviolet Absorption. Am. Water Works Assoc. 53 , 187–191 (1961).

Allen, R.G., Pereira, L.S., Raes, D., Smith, M. Crop Evapotranspiration-Guidelines for Computing Crop Water Requirements. FAO Irrigation and Drainage Paper 56, FAO: Rome, Italy, 1998; pp. 2–15 (1998).

Weihermüller, L. et al. In situ soil water extraction: A review. J. Environ. Qual. 36 , 1735–1748 (2007).

Article   PubMed   Google Scholar  

Hu, K., Li, B., Chen, D. & White, R. E. Estimation of water percolation and nitrogen leaching in farmland: Comparison of two models. Adv. Water Sci. 15 , 87–93 (2004).

CAS   Google Scholar  

Alizadehtazi, B., Gurian, P. L. & Montalto, F. A. Impact of successive rainfall events on the dynamic relationship between vegetation canopies, infiltration, and recharge in engineered urban green infrastructure systems. Ecohydrology 13 , e2185 (2020).

Liu, G. et al. Interactive effects of raindrop impact and groundwater seepage on soil erosion. J. Hydrol. 578 , 124066 (2019).

Wang, H. et al. Effects of rainfall intensity on groundwater recharge based on simulated rainfall experiments and a groundwater flow model. CATENA 127 , 80–91 (2015).

Wang, J., Chen, L. & Yu, Z. Modeling rainfall infiltration on hillslopes using Flux-concentration relation and time compression approximation. J. Hydrol. 557 , 243–253 (2018).

Article   ADS   Google Scholar  

Sołtysiak, M. & Rakoczy, M. An overview of the experimental research use of lysimeters. Environ. Socio-Econ. Stud. 7 , 49–56 (2019).

Beven, K. & Germann, P. Macropores and water flow in soils. Water Resour. Res. 18 , 1311–1325 (1982).

Butters, G. L., Jury, W. A. & Ernst, F. F. Field scale transport of bromide in an unsaturated soil: 1. Experimental methodology and results. Water Resour. Res. 25 , 1575–1581 (1989).

Roth, K., Jury, W. A., Flühler, H. & Attinger, W. Transport of chloride through an unsaturated field soil. Water Resour. Res. 27 , 2533–2541 (1991).

Ellsworth, T. R., Jury, W. A., Ernst, F. F. & Shouse, P. J. A three-dimensional field study of solute transport through unsaturated, layered, porous media: 1. Methodology, mass recovery, and mean transport. Water Resour. Res. 27 , 951–965 (1991).

Bronswijk, J. J. B., Hamminga, W. & Oostindie, K. Rapid nutrient leaching to groundwater and surface water in clay soil areas. Eur. J. Agron. 4 , 431–439 (1995).

Grossmann, J., Bredemeier, M. & Udluft, P. Sorption of trace elements by suction cups of aluminum-oxide, ceramic, and plastics. Z. Pflanzenernähr. Bodenkd. 153 , 359–364 (1990).

Download references

Acknowledgements

This work was financially supported by the National Key Research and Development Program of China (2022YFF0606802) and the Earmarked Fund for China Agriculture Research System (CARS-19).

Author information

Authors and affiliations.

Key Laboratory of Crop Breeding in South Zhejiang, Wenzhou Academy of Agricultural Sciences, Wenzhou, 325006, China

Shenghong Zheng, Hongling Chai & Huajing Kang

Key Laboratory of Tea Biology and Resource Utilization of Tea (Ministry of Agriculture), Tea Research Institute, Chinese Academy of Agriculture Sciences, Hangzhou, 310008, China

Shenghong Zheng, Kang Ni & Jianyun Ruan

Lishui Academy of Agricultural and Forestry Sciences, Lishui, 323000, China

Qiuyan Ning

College of Ecology, Lishui University, Lishui, 323000, China

Xihu National Agricultural Experimental Station for Soil Quality, Hangzhou, 310008, China

Jianyun Ruan

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: SZ and JR; writing-original draft preparation: SZ;Writing-review and editing: KN, HC and JR; formal analysis: QN and CC; resources: HK; funding acquisition: JR. All authors have read and agreed to the published version of the manuscript.

Corresponding authors

Correspondence to Huajing Kang or Jianyun Ruan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zheng, S., Ni, K., Chai, H. et al. Comparative research on monitoring methods for nitrate nitrogen leaching in tea plantation soils. Sci Rep 14 , 20747 (2024). https://doi.org/10.1038/s41598-024-71081-3

Download citation

Received : 10 July 2024

Accepted : 23 August 2024

Published : 05 September 2024

DOI : https://doi.org/10.1038/s41598-024-71081-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Drainage flux
• Nitrate leaching
• Water balance model
• Suction cup sampler
• Buried lysimeter

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Comparative Analysis

• First Online: 02 January 2023

Cite this chapter

• Kenisha Blair-Walcott 4  

Part of the book series: Springer Texts in Education ((SPTE))

5912 Accesses

Comparative analysis is a multidisciplinary method, which spans a wide cross-section of disciplines (Azarian, International Journal of Humanities and Social Science, 1(4), 113–125 (2014)). It is the process of comparing multiple units of study for the purpose of scientific discovery and for informing policy decisions (Rogers, Comparative effectiveness research, 2014). Even though there has been a renewed interest in comparative analysis as a research method over the last decade in fields such as education, it has been used in studies for decades.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Cross-National Comparative Research—Analytical Strategies, Results, and Explanations

Multilevel Structural Equation Modeling for Cross-National Comparative Research

Comparative Case Studies: Methodological Discussion

Adiyia, M., & Ashton, W. (2017). Comparative research. Brandon University.

Google Scholar  

Azarian, R. (2011). Potentials and limitations of comparative method in social science. International Journal of Humanities and Social Science, 1 (4), 113–125. http://www.ijhssnet.com/journals/Vol._1_No._4%3b_April_2011/15.pdf

Bray, M., Adamson, B., & Mason, M. (2014). Comparative education research: Approaches and methods . Springer.

Crossley, M. (2002). Comparative and international education: Contemporary challenges, reconceptualization and new directions for the field. Current Issues in Comparative Education, 4 (2), 81–86. https://www.tc.columbia.edu/cice/pdf/25691_4_2_Crossley.pdf

Esser, F., & Vliegenthart, R. (2017). Comparative research methods. The International Encyclopedia of Communication Research Methods, 1 , 1–22. https://doi.org/10.1002/9781118901731.iecrm0035

Article   Google Scholar  

Henry, I. (Ed.). (2007). Transnational and comparative research in sport globalisation, governance and sport policy. Routledge. https://doi-org.cyber.usask.ca/ https://doi.org/10.4324/9780203944738

Mills, M., Bunt, G., & Bruijn, J. (2006). Comparative research: Persistent problems and promising solutions. International Sociology, 21 (5), 619–631. https://doi.org/10.1177/0268580906067833

Nóvoa, A., & Yariv-Mashal, T. (2003). Comparative research in education: A mode of governance or a historical journey? Comparative Education, 39 (4), 423–438. https://repositorio.ul.pt/bitstream/10451/680/1/21185_0305-0068_423-438.pdf

Peters, G. (2013). Strategies for comparative research in political science . Macmillan.

Book   Google Scholar  

Pickvance, C. (2005). The four varieties of comparative analysis: The case of environmental regulation. Journal of Housing and the Built Environment, 16 , 7–28.

Rogers, M. (2014). Comparative effectiveness research .

Rokkan, S. (1968). The structuring of mass politics in the smaller European democracies: A developmental typology. Comparative Studies in Society and History, 10 (2), 173–210. https://www.jstor.org/stable/177728

Tilly, C. (1984). Big structures, large processes, huge comparisons . SAGE.

Wang, G., & Huang, Y. (2016). Contextuality, commensurability, and comparability in comparative research: Learning from Chinese relationship research. Cross-Cultural Research, 50 (2), 154–177. https://doi.org/10.1177/1069397116630241

Download references

Author information

Authors and affiliations.

University of Saskatchewan, Saskatoon, Canada

Kenisha Blair-Walcott

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kenisha Blair-Walcott .

Editor information

Editors and affiliations.

Department of Educational Administration, College of Education, University of Saskatchewan, Saskatoon, SK, Canada

Janet Mola Okoko

Scott Tunison

Department of Educational Administration, University of Saskatchewan, Saskatoon, SK, Canada

Keith D. Walker

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Blair-Walcott, K. (2023). Comparative Analysis. In: Okoko, J.M., Tunison, S., Walker, K.D. (eds) Varieties of Qualitative Research Methods. Springer Texts in Education. Springer, Cham. https://doi.org/10.1007/978-3-031-04394-9_13

Download citation

DOI : https://doi.org/10.1007/978-3-031-04394-9_13

Published : 02 January 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-04396-3

Online ISBN : 978-3-031-04394-9

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research