mental health in research methodology

• Systematic review
• Open access
• Published: 10 October 2019

An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance

• Eric Badu   ORCID: orcid.org/0000-0002-0593-3550 1 ,
• Anthony Paul O’Brien 2 &
• Rebecca Mitchell 3  

Archives of Public Health volume  77 , Article number:  37 ( 2019 ) Cite this article

21k Accesses

19 Citations

1 Altmetric

Metrics details

Several typologies and guidelines are available to address the methodological and practical considerations required in mental health research. However, few studies have actually attempted to systematically identify and synthesise these considerations. This paper provides an integrative review that identifies and synthesises the available research evidence on mental health research methodological considerations.

A search of the published literature was conducted using EMBASE, Medline, PsycINFO, CINAHL, Web of Science, and Scopus. The search was limited to papers published in English for the timeframe 2000–2018. Using pre-defined inclusion and exclusion criteria, three reviewers independently screened the retrieved papers. A data extraction form was used to extract data from the included papers.

Of 27 papers meeting the inclusion criteria, 13 focused on qualitative research, 8 mixed methods and 6 papers focused on quantitative methodology. A total of 14 papers targeted global mental health research, with 2 papers each describing studies in Germany, Sweden and China. The review identified several methodological considerations relating to study design, methods, data collection, and quality assurance. Methodological issues regarding the study design included assembling team members, familiarisation and sharing information on the topic, and seeking the contribution of team members. Methodological considerations to facilitate data collection involved adequate preparation prior to fieldwork, appropriateness and adequacy of the sampling and data collection approach, selection of consumers, the social or cultural context, practical and organisational skills; and ethical and sensitivity issues.

The evidence confirms that studies on methodological considerations in conducting mental health research largely focus on qualitative studies in a transcultural setting, as well as recommendations derived from multi-site surveys. Mental health research should adequately consider the methodological issues around study design, sampling, data collection procedures and quality assurance in order to maintain the quality of data collection.

Peer Review reports

In the past decades there has been considerable attention on research methods to facilitate studies in various academic fields, such as public health, education, humanities, behavioural and social sciences [ 1 , 2 , 3 , 4 ]. These research methodologies have generally focused on the two major research pillars known as quantitative or qualitative research. In recent years, researchers conducting mental health research appear to be either employing both qualitative and quantitative research methods separately, or mixed methods approaches to triangulate and validate findings [ 5 , 6 ].

A combination of study designs has been utilised to answer research questions associated with mental health services and consumer outcomes [ 7 , 8 ]. Study designs in the public health and clinical domains, for example, have largely focused on observational studies (non-interventional) and experimental research (interventional) [ 1 , 3 , 9 ]. Observational design in non-interventional research requires the investigator to simply observe, record, classify, count and analyse the data [ 1 , 2 , 10 ]. This design is different from the observational approaches used in social science research, which may involve observing (participant and non- participant) phenomena in the fieldwork [ 1 ]. Furthermore, the observational study has been categorized into five types, namely cross-sectional design, case-control studies, cohort studies, case report and case series studies [ 1 , 2 , 3 , 9 , 10 , 11 ]. The cross-sectional design is used to measure the occurrence of a condition at a one-time point, sometimes referred to as a prevalence study. This approach of conducting research is relatively quick and easy but does not permit a distinction between cause and effect [ 1 ]. Conversely, the case-control is a design that examines the relationship between an attribute and a disease by comparing those with and without the disease [ 1 , 2 , 12 ]. In addition, the case-control design is usually retrospective and aims to identify predictors of a particular outcome. This type of design is relevant when investigating rare or chronic diseases which may result from long-term exposure to particular risk factors [ 10 ]. Cohort studies measure the relationship between exposure to a factor and the probability of the occurrence of a disease [ 1 , 10 ]. In a case series design, medical records are reviewed for exposure to determinants of disease and outcomes. More importantly, case series and case reports are often used as preliminary research to provide information on key clinical issues [ 12 ].

The interventional study design describes a research approach that applies clinical care to evaluate treatment effects on outcomes [ 13 ]. Several previous studies have explained the various forms of experimental study design used in public health and clinical research [ 14 , 15 ]. In particular, experimental studies have been categorized into randomized controlled trials (RCTs), non-randomized controlled trials, and quasi-experimental designs [ 14 ]. The randomized trial is a comparative study where participants are randomly assigned to one of two groups. This research examines a comparison between a group receiving treatment and a control group receiving treatment as usual or receiving a placebo. Herein, the exposure to the intervention is determined by random allocation [ 16 , 17 ].

Recently, research methodologists have given considerable attention to the development of methodologies to conduct research in vulnerable populations. Vulnerable population research, such as with mental health consumers often involves considering the challenges associated with sampling (selecting marginalized participants), collecting data and analysing it, as well as research engagement. Consequently, several empirical studies have been undertaken to document the methodological issues and challenges in research involving marginalized populations. In particular, these studies largely addresses the typologies and practical guidelines for conducting empirical studies in mental health. Despite the increasing evidence, however, only a few studies have yet attempted to systematically identify and synthesise the methodological considerations in conducting mental health research from the perspective of consumers.

A preliminary search using the search engines Medline, Web of Science, Google Scholar, and Scopus Index and EMBASE identified only two reviews of mental health based research. Among these two papers, one focused on the various types of mixed methods used in mental health research [ 18 ], whilst the other paper, focused on the role of qualitative studies in mental health research involving mixed methods [ 19 ]. Even though the latter two studies attempted to systematically review mixed methods mental health research, this integrative review is unique, as it collectively synthesises the design, data collection, sampling, and quality assurance issues together, which has not been previously attempted.

This paper provides an integrative review addressing the available evidence on mental health research methodological considerations. The paper also synthesises evidence on the methods, study designs, data collection procedures, analyses and quality assurance measures. Identifying and synthesising evidence on the conduct of mental health research has relevance to clinicians and academic researchers where the evidence provides a guide regarding the methodological issues involved when conducting research in the mental health domain. Additionally, the synthesis can inform clinicians and academia about the gaps in the literature related to methodological considerations.

Methodology

An integrative review was conducted to synthesise the available evidence on mental health research methodological considerations. To guide the review, the World Health Organization (WHO) definition of mental health has been utilised. The WHO defines mental health as: “a state of well-being, in which the individual realises his or her own potentials, ability to cope with the normal stresses of life, functionality and work productivity, as well as the ability to contribute effectively in community life” [ 20 ]. The integrative review enabled the simultaneous inclusion of diverse methodologies (i.e., experimental and non-experimental research) and varied perspectives to fully understand a phenomenon of concern [ 21 , 22 ]. The review also uses diverse data sources to develop a holistic understanding of methodological considerations in mental health research. The methodology employed involves five stages: 1) problem identification (ensuring that the research question and purpose are clearly defined); 2) literature search (incorporating a comprehensive search strategy); 3) data evaluation; 4) data analysis (data reduction, display, comparison and conclusions) and; 5) presentation (synthesising findings in a model or theory and describing the implications for practice, policy and further research) [ 21 ].

Inclusion criteria

The integrative review focused on methodological issues in mental health research. This included core areas such as study design and methods, particularly qualitative, quantitative or both. The review targeted papers that addressed study design, sampling, data collection procedures, quality assurance and the data analysis process. More specifically, the included papers addressed methodological issues on empirical studies in mental health research. The methodological issues in this context are not limited to a particular mental illness. Studies that met the inclusion criteria were peer-reviewed articles published in the English Language, from January 2000 to July 2018.

Exclusion criteria

Articles that were excluded were based purely on general health services or clinical effectiveness of a particular intervention with no connection to mental health research. Articles were also excluded when it addresses non-methodological issues. Other general exclusion criteria were book chapters, conference abstracts, papers that present opinion, editorials, commentaries and clinical case reviews.

Search strategy and selection procedure

The search of published articles was conducted from six electronic databases, namely EMBASE, CINAHL (EBSCO), Web of Science, Scopus, PsycINFO and Medline. We developed a search strategy based on the recommended guidelines by the Joanna Briggs Institute (JBI) [ 23 ]. Specifically, a three-step search strategy was utilised to conduct the search for information (see Table  1 ). An initial limited search was conducted in Medline and Embase (see Table 1 ). We analysed the text words contained in the title and abstract and of the index terms from the initial search results [ 23 ]. A second search using all identified keywords and index terms was then repeated across all remaining five databases (see Table 1 ). Finally, the reference lists of all eligible studies were manually hand searched [ 23 ].

The selection of eligible articles adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 24 ] (see Fig.  1 ). Firstly, three authors independently screened the titles of articles that were retrieved and then approved those meeting the selection criteria. The authors reviewed all the titles and abstracts and agreed on those needing full-text screening. E.B (Eric Badu) conducted the initial screening of titles and abstracts. A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell) conducted the second screening of titles and abstracts of all the identified papers. The authors (E.B, A.P.O’B and R.M) conducted full-text screening according to the inclusion and exclusion criteria.

Flow Chart of studies included in the review

Data management and extraction

The integrative review used Endnote ×8 to screen and handle duplicate references. A predefined data extraction form was developed to extract data from all included articles (see Additional file 1 ). The data extraction form was developed according to Joanna Briggs Institute (JBI) [ 23 ] and Cochrane [ 24 ] manuals, as well as the literature associated with concepts and methods in mental health research. The data extraction form was categorised into sub-sections, such as study details (citation, year of publication, author, contact details of lead author, and funder/sponsoring organisation, publication type), objective of the paper, primary subject area of the paper (study design, methods, sampling, data collection, data analysis, quality assurance). The data extraction form also had a section on additional information on methodological consideration, recommendations and other potential references. The authors extracted results of the included papers in numerical and textual format [ 23 ]. EB (Eric Badu) conducted the data extraction, A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell), conducted the second review of the extracted data.

Data synthesis

Content analysis was used to synthesise the extracted data. The content analysis process involved several stages which involved noting patterns and themes, seeing plausibility, clustering, counting, making contrasts and comparisons, discerning common and unusual patterns, subsuming particulars into general, noting relations between variability, finding intervening factors and building a logical chain of evidence [ 21 ] (see Table  2 ).

Study characteristics

The integrative review identified a total of 491 records from all databases, after which 19 duplicates were removed. Out of this, 472 titles and abstracts were assessed for eligibility, after which 439 articles were excluded. Articles not meeting the inclusion criteria were excluded. Specifically, papers excluded were those that did not address methodological issues as well as papers addressing methodological consideration in other disciplines. A total of 33 full-text articles were assessed – 9 articles were further excluded, whilst an additional 3 articles were identified from reference lists. Overall, 27 articles were included in the final synthesis (see Fig. 1 ). Of the total included papers, 12 contained qualitative research, 9 were mixed methods (both qualitative and quantitative) and 6 papers focused on quantitative data. Conversely, a total of 14 papers targeted global mental health research and 2 papers each describing studies in Germany, Sweden and China. The papers addressed different methodological issues, such as study design, methods, data collection, and analysis as well as quality assurance (see Table  3 ).

Mixed methods design in mental health research

Mixed methods research is defined as a research process where the elements of qualitative and quantitative research are combined in the design, data collection, and its triangulation and validation [ 48 ]. The integrative review identified four sub-themes that describe mixed methods design in the context of mental health research. The sub-themes include the categories of mixed methods, their function, structure, process and further methodological considerations for mixed methods design. These sub-themes are explained as follows:

Categorizing mixed methods in mental health research

Four studies highlighted the categories of mixed methods design applicable to mental health research [ 18 , 19 , 43 , 48 ]. Generally, there are differences in the categories of mixed methods design, however, three distinct categories predominantly appear to cross cut in all studies. These categories are function, structure and process. Some studies further categorised mixed method design to include rationale, objectives, or purpose. For instance, Schoonenboom and Johnson [ 48 ] categorised mixed methods design into primary and secondary dimensions.

The function of mixed methods in mental health research

Six studies explain the function of conducting mixed methods design in mental health research. Two studies specifically recommended that mixed methods have the ability to provide a more robust understanding of services by expanding and strengthening the conclusions from the study [ 42 , 45 ]. More importantly, the use of both qualitative and quantitative methods have the ability to provide innovative solutions to important and complex problems, especially by addressing diversity and divergence [ 48 ]. The review identified five underlying functions of a mixed method design in mental health research which include achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ].

The use of mixed methods to achieve convergence aims to employ both qualitative and quantitative data to answer the same question, either through triangulation (to confirm the conclusions from each of the methods) or transformation (using qualitative techniques to transform quantitative data). Similarly, complementarity in mixed methods integrates both qualitative and quantitative methods to answer questions for the purpose of evaluation or elaboration [ 18 , 19 , 43 ]. Two papers recommend that qualitative methods are used to provide the depth of understanding, whilst the quantitative methods provide a breadth of understanding [ 18 , 43 ]. In mental health research, the qualitative data is often used to examine treatment processes, whilst the quantitative methods are used to examine treatment outcomes against quality care key performance targets.

Additionally, three papers indicated that expansion as a function of mixed methods uses one type of method to answer questions raised by the other type of method [ 18 , 19 , 43 ]. For instance, qualitative data is used to explain findings from quantitative analysis. Also, some studies highlight that development as a function of mixed methods aims to use one method to answer research questions, and use the findings to inform other methods to answer different research questions. A qualitative method, for example, is used to identify the content of items to be used in a quantitative study. This approach aims to use qualitative methods to create a conceptual framework for generating hypotheses to be tested by using a quantitative method [ 18 , 19 , 43 ]. Three papers suggested that using mixed methods for the purpose of sampling utilize one method (eg. quantitative) to identify a sample of participants to conduct research using other methods (eg. qualitative) [ 18 , 19 , 43 ]. For instance, quantitative data is sequentially utilized to identify potential participants to participate in a qualitative study and the vice versa.

Structure of mixed methods in mental health research

Five studies categorised the structure of conducting mixed methods in mental health research, into two broader concepts including simultaneous (concurrent) and sequential (see Table 3 ). In both categories, one method is regarded as primary and the other as secondary, although equal weight can be given to both methods [ 18 , 19 , 42 , 43 , 48 ]. Two studies suggested that the sequential design is a process where the data collection and analysis of one component (eg. quantitative) takes place after the data collection and analysis of the other component (eg qualitative). Herein, the data collection and analysis of one component (e.g. qualitative) may depend on the outcomes of the other component (e.g. quantitative) [ 43 , 48 ]. An earlier review suggested that the majority of contemporary studies in mental health research use a sequential design, with qualitative methods, more often preceding quantitative methods [ 18 ].

Alternatively, the concurrent design collects and analyses data of both components (e.g. quantitative and qualitative) simultaneously and independently. Palinkas, Horwitz [ 42 ] recommend that one component is used as secondary to the other component, or that both components are assigned equal priority. Such a mixed methods approach aims to provide a depth of understanding afforded by qualitative methods, with the breadth of understanding offered by the quantitative data to elaborate on the findings of one component or seek convergence through triangulation of the results. Schoonenboom and Johnson [ 48 ] recommended the use of capital letters for one component and lower case letters for another component in the same design to indicate that one component is primary and the other is secondary or supplemental.

Process of mixed methods in mental health research

Five papers highlighted the process for the use of mixed methods in mental health research [ 18 , 19 , 42 , 43 , 48 ]. The papers suggested three distinct processes or strategies for combining qualitative and quantitative data. These include merging or converging the two data sets, connecting the two datasets by having one build upon the other; and embedding one data set within the other [ 19 , 43 ]. The process of connecting occurs when the analysis of one dataset leads to the need for the other data set. For instance, in the situation where quantitative results lead to the subsequent collection and analysis of qualitative data [ 18 , 43 ]. A previous study suggested that most studies in mental health sought to connect the data sets. Similarly, the process of merging the datasets brings together two sets of data during the interpretation, or transforms one type of data into the other type, by combining the data into new variables [ 18 ]. The process of embedding data into mixed method designs in mental health uses one dataset to provide a supportive role to the other dataset [ 43 ].

Consideration for using mixed methods in mental health research

Three studies highlighted several factors that need to be considered when conducting mixed methods design in mental health research [ 18 , 19 , 45 ]. Accordingly, these factors include developing familiarity with the topic under investigation based on experience, willingness to share information on the topic [ 19 ], establishing early collaboration, willingness to negotiate emerging problems, seeking the contribution of team members, and soliciting third-party assistance to resolve any emerging problems [ 45 ]. Additionally, Palinkas, Horwitz [ 18 ] recommended that mixed methods in the context of mental health research are mostly applied in studies that assess needs of services, examine existing services, developing new or adapting existing services, evaluating services in randomised control trials, and examining service implementation.

Qualitative study in mental health research

This theme describes the various qualitative methods used in mental health research. The theme also addresses methodological considerations for using qualitative methods in mental health research. The key emerging issues are discussed below:

Considering qualitative components in conducting mental health research

Six studies recommended the use of qualitative methods in mental health research [ 19 , 26 , 28 , 32 , 36 , 44 ]. Two qualitative research paradigms were identified, including the interpretive and critical approach [ 32 ]. The interpretive methodologies predominantly explore the meaning of human experiences and actions, whilst the critical approach emphasises the social and historical origins and contexts of meaning [ 32 ]. Two studies suggested that the interpretive qualitative methods used in mental health research are ethnography, phenomenology and narrative approaches [ 32 , 36 ].

The ethnographic approach describes the everyday meaning of the phenomena within a societal and cultural context, for instance, the way phenomena or experience is contrasted within a community, or by collective members over time [ 32 ]. Alternatively, the phenomenological approach explores the claims and concerns of a subject with a speculative development of an interpretative account within their cultural and physical environments focusing on the lived experience [ 32 , 36 ].

Moreover, the critical qualitative approaches used in mental health research are predominantly emancipatory (for instance, socio-political traditions) and participatory action-based research. The emancipatory traditions recognise that knowledge is acquired through critical discourse and debate but are not seen as discovered by objective inquiry [ 32 ]. Alternatively, the participatory action based approach uses critical perspectives to engage key stakeholders as participants in the design and conduct of the research [ 32 ].

Some studies highlighted several reasons why qualitative methods are relevant to mental health research. In particular, qualitative methods are significant as they emphasise naturalistic inquiry and have a discovery-oriented approach [ 19 , 26 ]. Two studies suggested that qualitative methods are often relevant in the initial stages of research studies to understand specific issues such as behaviour, or symptoms of consumers of mental services [ 19 ]. Specifically, Palinkas [ 19 ] suggests that qualitative methods help to obtain initial pilot data, or when there is too little previous research or in the absence of a theory, such as provided in exploratory studies, or previously under-researched phenomena.

Three studies stressed that qualitative methods can help to better understand socially sensitive issues, such as exploring the solutions to overcome challenges in mental health clinical policies [ 19 , 28 , 44 ]. Consequently, Razafsha, Behforuzi [ 44 ] recommended that the natural holistic view of qualitative methods can help to understand the more recovery-oriented policy of mental health, rather than simply the treatment of symptoms. Similarly, the subjective experiences of consumers using qualitative approaches have been found useful to inform clinical policy development [ 28 ].

Sampling in mental health research

The theme explains the sampling approaches used in mental health research. The section also describes the methodological considerations when sampling participants for mental health research. The sub-themes emerging are explained in the following sections:

Sampling approaches (quantitative)

Some studies reviewed highlighted the sampling approaches previously used in mental health research [ 25 , 34 , 35 ]. Generally, all quantitative studies tend to use several probability sampling approaches, whilst qualitative studies used non-probability techniques. The quantitative mental health studies conducted at community and population level employ multi-stage sampling techniques usually involving systematic sampling, stratified and random sampling [ 25 , 34 ]. Similarly, quantitative studies that recruit consumers in the hospital setting employ consecutive sampling [ 35 ]. Two studies reviewed highlighted that the identification of consumers of mental health services for research is usually conducted by service providers. For instance, Korver, Quee [ 35 ] research used a consecutive sampling approach by identifying consumers through clinicians working in regional psychosis departments, or academic centres.

Sampling approaches (qualitative)

Seven studies suggested that the sampling procedures widely used in mental health research involving qualitative methods are non-probability techniques, which include purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ]. The purposive sampling identifies participants that possess relevant characteristics to answer a research question [ 28 ]. Purposive sampling can be used in a single case study, or for multiple cases. The purposive sampling used in mental health research is usually extreme, or deviant case sampling, criterion sampling, and maximum variation sampling [ 19 ]. Furthermore, it is advised when using purposive sampling in a multistage level study, that it should aim to begin with the broader picture to achieve variation, or dispersion, before moving to the more focused view that considers similarity, or central tendencies [ 42 ].

Two studies added that theoretical sampling involved sampling participants, situations and processes based on concepts on theoretical grounds and then using the findings to build theory, such as in a Grounded Theory study [ 31 , 32 ]. Some studies highlighted that snowball sampling is another strategy widely used in mental health research [ 30 , 32 , 46 ]. This is ascribed to the fact that people with mental illness are perceived as marginalised in research and practically hard-to-reach using conventional sampling [ 30 , 32 ]. Snowballing sampling involves asking the marginalised participants to recommend individuals who might have direct knowledge relevant to the study [ 30 , 32 , 46 ]. Although this approach is relevant, some studies advise the limited possibility of generalising the sample, because of the likelihood of selection bias [ 30 ].

Sampling consideration

Four studies in this section highlighted some of the sampling considerations in mental health research [ 30 , 31 , 32 , 46 ]. Generally, mental health research should consider the appropriateness and adequacy of sampling approach by applying attributes such as shared social, or cultural experiences, or shared concern related to the study [ 32 ], diversity and variety of participants [ 31 ], practical and organisational skills, as well as ethical and sensitivity issues [ 46 ]. Robinson [ 46 ] further suggested that sampling can be homogenous or heterogeneous depending on the research questions for the study. Achieving homogeneity in sampling should employ a variety of parameters, which include demographic, graphical, physical, psychological, or life history homogeneity [ 46 ]. Additionally, applying homogeneity in sampling can be influenced by theoretical and practical factors. Alternatively, some samples are intentionally selected based on heterogeneous factors [ 46 ].

Data collection in mental health research

This theme highlights the data collection methods used in mental health research. The theme is explained according to three sub-themes, which include approaches for collecting qualitative data, methodological considerations, as well as preparations for data collection. The sub-themes are as follows:

Approaches for collecting qualitative data

The studies reviewed recommended the approaches that are widely applied in collecting data in mental health research. The widely used qualitative data collection approaches in mental health research are focus group discussions (FGDs) [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ], extended in-depth interviews [ 19 , 30 , 34 ], participant and non-participant observation [ 19 ], Delphi data collection, quasi-statistical techniques [ 19 ] and field notes [ 31 , 40 ]. Seven studies suggest that FGDs are widely used data collection approaches [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ] because they are valuable in gathering information on consumers’ perspectives of services, especially regarding satisfaction, unmet/met service needs and the perceived impact of services [ 47 ]. Conversely, Ekblad and Baarnhielm [ 31 ] recommended that this approach is relevant to improve clinical understanding of the thoughts, emotions, meanings and attitudes towards mental health services.

Such data collection approaches are particularly relevant to consumers of mental health services, due to their low self-confidence and self-esteem [ 41 ]. The approach can help to understand specific terms, vocabulary, opinions and attitudes of consumers of mental health services, as well as their reasoning about personal distress and healing [ 31 ]. Similarly, the reliance on verbal rather than written communication helps to promote the participation of participants with serious and enduring mental health problems [ 31 , 41 ]. Although FGD has several important outcomes, there are some limitations that need critical consideration. Ekblad and Baarnhielm [ 31 ] for example suggest, that marginalised participants may not always feel free to talk about private issues regarding their condition at the group level mostly due to perceived stigma and group confidentiality.

Some studies reviewed recommended that attempting to capture comprehensive information and analysing group interactions in mental health research requires the research method to use field notes as a supplementary data source to help validate the FGDs [ 31 , 40 , 41 ]. The use of field notes in addition to FGDs essentially provides greater detail in the accounts of consumers’ subjective experiences. Furthermore, Montgomery and Bailey [ 40 ] suggest that field notes require observational sensitivity, and also require having specific content such as descriptive and interpretive data.

Three studies in this section suggested that in-depth interviews are used to collect data from consumers of mental health services [ 19 , 30 , 34 ]. This approach is particularly important to explore the behaviour, subjective experiences and psychological processes; opinions, and perceptions of mental health services. de Jong and Van Ommeren [ 30 ] recommend that in-depth interviews help to collect data on culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, symptom symbols and protective mediators. Palinkas [ 19 ] also highlights that the structured narrative form of extended interviewing is the type of in-depth interview used in mental health research. This approach provides participants with the opportunity to describe the experience of living with an illness and seeking services that assist them.

Consideration for data collection

Six studies recommended consideration required in the data collection process [ 31 , 32 , 37 , 41 , 47 , 49 ]. Some studies highlighted that consumers of mental health services might refuse to participate in research due to several factors [ 37 ] like the severity of their illness, stigma and discrimination [ 41 ]. Subsequently, such issues are recommended to be addressed by building confidence and trust between the researcher and consumers [ 31 , 37 ]. This is a significant prerequisite, as it can sensitise and normalise the research process and aims with the participants prior to discussing their personal mental health issues. Similarly, some studies added that the researcher can gain the confidence of service providers who manage consumers of mental health services [ 41 , 47 ], seek ethical approval from the relevant committee(s) [ 41 , 47 ], meet and greet the consumers of mental health services before data collection, and arrange a mutually acceptable venue for the groups and possibly supply transport [ 41 ].

Two studies further suggested that the cultural and social differences of the participants need consideration [ 26 , 31 ]. These factors could influence the perception and interpretation of ethical issues in the research situation.

Additionally, two studies recommended the use of standardised assessment instruments for mental health research that involve quantitative data collection [ 33 , 49 ]. A recent survey suggested that measures to standardise the data collection approach can convert self-completion instruments to interviewer-completion instruments [ 49 ]. The interviewer can then read the items of the instruments to respondents and record their responses. The study further suggested the need to collect demographic and behavioural information about the participant(s).

Preparing for data collection

Eight studies highlighted the procedures involved in preparing for data collection in mental health research [ 25 , 30 , 33 , 34 , 35 , 39 , 41 , 49 ]. These studies suggest that the preparation process involve organising meetings of researchers, colleagues and representatives of the research population. The meeting of researchers generally involves training of interviewers about the overall design, objectives and research questions associated with the study. de Jong and Van Ommeren [ 30 ] recommended that preparation for the use of quantitative data encompasses translating and adapting instruments with the aim of achieving content, semantic, concept, criterion and technical equivalence.

Quality assurance procedures in mental health research

This section describes the quality assurance procedures used in mental health research. Quality assurance is explained according to three sub-themes: 1) seeking informed consent, 2) the procedure for ensuring quality assurance in a quantitative study and 3) the procedure for ensuring quality control in a qualitative study. The sub-themes are explained in the following content.

Seeking informed consent

The papers analysed for the integrative review suggested that the rights of participants to safeguard their integrity must always be respected, and so each potential subject must be adequately informed of the aims, methods, anticipated benefits and potential hazards of the study and any potential discomforts (see Table 3 ). Seven studies highlight that potential participants of mental health research must be consented to the study prior to data collection [ 25 , 26 , 33 , 35 , 37 , 39 , 47 ]. The consent process helps to assure participants of anonymity and confidentiality and further explain the research procedure to them. Baarnhielm and Ekblad [ 26 ] argue that the research should be guided by four basic moral values for medical ethics, autonomy, non-maleficence, beneficence, and justice. In particular, potential consumers of mental health services who may have severe conditions and unable to consent themselves are expected to have their consent signed by a respective family caregiver [ 37 ]. Latvala, Vuokila-Oikkonen [ 37 ] further suggested that researchers are responsible to agree on the criteria to determine the competency of potential participants in mental health research. The criteria are particularly relevant when potential participants have difficulties in understanding information due to their mental illness.

Procedure for ensuring quality control (quantitative)

Several studies highlighted procedures for ensuring quality control in mental health research (see Table 3 ). The quality control measures are used to achieve the highest reliability, validity and timeliness. Some studies demonstrate that ensuring quality control should consider factors such as pre-testing tools [ 25 , 49 ], minimising non-response rates [ 25 , 39 ] and monitoring of data collection processes [ 25 , 33 , 49 ].

Accordingly, two studies suggested that efforts should be made to re-approach participants who initially refuse to participate in the study. For instance, Liu, Huang [ 39 ] recommended that when a consumer of mental health services refuse to participate in a study (due to low self-esteem) when approached for the first time, a different interviewer can re-approach the same participant to see if they are more comfortable to participate after the first invitation. Three studies further recommend that monitoring data quality can be accomplished through “checks across individuals, completion status and checks across variables” [ 25 , 33 , 49 ]. For example, Alonso, Angermeyer [ 25 ] advocate that various checks are used to verify completion of the interview, and consistency across instruments against the standard procedure.

Procedure for ensuring quality control (qualitative)

Four studies highlighted the procedures for ensuring quality control of qualitative data in mental health research [ 19 , 32 , 37 , 46 ]. A further two studies suggested that the quality of qualitative research is governed by the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. Some studies explain that the credibility or trustworthiness of qualitative research in mental health is determined by methodological and interpretive rigour of the phenomenon being investigated [ 32 , 37 ]. Consequently, Fossey, Harvey [ 32 ] propose that the methodological rigour for assessing the credibility of qualitative research are congruence, responsiveness or sensitivity to social context, appropriateness (importance and impact), adequacy and transparency. Similarly, interpretive rigour is classified as authenticity, coherence, reciprocity, typicality and permeability of the researcher’s intentions; including engagement and interpretation [ 32 ].

Robinson [ 46 ] explained that transparency (openness and honesty) is achieved if the research report explicitly addresses how the sampling, data collection, analysis, and presentation are met. In particular, efforts to address these methodological issues highlight the extent to which the criteria for quality profoundly interacts with standards for ethics. Similarly, responsiveness, or sensitivity, helps to situate or locate the study within a place, a time and a meaningful group [ 46 ]. The study should also consider the researcher’s background, location and connection to the study setting, particularly in the recruitment process. This is often described as role conflict or research bias.

In the interpretive phenomenon, coherence highlights the ability to select an appropriate sampling procedure that mutually matches the research aims, questions, data collection, analysis, as well as any theoretical concepts or frameworks [ 32 , 46 ]. Similarly, authenticity explains the appropriate representation of participants’ perspectives in the research process and the interpretation of results. Authenticity is maximised by providing evidence that participants are adequately represented in the interpretive process, or provided an opportunity to give feedback on the researcher’s interpretation [ 32 ]. Again, the contribution of the researcher’s perspective to the interpretation enhances permeability. Fossey, Harvey [ 32 ] further suggest that reflexive reporting, which distinguishes the participants’ voices from that of the researcher in the report, enhances the permeability of the researcher’s role and perspective.

One study highlighted the approaches used to ensure validity in qualitative research, which includes saturation, identification of deviant or non-confirmatory cases, member checking and coding by consensus. Saturation involves completeness in the research process, where all relevant data collection, codes and themes required to answer the phenomenon of inquiry are achieved; and no new data emerges [ 19 ]. Similarly, member checking is the process whereby participants or others who share similar characteristics review study findings to elaborate on confirming them [ 19 ]. The coding by consensus involves a collaborative approach to analysing the data. Ensuring regular meetings among coders to discuss procedures for assigning codes to segments of data and resolve differences in coding procedures, and by comparison of codes assigned on selected transcripts to calculate a percentage agreement or kappa measure of interrater reliability, are commonly applied [ 19 ].

Two studies recommend the need to acknowledge the importance of generalisability (transferability). This concept aims to provide sufficient information about the research setting, findings and interpretations for readers to appropriately determine the replicability of the findings from one context, or population to another, otherwise known as reliability in quantitative research [ 19 , 32 ]. Similarly, the researchers should employ reflexivity as a means of identifying and addressing potential biases in data collection and interpretation. Palinkas [ 19 ] suggests that such bias is associated with theoretical orientations; pre-conceived beliefs, assumptions, and demographic characteristics; and familiarity and experience with the methods and phenomenon. Another approach to enhance the rigour of analysis involves peer debriefing and support meetings held among team members which facilitate detailed auditing during data analysis [ 19 ].

The integrative review was conducted to synthesise evidence into recommended methodological considerations when conducting mental health research. The evidence from the review has been discussed according to five major themes: 1) mixed methods study in mental health research; 2) qualitative study in mental health research; 3) sampling in mental health research; 4) data collection in mental health research; and 5) quality assurance procedures in mental health research.

Mixed methods study in mental health research

The evidence suggests that mixed methods approach in mental health are generally categorised according to their function (rationale, objectives or purpose), structure and process [ 18 , 19 , 43 , 48 ]. The mixed methods study can be conducted for the purpose of achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ]. Researchers conducting mental health studies should understand the underlying functions or purpose of mixed methods. Similarly, mixed methods in mental health studies can be structured simultaneously (concurrent) and sequential [ 18 , 19 , 42 , 43 , 48 ]. More importantly, the process of combining qualitative and quantitative data can be achieved through merging or converging, connecting and embedding one data set within the other [ 18 , 19 , 42 , 43 , 48 ]. The evidence further recommends that researchers need to understand the stage of integrating the two sets of data and the rationale for doing so. This can inform researchers regarding the best stage and appropriate ways of combining the two components of data to adequately address the research question(s).

The evidence recommended some methodological consideration in the design of mixed methods projects in mental health [ 18 , 19 , 45 ]. These issues include establishing early collaboration, becoming familiar with the topic, sharing information on the topic, negotiating any emerging problems and seeking contributions from team members. The involvement of various expertise could ensure that methodological issues are clearly identified. However, addressing such issues midway, or late through the design can negatively affect the implementation [ 45 ]. Any robust discoveries can rarely be accommodated under the existing design. Therefore, the inclusion of various methodological expertise during inception can lead to a more robust mixed-methods design which maximises the contributions of team members. Whilst fundamental and philosophical differences in qualitative and quantitative methods may not be resolved, some workable solutions can be employed, particularly if challenges are viewed as philosophical rather than personal [ 45 ]. The cultural issues can be alleviated by understanding the concepts, norms and values of the setting, further to respecting and including perspectives of the various stakeholders.

The review findings suggest that qualitative methods are relevant when conducting mental health research. The qualitative methods are mostly used where there has been limited previous research and an absence of theoretical perspectives. The approach is also used to gather initial pilot data. More importantly, the qualitative methods are relevant when we want to understand sensitive issues, especially from consumers of mental health services, where the ‘lived experience is paramount [ 19 , 28 , 44 ]. Qualitative methods can help understand the experiences of consumers in the process of treatment, as well as their therapeutic relationship with mental health professionals. The experiences of consumers from qualitative data are particularly important in developing clinical policy [ 28 ]. The review findings find two paradigms of qualitative methods are used in mental health research. These paradigms are the interpretive and critical approach [ 32 ]. The interpretive qualitative method(s) include phenomenology, ethnography and narrative approaches [ 32 , 36 ]. Conversely, critical qualitative approaches are participatory action research and emancipatory approach. The review findings suggest that these approaches to qualitative methods need critical considerations, particularly when dealing with consumers of mental health services.

The review findings identified several sampling techniques used in mental health research. Quantitative studies, usually employ probability sampling, whilst qualitative studies use non-probability sampling [ 25 , 34 ]. The most common sampling techniques for quantitative studies are multi-stage sampling, which involves systematic, stratified, random sampling and consecutive sampling. In contrast, the predominant sampling approaches for qualitative studies are purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ].

The sampling of consumers of mental health services requires some important considerations. The sampling should consider the appropriateness and adequacy of the sampling approach, diversity and variety of consumers of services, attributes such as social, or cultural experiences, shared concerns related to the study, practical and organisational skills, as well as ethical and sensitivity issues are all relevant [ 31 , 32 , 46 ]. Sampling consumers of mental health services should also consider the homogeneity and heterogeneity of consumers. However, failure to address these considerations can present difficulty in sampling and subsequently result in selection and reporting bias in mental health research.

The evidence recommends several data collection approaches in collecting data in mental health research, including focus group discussion, extended in-depth interviews, observations, field notes, Delphi data collection and quasi-statistical techniques. The focus group discussions appear as an approach widely used to collect data from consumers of mental health services [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ]. The focus group discussion appears to be a significant source of obtaining information. This approach promotes the participation of consumers with severe conditions, particularly at the group level interaction. Mental health researchers are encouraged to use this approach to collect data from consumers, in order to promote group level interaction. Additionally, field notes can be used to supplement information and to more deeply analyse the interactions of consumers of mental health services. Field notes are significant when wanting to gather detailed accounts about the subjective experiences of consumers of mental health services [ 40 ]. Field notes can help researchers to capture the gestures and opinions of consumers of mental health services which cannot be covered in the audio-tape recording. Particularly, the field note is relevant to complement the richness of information collected through focus group discussion from consumers of mental health services.

Furthermore, it was found that in-depth interviews can be used to explore specific mental health issues, particularly culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, as well as symptom symbols and protective mediators [ 19 , 30 , 34 ]. The in-depth interviews are particularly relevant if the study is interested in the lived experiences of consumers without the contamination of others in a group situation. The in-depth interviews are relevant when consumers of mental health services are uncomfortable in disclosing their confidential information in front of others [ 31 ]. The lived experience in a phenomenological context preferably allows the consumer the opportunity to express themselves anonymously without any tacit coercion created by a group context.

The review findings recommend significant factors requiring consideration when collecting data in mental health research. These considerations include building confidence and trust between the researcher and consumers [ 31 , 37 ], gaining confidence of mental health professionals who manage consumers of mental health services, seeking ethical approval from the relevant committees, meeting consumers of services before data collection as well as arranging a mutually acceptable venue for the groups and providing transport services [ 41 , 47 ]. The evidence confirms that the identification of consumers of mental health services to participate in research can be facilitated by mental health professionals. Similarly, the cultural and social differences of the consumers of mental health services need consideration when collecting data from them [ 26 , 31 ].

Moreover, our review advocates that standardised assessment instruments can be used to collect data from consumers of mental health services, particularly in quantitative data. The self-completion instruments for collecting such information can be converted to interviewer-completion instruments [ 33 , 49 ]. The interviewer can read the questions to consumers of mental health services and record their responses. It is recommended that collecting data from consumers of mental health services requires significant preparation, such as training with co-investigators and representatives from consumers of mental health services [ 25 , 30 , 33 , 34 , 35 , 39 , 49 ]. The training helps interviewers and other investigators to understand the research project, particularly translating and adapting an instrument for the study setting with the aim to achieve content, semantic, concept, criteria and technical equivalence [ 30 ]. The evidence indicates that there is a need to adequately train interviewers when preparing for fieldwork to collect data from consumers of mental health services.

The evidence provides several approaches that can be employed to ensure quality assurance in mental health research involving quantitative methods. The quality assurance approach encompasses seeking informed consent from consumers of mental health services [ 26 , 37 ], pre-testing of tools [ 25 , 49 ], minimising non-response rates and monitoring of the data collection process [ 25 , 33 , 49 ]. The quality assurance process in mental health research primarily aims to achieve the highest reliability, validity and timeliness, to improve the quality of care provided. For instance, the informed consent exposes consumers of mental health services to the aim(s), methods, anticipated benefits and potential hazards and discomforts of participating in the study. Herein, consumers of mental health services who cannot respond to the inform consent process because of the severity of their illness can have it signed by their family caregivers. The implication is that researchers should determine which category of consumers of mental health services need family caregivers involved in the consent process [ 37 ].

The review findings advises that researchers should use pre-testing to evaluate the data collection procedure on a small scale and then to subsequently make any necessary changes [ 25 ]. The pre-testing aims to help the interviewers get acquainted with the procedures and to detect any potential problems [ 49 ]. The researchers can discuss the findings of the pre-testing and then further resolve any challenges that may arise prior to the actual field work being commenced. The non-response rates in mental health research can be minimised by re-approaching consumers of mental health services who initially refuse to participate in the study.

In addition, quality assurance for qualitative data can be ensured by applying the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. It was found that the credibility of qualitative research in mental health is achieved through methodological and interpretive rigour [ 32 , 37 ]. The methodological rigour for assessing credibility relates to congruence, responsiveness or sensitivity to a social context, appropriateness, adequacy and transparency. By contrast, ensuring interpretive rigour is achieved through authenticity, coherence, reciprocity, typicality and permeability of researchers’ intentions, engagement and interpretation [ 32 , 46 ].

Strengths and limitations

The evidence has several strengths and limitations that require interpretation and explanation. Firstly, we employed a systematic approach involving five stages of problem identification, literature search, data evaluation, data synthesis and presentation of results [ 21 ]. Similarly, we searched six databases and developed a data extraction form to extract information. The rigorous process employed in this study, for instance, searching databases and data extraction forms, helped to capture comprehensive information on the subject.

The integrative review has several limitations largely related to the search words, language limitations, time period and appraisal of methodological quality of included papers. In particular, the differences in key terms and words concerning methodological issues in the context of mental health research across cultures and organisational contexts may possibly have missed some relevant articles pertaining to the study. Similarly, limiting included studies to only English language articles and those published from January 2000 to July 2018 could have missed useful articles published in other languages and those published prior to 2000. The review did not assess the methodological quality of included papers using a critical appraisal tool, however, the combination of clearly articulated search methods, consultation with the research librarian, and reviewing articles with methodological experts in mental health research helped to address the limitations.

The review identified several methodological issues that need critical attention when conducting mental health research. The evidence confirms that studies that addressed methodological considerations in conducting mental health research largely focuses on qualitative studies in a transcultural setting, in addition to lessons from multi-site surveys in mental health research. Specifically, the methodological issues related to the study design, sampling, data collection processes and quality assurance are critical to the research design chosen for any particular study. The review highlighted that researchers conducting mental health research can establish early collaboration, familiarise themselves with the topic, share information on the topic, negotiate to resolve any emerging problems and seek the contribution of clinical (or researcher) team members on the ground. In addition, the recruitment of consumers of mental health services should consider the appropriateness and adequacy of sampling approaches, diversity and variety of consumers of services, their social or cultural experiences, practical and organisational skills, as well as ethical and sensitivity issues.

The evidence confirms that in an attempt to effectively recruit and collect data from consumers of mental health services, there is the need to build confidence and trust between the researcher and consumers; and to gain the confidence of mental health service providers. Furthermore, seeking ethical approval from the relevant committee, meeting with consumers of services before data collection, arranging a mutually acceptable venue for the groups, and providing transport services, are all further important considerations. The review findings establish that researchers conducting mental health research should consider several quality assurance issues. Issues such as adequate training prior to data collection, seeking informed consent from consumers of mental health services, pre-testing of tools, minimising non-response rates and monitoring of the data collection process. More specifically, quality assurance for qualitative data can be achieved by applying the principles of credibility, dependability, transferability, reflexivity, confirmability.

Based on the findings from this review, it is recommended that mental health research should adequately consider the methodological issues regarding study design, sampling, data collection procedures and quality assurance issues to effectively conduct meaningful research.

Availability of data and materials

Not applicable

Abbreviations

focus group discussions

Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

National Ethics Advisory Committee. Ethical guidelines for intervention studies: revised edition. Wellington (New Zealand): Ministry of Health. 2012.

Mann C. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J. 2003;20(1):54–60.

Article   CAS   Google Scholar  

DiPietro NA. Methods in epidemiology: observational study designs. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2010;30(10):973–84.

Article   Google Scholar  

Hong NQ, Pluyr P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed Methods Appraisal Tool (MMAT). Canada.: Intellectual Property Office, Canada; 2018.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches: sage publications; 2017.

Google Scholar  

Wisdom J, Creswell JW. Mixed methods: integrating quantitative and qualitative data collection and analysis while studying patient-centered medical home models. Rockville: Agency for Healthcare Research and Quality; 2013.

Bonita R, Beaglehole R, Kjellström T. Basic epidemiology: World Health Organization; 2006.

Centers for Disease Control Prevention [CDC]. Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics. Atlanta, GA: US Dept. of Health and Human Services, Centers for Disease Control and Prevention (CDC), Office of Workforce and Career Development; 2012.

Parab S, Bhalerao S. Study designs. International journal of Ayurveda research. 2010;1(2):128.

Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trials. Diabetes Res Clin Pract. 2010;88:S3–9.

Department of Family Medicine (McGill University). Mixed Methods Appraisal Tool (MMAT) – Version 2011 Canada: McGill University; 2011 [Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/84371689/MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.08.21.pdf .

Besen J, Gan DS. A critical evaluation of clinical research study designs. J Investig Dermatol. 2014;134.

Axelrod DA, Hayward R. Nonrandomized interventional study designs (quasi-experimental designs). Clinical research methods for surgeons: Springer; 2006. p. 63–76.

Thiese MS. Observational and interventional study design types; an overview. Biochemia medica: Biochemia medica. 2014;24(2):199–210.

Velengtas P, Mohr P, Messner DA. Making informed decisions: assessing the strengths and weaknesses of study designs and analytic methods for comparative effectiveness research. National Pharmaceutical Council 2012.

Guerrera F, Renaud S, Tabbò F, Filosso PL. How to design a randomized clinical trial: tips and tricks for conduct a successful study in thoracic disease domain. Journal of thoracic disease. 2017;9(8):2692.

Bhide A, Shah PS, Acharya G. A simplified guide to randomized controlled trials. Acta Obstet Gynecol Scand. 2018;97(4):380–7.

Palinkas L, Horwitz SM, Chamberlain P, Hurlburt MS, Landsverk J. Mixed-methods designs in mental health services research: a review. Psychiatr Serv. 2011;62(3):255–63.

Palinkas L. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

World Health Organization [WHO]. Mental health: a state of well-being 2014 [Available from: http://www.who.int/features/factfiles/mental_health/en/ .

Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

Hopia H, Latvala E, Liimatainen L. Reviewing the methodology of an integrative review. Scand J Caring Sci. 2016;30(4):662–9.

Pearson A, White H, Bath-Hextall F, Apostolo J, Salmond S, Kirkpatrick P. Methodology for JBI mixed methods systematic reviews. The Joanna Briggs Institute Reviewers Manual. 2014;1:5–34.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, et al. Sampling and methods of the European study of the epidemiology of mental disorders (ESEMeD) project. Acta Psychiatr Scand Suppl. 2004;109(420):8–20.

Baarnhielm S, Ekblad S. Qualitative research, culture and ethics: a case discussion. Transcultural Psychiatry. 2002;39(4):469–83.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Brown C, Lloyd K. Qualitative methods in psychiatric research. Adv Psychiatr Treat. 2001;7(5):350–6.

Davidsen AS. Phenomenological approaches in psychology and health sciences. Qual Res Psychol. 2013;10(3):318–39.

de Jong JT, Van Ommeren M. Toward a culture-informed epidemiology: combining qualitative and quantitative research in transcultural contexts. Transcultural Psychiatry. 2002;39(4):422–33.

Ekblad S, Baarnhielm S. Focus group interview research in transcultural psychiatry: reflections on research experiences. Transcultural Psychiatry. 2002;39(4):484–500.

Fossey E, Harvey C, McDermott F, Davidson L. Understanding and evaluating qualitative research. Aust N Z J Psychiatry. 2002;36(6):717–32.

Jacobi F, Wittchen H-U, Holting C, Sommer S, Lieb R, Hofler M, et al. Estimating the prevalence of mental and somatic disorders in the community: aims and methods of the German National Health Interview and examination survey. Int J Methods Psychiatr Res 2002;11(1):1–18.

Koch A, Vogel A, Holzmann M, Pfennig A, Salize HJ, Puschner B, et al. MEMENTA-‘Mental healthcare provision for adults with intellectual disability and a mental disorder’. A cross-sectional epidemiological multisite study assessing prevalence of psychiatric symptomatology, needs for care and quality of healthcare provision for adults with intellectual disability in Germany: a study protocol. BMJ Open. 2014;4(5):e004878.

Korver N, Quee PJ, Boos HB, Simons CJ, de Haan L, Investigators G. Genetic risk and outcome of psychosis (GROUP), a multi site longitudinal cohort study focused on gene–environment interaction: objectives, sample characteristics, recruitment and assessment methods. Int J Methods Psychiatr Res. 2012;21(3):205–21.

Larkin M, Watts S, Clifton E. Giving voice and making sense in interpretative phenomenological analysis. Qual Res Psychol. 2006;3(2):102–20.

Latvala E, Vuokila-Oikkonen P, Janhonen S. Videotaped recording as a method of participant observation in psychiatric nursing research. J Adv Nurs. 2000;31(5):1252–7.

Leese MN, White IR, Schene AH, Koeter MW, Ruggeri M, Gaite L. Reliability in multi-site psychiatric studies. Int J Methods Psychiatr Res. 2001;10(1):29–42.

Liu Z, Huang Y, Lv P, Zhang T, Wang H, Li Q, et al. The China mental health survey: II. Design and field procedures. Soc Psychiatry Psychiatr Epidemiol. 2016;51(11):1547–57.

Montgomery P, Bailey PH. Field notes and theoretical memos in grounded theory. West J Nurs Res. 2007;29(1):65–79.

Owen S. The practical, methodological and ethical dilemmas of conducting focus groups with vulnerable clients. J Adv Nurs. 2001;36(5):652–8.

Palinkas L, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health Ment Health Serv Res. 2015;42(5):533–44.

Palinkas L, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):44–53.

Razafsha M, Behforuzi H, Azari H, Zhang Z, Wang KK, Kobeissy FH, et al. Qualitative versus quantitative methods in psychiatric research. Methods Mol Biol. 2012;829:49–62.

Robins CS, Ware NC, Dosreis S, Willging CE, Chung JY, Lewis-Fernández R. Dialogues on mixed-methods and mental health services research: anticipating challenges, building solutions. Psychiatr Serv. 2008;59(7):727–31.

Robinson OC. Sampling in interview-based qualitative research: a theoretical and practical guide. Qual Res Psychol. 2014;11(1):25–41.

Schilder K, Tomov T, Mladenova M, Mayeya J, Jenkins R, Gulbinat W, et al. The appropriateness and use of focus group methodology across international mental health communities. International Review of Psychiatry. 2004;16(1–2):24–30.

Schoonenboom J, Johnson RB. How to construct a mixed methods research DesignWie man ein mixed methods-Forschungs-design konstruiert. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie. 2017;69(2):107–31.

Yin H, Phillips MR, Wardenaar KJ, Xu G, Ormel J, Tian H, et al. The Tianjin mental health survey (TJMHS): study rationale, design and methods. Int J Methods Psychiatr Res. 2017;26(3):09.

Download references

Acknowledgements

The authors wish to thank the University of Newcastle Graduate Research and the School of Nursing and Midwifery, for the Doctoral Scholarship offered to the lead author. The authors are also grateful for the support received from Ms. Debbie Booth, the Librarian for supporting the literature search.

Author information

Authors and affiliations.

School of Nursing and Midwifery, The University of Newcastle, Callaghan, Australia

Faculty of Health and Medicine, School Nursing and Midwifery, University of Newcastle, Callaghan, Australia

Anthony Paul O’Brien

Faculty of Business and Economics, Macquarie University, North Ryde, Australia

Rebecca Mitchell

You can also search for this author in PubMed   Google Scholar

Contributions

EB, APO’B, and RM conceptualized the study. EB conducted the data extraction, APO’B, and RM, conducted the second review of the extracted data. EB, working closely with APO’B and RM performed the content analysis and drafted the manuscript. EB, APO’B, and RM, reviewed and made inputs into the intellectual content and agreed on its submission for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eric Badu .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Data extraction form. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Badu, E., O’Brien, A.P. & Mitchell, R. An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance. Arch Public Health 77 , 37 (2019). https://doi.org/10.1186/s13690-019-0363-z

Download citation

Received : 13 November 2018

Accepted : 22 July 2019

Published : 10 October 2019

DOI : https://doi.org/10.1186/s13690-019-0363-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mental health
• Methodological approach
• Mixed methods
• Data collection

Archives of Public Health

ISSN: 2049-3258

• General enquiries: [email protected]

Change Password

Your password must have 6 characters or more:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

Create your account

Forget yout password.

Enter your email address below and we will send you the reset instructions

If the address matches an existing account you will receive an email with instructions to reset your password

Forgot your Username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

• March 01, 2024 | VOL. 75, NO. 3 CURRENT ISSUE pp.203-304
• February 01, 2024 | VOL. 75, NO. 2 pp.107-201
• January 01, 2024 | VOL. 75, NO. 1 pp.1-71

The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use , including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

Mixed-Methods Designs in Mental Health Services Research: A Review

• Lawrence A. Palinkas , Ph.D. ,
• Sarah M. Horwitz , Ph.D. ,
• Patricia Chamberlain , Ph.D. ,
• Michael S. Hurlburt , Ph.D. , and
• John Landsverk , Ph.D.

Search for more papers by this author

Despite increased calls for use of mixed-methods designs in mental health services research, how and why such methods are being used and whether there are any consistent patterns that might indicate a consensus about how such methods can and should be used are unclear.

Use of mixed methods was examined in 50 peer-reviewed journal articles found by searching PubMed Central and 60 National Institutes of Health (NIH)-funded projects found by searching the CRISP database over five years (2005–2009). Studies were coded for aims and the rationale, structure, function, and process for using mixed methods.

A notable increase was observed in articles published and grants funded over the study period. However, most did not provide an explicit rationale for using mixed methods, and 74% gave priority to use of quantitative methods. Mixed methods were used to accomplish five distinct types of study aims (assess needs for services, examine existing services, develop new or adapt existing services, evaluate services in randomized controlled trials, and examine service implementation), with three categories of rationale, seven structural arrangements based on timing and weighting of methods, five functions of mixed methods, and three ways of linking quantitative and qualitative data. Each study aim was associated with a specific pattern of use of mixed methods, and four common patterns were identified.

Conclusions:

These studies offer guidance for continued progress in integrating qualitative and quantitative methods in mental health services research consistent with efforts by NIH and other funding agencies to promote their use. ( Psychiatric Services 62:255–263, 2011)

In the past decade, mental health services researchers have increasingly used qualitative methods in combination with quantitative methods ( 1 , 2 ). This use of mixed methods has been partly driven by theoretical models that encourage assessment of consumer perspectives and of contextual influences on disparities in the delivery of mental health services and the dissemination and implementation of evidence-based practices ( 3 , 4 ). These models call for research designs that use quantitative and qualitative data collection and analysis for a better understanding of a research problem than might be possible with use of either methodological approach alone ( 5 , 6 ). Numerous typologies and guidelines for the use of mixed-methods designs exist in the fields of nursing ( 7 , 8 ), evaluation ( 9 , 10 ), public health ( 11 , 12 ), primary care ( 13 ), education ( 14 ), and the social and behavioral sciences ( 5 , 15 ).

As Robins and colleagues ( 1 ) have observed, however, there has been little guidance in mental health services research on how to blend quantitative and qualitative methods to build upon the strengths of their respective epistemologies. Such guidance has been limited by the lack of consensus on the criteria that might be used to evaluate the quality of such research ( 5 ). From a policy perspective, the impact of the efforts of the National Institute of Mental Health (NIMH) ( 3 , 4 ) and other institutes of the National Institutes of Health (NIH) and funding agencies in encouraging the use of mixed methods in mental health services research also remains poorly understood.

To address these issues, we examined the application of mixed-methods designs in a sample of mental health services research studies published in peer-reviewed journals and in NIMH-funded research projects over five years. Our aim was to determine how and why such methods were being used and whether there are any consistent patterns that might indicate a consensus among researchers as to how such methods can and should be used. This aim is viewed as an initial step toward the development of standards for effective uses of mixed methods in mental health services research and articulation of criteria for evaluating the quality and impact of this research.

We conducted a literature review of mental health services research publications over a five-year period (January 2005 to September 2009), using the PubMed Central database and the following search terms: mental health services, mixed methods, and qualitative methods. Data were taken from the full text of each research article. Articles identified as potential candidates for inclusion had to report empirical research and meet one of the following selection criteria: a study specifically identified as a mixed-methods study in the title or abstract or through keywords; a qualitative study conducted as part of a larger project, including a randomized controlled trial, that also included use of quantitative methods; or a study that “quantitized” qualitative data ( 16 ) or “qualitized” quantitative data ( 17 ). On the basis of criteria used by McKibbon and Gadd ( 18 ) and Cresswell and Plano Clark ( 5 ), the analysis had to be fairly substantial; for example, a simple descriptive analysis of baseline demographic characteristics of participants was not sufficient to be included as a mixed-methods study. Further, qualitative studies that were not clearly linked to quantitative studies or methods were excluded from our review.

Using the same criteria and search terms, we also reviewed the NIH CRISP database (Computer Retrieval of Information on Scientific Projects) of projects funded over the same five-year period. Projects were limited to R series (independent research awards), F series (predissertation research awards), and K series (career development awards) grants. Data were taken from only the project descriptions provided by the applicant and contained in the database.

Using typologies employed in other fields of inquiry ( 5 – 7 , 9 ), we next assessed the use of mixed methods in each study to determine the study aims, rationale, structure, function, and process. Study aims referred to the objectives of the overall project that included both quantitative and qualitative studies or methods. The rationale for using mixed methods included conceptual reasons, such as exploration and confirmation ( 5 ), breadth and depth of understanding ( 19 ), and inductive and deductive theoretical drive ( 20 ). Pragmatic reasons for using mixed methods, such as addressing the weaknesses of one method by use of the other, and suitability to address research questions were also examined. Assessment of the structure of the research design was based on Morse's ( 7 ) taxonomy, which gives emphasis to timing (for example, using methods in sequence [represented by a → symbol] versus using them simultaneously [represented by a + symbol]) and to weighting (for example, primary method [represented by capital letters such as QUAN] versus secondary method [represented in lowercase letters such as qual]).

Assessment of the function of mixed methods was based on whether the two methods were being used to answer the same question or to answer related questions and whether they were used to achieve convergence, complementarity, expansion, development, or sampling ( 9 ). Finally, the process or strategies for combining qualitative and quantitative methods were assessed with the typology proposed by Cresswell and Plano Clark ( 5 ): merging or converging the two methods by actually bringing them together in the analysis or interpretation phase, connecting the two methods by having one build upon the results obtained by the other, or embedding one data set within the other so that one type of method provides a supportive role for the other method.

Our search identified 50 articles and 67 NIH-funded research projects published or funded between 2005 and 2009 that met our criteria for analysis. Seven of the NIH projects were excluded from further review because of missing data on the use of mixed methods. Three of the publications were based on one of the NIH-funded projects, and two other publications were based on one funded project each. Any redundant aims or strategies for combining qualitative and quantitative methods identified in linked publications and projects were counted only once in our analysis.

A list of the 26 journals in which the articles were published and the journals' impact factors (IFs) is presented in Table 1 . One-fifth of the articles were published in Psychiatric Services . The 2008 IFs of the journals for which information was available ranged from .74 ( Psychiatric Rehabilitation Journal ) to 4.84 ( Journal of the American Academy of Child and Adolescent Psychiatry ). Twenty-one of the 50 articles (42%) had an IF of 2.0 or greater. Of the funded grants, three were predissertation research grants (F31s), 28 were career development awards (K01, K08, K23, K24, and K99), and 29 were independent research awards (R01, R03, R18, R21, R24, and R34).

Table 2 presents the year of publication for the 50 articles and the start date of the 60 funded projects. Sixteen of the projects funded during this period had a start date before 2005. The smaller numbers of publications and of projects in 2009 reflect the shorter period of observation (nine months) for that year. There was an exponential increase in the number of publications between 2005 and 2008, and the number of grants from 2005 to 2009 was more than twice that of the previous five-year period (2000–2004).

Table 3 summarizes for comparison the use of mixed-methods designs on the basis of study aims. Our analyses revealed the use of mixed methods to accomplish five distinct types of study aims and three categories of rationale. We further identified seven structural arrangements, five uses or functions of mixed methods, and three ways of linking quantitative and qualitative data together. Some papers and projects included more than one objective, structure, or function; hence the raw numbers may occasionally sum to more than the total number of studies examined. Twelve of the 50 articles presented qualitative data only but were part of larger studies that included the use of quantitative measures. Further, we identified four commonly used designs, with each design associated with a specific aim or set of aims ( Figure 1 ).

As shown in Table 3 , the largest number of publications and projects (41 of 110, 37%) used mixed methods in observational or quasi-experimental studies of existing services. Almost one-quarter (24%) used mixed methods to study the implementation and dissemination of evidence-based practices. Mixed methods were also used to develop evidence-based practices, treatment, and interventions (17%); to conduct randomized controlled trials of interventions (14%); or to assess the needs of populations for mental health services (14%). Six studies had more than one aim (for example, two studies conducted a needs assessment before developing new interventions, and two studies examined implementation of an evidence-based practice within the context of a randomized controlled trial examining the practice's effectiveness.

Mixed-methods rationale

Forty-one of the 60 project abstracts (68%) and 25 of the 50 published articles (50%) did not provide an explicit rationale for the use of mixed methods; consequently, the rationale was inferred from statements found in project objectives. Of the 25 published articles that did provide an explicit rationale, only 11 provided one or more citations to justify use of mixed methods. The most common reason (93% of all articles and projects) for using mixed methods was based on the specific objectives of the study (for example, qualitative methods were needed for exploration or depth of understanding or quantitative methods were needed to test hypotheses). In other instances, use of mixed methods was dictated by the nature of the data; studies that included a focus on variables related to values and beliefs, the process of service delivery, or the context in which services are delivered relied on qualitative methods to describe and examine these phenomena. In 9% of articles and projects, investigators specifically indicated that both methods were used so that the strengths of one method could offset the weaknesses of the other ( Table 3 ).

Mixed-methods structure

The majority (58%) of the publications and projects used the methods in sequence, with qualitative methods more often preceding quantitative methods. Quantitative methods were the primary or dominant method in 74% of the publications and projects reviewed, and in 16 studies, qualitative and quantitative methods were given equal weight. In seven of the published studies, qualitative analyses were conducted on one or two open-ended questions attached to a survey, and 17 of the 50 published studies (34%) provided no references justifying their procedures for qualitative data collection or analysis. Only one published study ( 21 ) provided a figure that illustrated the timing and weighting of qualitative and quantitative data collection and analysis, and none used terms like QUAN and qual to describe this structure.

In studies that aimed to assess needs for mental health services, examine existing services, or develop new services or adapt existing services to new populations, sequential designs were used two to four times more frequently than simultaneous designs. The latter type of design was more commonly used in randomized controlled trials and in implementation studies.

Mixed-methods functions

Our review of the publications and projects revealed five distinct functions of mixing methods ( Table 3 ). The first function was convergence, in which qualitative and quantitative methods were used sequentially or simultaneously to answer the same question, either through triangulation (that is, the simultaneous use of one type of data to validate or confirm conclusions reached from analysis of the other type of data) or transformation (that is, the sequential quantification of qualitative data or use of qualitative techniques to transform quantitative data). For instance, Griswold and colleagues ( 22 ) triangulated quantitative trends in functional and health outcomes of psychiatric emergency department patients with qualitative findings of perceived benefits of care management and the value of integrated medical and mental health care to determine whether both types of data provided support for the effectiveness of a care management intervention (QUAN + QUAL). Using the technique of concept mapping ( 23 ), Aarons and colleagues ( 24 ) collected qualitative data on factors likely to have an impact on implementation of evidence-based practices in public-sector mental health settings. These data were then entered in a software program that uses multidimensional scaling and hierarchical cluster analysis to generate a visual display of statement clusters (QUAL → quan).

A second function of integrating quantitative and qualitative methods was complementarity, in which each method was used to answer related questions for the purpose of evaluation or elaboration. This function was evident in a majority (65%) of the published studies and projects examined. In evaluative designs, quantitative data were used to evaluate outcomes, whereas qualitative data were used to evaluate process. For instance, Bearsley-Smith and colleagues ( 25 ) described the use of quantitative methods to investigate the impact on clinical care of implementing interpersonal psychotherapy for adolescents within a rural mental health service and the use of qualitative methods to record the process and challenges (that is, feasibility, acceptability, and sustainability) associated with implementation and evaluation (QUAN + qual). In elaborative designs, qualitative methods were used to provide depth of understanding and quantitative methods were used to provide breadth of understanding. For instance, in a longitudinal study of mental health consumer-run organizations, Janzen and colleagues ( 26 ) used a quantitative tracking log for breadth of information about system-level activities and outcomes and key informant interviews and focus groups for greater insight into the impacts of these activities (QUAL + quan).

A third function of integrating qualitative and quantitative methods was expansion, in which one method was used in sequence to answer questions raised by the other method. This function was evident in 24% of the published studies and projects examined. In each instance, qualitative data were used to explain findings from the analyses of quantitative data. Brunette and colleagues ( 27 ) interviewed key informants and conducted ethnographic observations of implementation efforts to understand why some agencies adhered to established principles for integrated dual disorders treatment and others did not (QUAN + qual).

A fourth function of mixed methods was development, in which qualitative methods were used sequentially to identify form and content of items to be used in a quantitative study (for example, survey questions), to create a conceptual framework for generating hypotheses to be tested by using quantitative methods, or to develop new interventions or adapt existing interventions to new populations (qual → QUAN). This function was used in 34% of the published studies and projects. Blasinsky and colleagues ( 28 ) used qualitative findings from site visits to develop quantitative rating scales to construct predictors of outcomes and sustainability of a collaborative care intervention for older adults who had major depressive disorder or dysthymia. Green and colleagues ( 29 ) used qualitative data to generate a theoretical model of how relationships with clinics and clinicians' approach affect quality of life and recovery from serious mental illness and then tested the model using questionnaire data and health-plan and interview-based data in a covariance structure model. Several of the research projects funded through the R34 mechanism (for example, MH074509-01, Kilbourne, principal investigator [PI]; MH078583-01, Druss, PI; and MH073087-01, Lewis-Fernandez, PI) used qualitative data obtained from focus groups of consumers and providers to develop or adapt interventions for clients with specific conditions (for example, bipolar disorder, chronic medical conditions, and depressive disorders) (qual − QUAN).

The final function of mixed methods was sampling, the sequential use of one method to identify a sample of participants for research that uses the other method. This technique was used in only 7% of all studies. One form of sampling was the sequential use of quantitative data to identify potential participants for a qualitative study (quan − QUAL). For instance, Aarons and Palinkas ( 30 ) purposefully sampled candidates for qualitative interviews who had the most positive or most negative views of an evidence-based practice on the basis of a Web-based quantitative survey. The other form of sampling used qualitative data to identify samples of participants for quantitative analysis (qual − QUAN). Woltmann and colleagues ( 31 ) created categories of low, medium, and high staff turnover on the basis of staff perceptions of relevance of turnover obtained from qualitative interviews and then quantitatively examined the relationship between these turnover categories and implementation outcomes (qual + QUAN).

Only six of the published studies and none of the project abstracts explicitly referred to the function of mixed methods by using terms such as triangulation (four published studies) or complementarity (two published studies). As expected, the development function was used in a majority (84%) of studies that aimed to develop new practices or adapt existing practices to new populations. A majority of observational and quasi-experimental studies of existing services (71%), randomized controlled trials (67%), implementation studies (65%), and needs assessment studies (60%) utilized mixed methods for the purposes of answering related questions in complementary fashion. The use of one set of methods to explain the results of a study using another set of methods appears to have been limited to implementation studies (46%), randomized controlled trial evaluations (40%), and studies of existing services (20%).

Process of mixing methods

The final characteristic of mixed-methods designs that we examined was the process of mixing the quantitative and qualitative methods. The largest percentage (47%) of articles and projects sought to connect the data sets ( Table 3 ). This occurs when the analysis of one data set leads to (and thereby connects to) the need for the other data set, such as when quantitative results lead to the subsequent collection and analysis of qualitative data (that is, expansion) or when qualitative results are used to build to the subsequent collection and analysis of quantitative data, (for example, development) ( 5 ). For instance, Frueh and colleagues ( 32 ) conducted focus groups to obtain information on the target population, their providers, and state-funded mental health systems that would enable the researchers to further adapt and improve a cognitive-behavioral therapy-based intervention for treatment of posttraumatic stress disorder before implementing it (qual → QUAN). This type of mixing was found in almost all of the studies with aims to develop new practices or adapt existing practices to new populations; it was also more likely to be found in needs assessment and studies of existing services than in randomized controlled trials or implementation studies.

Over one-third (37%) of the studies merged the knowledge gained from the quantitative and qualitative data, either during the interpretation phase when two sets of results that had been analyzed separately were brought together or during the analysis phase when one type of data was transformed into the other type by consolidating the data into new variables ( 5 ). This type of mixing was found in slightly less than half of the needs assessment, observational, and implementation studies. For instance, Lucksted and colleagues ( 33 ) reported that a qualitative analysis of responses to an open-ended postintervention question supported the quantitative findings of the benefits of a relapse prevention and wellness program (QUAN + qual).

The embedding of small qualitative or qualitative-quantitative studies within larger quantitative studies was observed in 35% of the published studies and projects reviewed and described as “nested designs” in six of the studies. This type of mixing was more commonly found in randomized controlled trials and in implementation studies, where qualitative studies of treatment or implementation process or context were embedded within larger quantitative studies of treatment or implementation outcome. For instance, to better understand the essential components of the patient-provider relationship in a public health setting, Sajatovic and colleagues ( 34 ) conducted a qualitative investigation of patients' attitudes toward a collaborative care model and how individuals with bipolar disorder perceive treatment adherence within the context of a randomized controlled trial evaluating a collaborative practice model (QUAN + qual).

In 20% of published studies, more than one process was evident. For instance, Proctor and colleagues ( 35 ) connected the data by generating frequencies and rankings of qualitative data on perceptions of competing psychosocial problems collected from a community sample of 49 clients with a history of depression. These data were then merged with quantitative measures of depression status obtained through administration of the Patient Health Questionnaire-9 to explore the relationship of depression severity to problem categories and ranks.

The results of our analysis indicate that there has been substantial progress in using mixed-methods designs in mental health services research in response to efforts by NIMH ( 2 , 3 ) and other funding agencies to promote their use. Evidence for this progress is found in the increasing number of research projects that use mixed methods. The number of projects with mixed-methods designs funded over the five-year study period was more than twice the number that began in the previous five-year period (2000–2004). Furthermore, a majority (52%) of these funded projects were predissertation or career development awards used by junior and midlevel investigators to acquire expertise in mixed-methods research.

We also observed a notable increase in the number of studies based on mixed-methods designs published each year during this five-year period. The number of published mental health services research studies with mixed-methods designs increased by 67% between 2005 and 2006, by 80% between 2006 and 2007, and by 155% between 2007 and 2008. Furthermore, 21 of the 50 published studies (42%) that we reviewed appeared in journals with 2008 IFs of 2.0 or higher, including ten articles published in Psychiatric Services; four articles appeared in a journal with an IF of 4.0 or higher. In contrast, McKibbon and Gadd ( 18 ) reported that only 11 of 37 (30%) mixed-methods studies of health services appeared in a journal with an IF of 2.0 or higher in the year 2000.

Despite this progress, however, our review also suggests that there is room for improvement in use of mixed-methods designs. Most studies did not make explicit or provide support for the reasons for choosing a mixed-methods design; rather, we were forced to infer the rationale based on statements explaining what the methods were used for. Researchers may have felt that such explicit statements were as unnecessary as statements explaining the rationale for using certain quantitative methods, such as analysis of variance or survival analysis. However, the absence of an explicit rationale may also reflect a lack of understanding or appreciation of mixed-methods designs or a decision to use them without necessarily integrating or “mixing” them ( 5 , 6 ).

Most studies failed to provide explicit descriptions of the design structure or function that used terminology found in the mixed-methods literature; use of such terminology is consistent with the general standards for high-quality mixed-methods research recommended by Cresswell and Plano Clark ( 5 ). Further, three-fourths of the 50 published studies reviewed assigned priority to the use of quantitative methods, seven of the studies performed qualitative analyses of one or two open-ended questions attached to a survey, and 17 of the studies provided no references justifying their procedures for qualitative data collection or analysis. This may reflect an underappreciation of qualitative methods, as Robins and colleagues ( 1 ) have argued, or it may reflect a greater need for quantitative methods at the present time.

Although it was beyond the scope of this review to determine whether each study used mixed methods in effective ways, we note that each study was subjected to rigorous peer review before being published or funded, and each was judged by this process to make a valuable contribution to the field of mental health services research. These studies also provide evidence of meaningful and sensible variations in mixed-methods approaches to achieving various kinds of study aims and offer some guidance for integrating quantitative and qualitative methods in mental health services research. For instance, the choice of a mixed-methods design appears to be dictated by the nature of the questions being asked by mental health services researchers. Qualitative methods were used to explore a phenomenon when there was little or no previous research or to examine that phenomenon in depth, whereas quantitative methods were used to confirm hypotheses or examine the generalizability of the phenomenon and its associated predictors.

A majority of studies aiming to develop new practices or adapt existing practices to new populations had the same structure (beginning with a small qualitative study before developing or adapting the practice that was to be evaluated by using quantitative methods, which was found in 84% of the studies and projects) and the same process (connecting the findings of one set of methods with those of another set, which was found in 90% of the studies and projects). These studies reflect a growing awareness of the need to incorporate the preferences and perspectives of both service consumers and providers to ensure that new practices will be acceptable as well as feasible ( 32 , 36 – 39 ).

Studies of existing services also tended to be sequential in structure, with qualitative methods used to elaborate or explain the findings of quantitative studies. In the majority of these studies, the process of mixing methods involved either merging two sets of data to achieve convergence or connecting them to achieve expansion ( 5 ). A similar pattern was observed in studies that aimed to explore issues related to the needs for mental health services or provide more depth to our understanding of those needs. Such studies also appeared more likely to transform or “quantitize” qualitative data ( 24 , 35 ).

Randomized controlled trials and studies of implementation also shared similar patterns in use of mixed methods, including simultaneous use of both methods to achieve complementarity by embedding a qualitative or qualitative-quantitative study within a larger quantitative study, such as a randomized controlled trial. In the randomized controlled trials, qualitative methods were usually used to evaluate the process of providing the practice or intervention, whereas quantitative methods were used to evaluate the outcomes ( 25 , 40 ). In implementation research studies, qualitative methods were used to explore or provide depth to understanding barriers and facilitators of intervention implementation, whereas quantitative methods were used to confirm hypotheses and provide breadth to understanding by assessing the generalizability of findings ( 41 , 42 ).

The choice of mixed-methods designs was also dictated by how the individual questions being addressed by each method were related to one another. Studies that used different types of data to answer the same question reflected the function of convergence in a simultaneous structure, where data were merged for the purpose of triangulation, or a sequential structure, where qualitative data were transformed into quantitative data. Studies that used different types of data to answer related questions reflected the function of complementarity, in which quantitative methods were used to measure outcomes, describe content (for example, fidelity of services used and the nature of the mental health problem), and provide breadth (generalizability) of understanding, whereas qualitative methods were used to evaluate the process of service delivery ( 43 – 45 ), describe context (for example, setting) ( 26 , 34 , 46 ), describe consumer values or attitudes ( 35 , 42 , 47 ), and provide depth (meaning) of understanding ( 28 , 48 ) in a simultaneous structure and embedded data process. Expansion, development, and sampling were also used to provide answers to related questions that could not be answered by one method alone, usually in a sequential structure in which data sets were merged or connected together ( 24 , 30 , 37 ).

Finally, the choice of design appears to be based on the strengths of one method relative to the weaknesses of the other. For instance, expansion was used to explain findings based on quantitative data with qualitative data because explanation was not possible with the quantitative methods alone ( 25 , 27 , 40 ). In convergence, both sets of methods were used to confirm or validate one another, especially in instances where limited samples precluded testing of hypotheses with sufficient statistical power ( 30 , 49 ) and where limitations to qualitative data collection raised concerns about objectivity and transferability of results. In studies developing new methods, conceptual models, and interventions, qualitative methods also served to enhance quantitative analysis by laying the groundwork essential for more valid measurement and theory and more effective, usable, and sustainable interventions ( 37 ). Sampling also worked to enhance validity by using qualitative methods to enhance quantitative methods by developing targeted comparisons or by using quantitative methods to enhance qualitative methods by establishing criteria for purposeful sampling ( 36 ).

In summary, the choice of a mixed-methods design appears to be associated with three considerations: the nature of the question being asked (inductive-exploratory or deductive-confirmatory), how the questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Caution should be exercised in interpreting these findings given limitations in our study design and analysis. Despite our efforts to be comprehensive in the search process and to select studies and projects on the basis of criteria with face validity, we undoubtedly excluded several articles or projects that used mixed methods. For example, we may have excluded mixed-methods projects listed in the CRISP database that did not specify use of qualitative or mixed methods in the abstracts. We may have also excluded published articles with qualitative data that were part of larger, primarily quantitative studies if the articles did not reference the larger studies, or we may have excluded articles not listed in PubMed Central. In the absence of explicit information, we were often forced to infer the structure, rationale, and function of the design based on statements contained in the available material. Similarly, the CRISP abstracts describe only what the investigators proposed to do with mixed methods and do not indicate what was actually done. Our use of existing typologies of structure, function, and process were intended to serve as a starting point in our analysis rather than an attempt to “pigeon-hole” each study into a specific typology. Our assessment of the progress made in the application of mixed-methods designs in response to calls for their use by funding agencies did not include indicators of whether these efforts had produced more useful, incisive, or insightful knowledge for the purpose of addressing mental health services questions and problems. Such an assessment would require comparisons with the products of studies based on monomethod designs, which was beyond the scope of this study.

Finally, it should be noted that the typology of mixed-methods use does not represent a set of standards for using mixed methods per se but is an important first step toward the development of such standards. Typologies by themselves do not explain why a particular method should be used and how to use a method appropriately. However, as Teddlie and Tashakkori ( 6 ) observed, there are five reasons or benefits to developing such a typology: typologies help to provide the field with an organizational structure, they provide examples of research designs that are clearly distinct from either qualitative or quantitative research designs, they help to establish a common language for the field, they help researchers decide how to proceed when designing their studies, and they are useful as a pedagogical tool. A consensus conference or workshop bringing together experts in mixed methods and mental health services research to evaluate the empirically generated typology found in current patterns of mixed-methods use would appear to be the next logical step in developing a set of standards. Such standards would also be required to adhere to the epistemological foundations of each method when used separately (for example, whether appropriate considerations are made to ensure the generalizability of quantitative results or theoretical saturation of qualitative data and whether each method is appropriately matched to the inductive or deductive theoretical drive of the study) and when combined (for example, whether the knowledge gained when using the two methods together is more insightful and of greater value than the knowledge gained when using them separately).

Conclusions

Despite the limitations described above, the findings suggest an increasing use of mixed-methods designs to address changing priorities in mental health services research and a consensus as to how such methods should be applied. The lack of explicit statements explaining the rationale for using mixed methods and the evident priority assigned to quantitative methods suggest that there is room for improvement. However, these studies appear to utilize a common set of designs and provide guidance for using mixed methods, with varying approaches based on the nature of the question being asked (exploratory or confirmatory), how questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Acknowledgments and disclosures

This study was funded through NIMH grant P50-MH50313-07.

The authors report no competing interests.

1 Robins CS , Ware NC , dosReis S , et al. : Dialogues on mixed methods and mental health services research: anticipating challenges, building solutions . Psychiatric Services 59:727–731, 2008 Link ,  Google Scholar

2 Hopper K : Qualitative and quantitative research: two cultures . Psychiatric Services 59:711, 2008 Link ,  Google Scholar

3 Hoagwood K , Jensen PS , Petti T , et al. : Outcomes of mental health care for children and adolescents: I. a comprehensive conceptual model . Journal of the American Academy of Child and Adolescent Psychiatry 35:1055–1063, 1996 Crossref , Medline ,  Google Scholar

4 Hohmann AA : A contextual model for clinical mental health effectiveness research . Mental Health Services Research 1:83–91, 1999 Crossref ,  Google Scholar

5 Cresswell JW , Plano Clark VL : Designing and Conducting Mixed Method Research . Thousand Oaks, Calif, Sage, 2007 Google Scholar

6 Teddlie C , Tashakkori A : Major issues and controversies in the use of mixed methods in the social and behavioral sciences; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by Tashakkori ATeddlie C Thousand Oaks, Calif, Sage, 2003 Google Scholar

7 Morse JM : Approaches to qualitative-quantitative methodological triangulation . Nursing Research 40:120–123, 1991 Crossref , Medline ,  Google Scholar

8 Sandelowski M : Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed method studies . Research in Nursing and Health 23:246–255, 2000 Crossref , Medline ,  Google Scholar

9 Greene JC , Caracelli VJ , Graham WF : Toward a conceptual framework for mixed method evaluation designs . Educational Evaluation and Policy Analysis 11:255–274, 1989 Crossref ,  Google Scholar

10 Patton MQ : Qualitative Evaluation and Research Methods , 3rd ed. Newbury Park, Calif, Sage, 2002 Google Scholar

11 Morgan DL : Practical strategies for combining qualitative and quantitative methods: applications to health research . Qualitative Health Research 8:262–276, 1998 Crossref ,  Google Scholar

12 Steckler A , McLeroy KR , Goodman RM , et al. : Toward integrating qualitative and quantitative methods: an introduction . Health Education Quarterly 19:1–8, 1992 Crossref , Medline ,  Google Scholar

13 Cresswell JW , Fetters MD , Ivankova NV : Designing a mixed methods study in primary care . Annals of Family Medicine 2:7–12, 2004 Crossref , Medline ,  Google Scholar

14 Cresswell JW : Mixed method research: introduction and application; in Handbook of Educational Policy . Edited by Cizek GJ San Diego, Academic Press, 1999 Crossref ,  Google Scholar

15 Waszak C , Sines MC : Mixed methods in psychological research; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by Tashakkori ATeddlie C Thousand Oaks, Calif, Sage 2003 Google Scholar

16 Miles MB , Huberman AM : Qualitative Data Analysis: An Expanded Sourcebook , 2nd ed. Thousand Oaks, Calif, Sage, 1994 Google Scholar

17 Tashakkori A , Teddlie C : Mixed Methodology: Combining the Qualitative and Quantitative Approaches . Thousand Oaks, Calif, Sage, 1998 Google Scholar

10.1186/1472-6947-4-11 Crossref , Medline ,  Google Scholar

19 Miller WL , Crabtree BF Clinical research; in Handbook of Qualitative Research , 2nd ed. Edited by Denzin NKLincoln YS Thousand Oaks, Calif, Sage, 2001 Google Scholar

20 Morse JM Principles of mixed methods and multimethod research design; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by , Tashakkori A , Teddlie C Thousand Oaks, Calif, Sage, 2003 Google Scholar

10.1186/1472-6963-8-156 Crossref , Medline ,  Google Scholar

22 Griswold KS , Zayas LE , Pastore PA , et al. : Primary care after psychiatric crisis: a qualitative analysis . Annals of Family Medicine 6:38–43, 2008 Crossref , Medline ,  Google Scholar

23 Trochim WM : An introduction to concept mapping for planning and evaluation . Evaluation and Program Planning 12:1–16, 1989 Crossref ,  Google Scholar

24 Aarons GA , Wells R , Zagursky K , et al. : Advancing a conceptual model of evidence-based practice implementation in child welfare . American Journal of Public Health 99:2087–2095, 2009 Crossref , Medline ,  Google Scholar

10.1186/1471-244X-7-53 Crossref ,  Google Scholar

26 Janzen R , Nelson G , Hausfather N , et al. : Capturing system level activities and impacts of mental health consumer-run organizations . American Journal of Community Health 39:287–299, 2007 Google Scholar

27 Brunette MF , Asher D , Whitley R , et al. : Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers . Psychiatric Services 59:989–995, 2008 Link ,  Google Scholar

28 Blasinsky M , Goldman HH , Unützer J : Project IMPACT: a report on barriers and facilitators to sustainability . Administration and Policy in Mental Health and Mental Health Services 33:718–729, 2006 Crossref , Medline ,  Google Scholar

29 Green CA , Polen MR , Janoff SL , et al. : Understanding how clinician-patient relationships and relational continuity of care affect recovery from serious mental illness: STARS Study results . Psychiatric Rehabilitation Journal 32:9–22, 2008 Crossref , Medline ,  Google Scholar

30 Aarons GA , Palinkas LA : Implementation of evidence-based practice in child welfare: service provider perspectives . Administration and Policy in Mental Health and Mental Health Services Research 34:411–419, 2007 Crossref , Medline ,  Google Scholar

31 Woltmann EM , Whitley R , McHugo GJ , et al. : The role of staff turnover in the implementation of evidence-based practices in health care . Psychiatric Services 59:732–737, 2008 Link ,  Google Scholar

32 Frueh BC , Cusack KJ , Grubaugh AL , et al. : Clinicians' perspectives on cognitive-behavioral treatment for PTSD among persons with severe mental illness . Psychiatric Services 57:1027–1031, 2006 Link ,  Google Scholar

33 Lucksted A , McNulty K , Brayboy L , et al. : Initial evaluation of the Peer-to-Peer Program . Psychiatric Services 60:250–253, 2009 Link ,  Google Scholar

34 Sajatovic M , Davies M , Bauer MS , et al. : Attitudes regarding the collaborative practice model and treatment adherence among individuals with bipolar disorder . Comprehensive Psychiatry 46:272–277, 2005 Crossref , Medline ,  Google Scholar

35 Proctor EK , Hascke L , Morrow-Howell N , et al. : Perceptions about competing psychosocial problems and treatment priorities among older adults with depression . Psychiatric Services 59:670–675, 2008 Link ,  Google Scholar

36 Woolfson R , Woolfson L , Mooney L , et al. : Young people's views of mental health education in secondary schools: a Scottish study . Child: Care, Health and Development 35:790–798, 2009 Crossref , Medline ,  Google Scholar

37 Proctor EK , Knudsen KL , Fedoravicius N , et al. : Implementation of evidence-based practice in community behavioral health: agency director perspectives . Administration and Policy in Mental Health and Mental Health Services Research 34:479–488, 2007 Crossref , Medline ,  Google Scholar

38 Institute of Medicine : Crossing the Quality Chasm: A New Health System for the 21st Century . Washington, DC, National Academies Press, 2000 Google Scholar

39 Mental Health : A Report of the Surgeon General . Rockville, Md, US Department of Health and Human Services, US Public Health Service, 1999 Google Scholar

10.1186/1472-6963-8-274 Crossref , Medline ,  Google Scholar

41 Gioia D , Dziadosz G : Adoption of evidence-based practices in community mental health: a mixed method study of practitioner experience . Community Mental Health Journal 44:347–357, 2008 Crossref , Medline ,  Google Scholar

42 Ostler T , Haight W , Black J , et al. : Case series: mental health needs and perspectives of rural children raised by parents who abuse methamphetamine . Journal of the American Academy of Child and Adolescent Psychiatry 46:500–507, 2007 Crossref , Medline ,  Google Scholar

10.1186/1748-3-14 Crossref , Medline ,  Google Scholar

10.1186/1472-6963-7-117 Crossref , Medline ,  Google Scholar

45 Young AT , Green CA , Estroff SE : New endeavors, risk taking, and personal growth in the recovery process: findings from the STARS study . Psychiatric Services 59:1430–1436, 2008 Link ,  Google Scholar

46 Sharkey S , MacIver S , Cameron D , et al. : An exploration of factors affecting the implementation of a randomized controlled trial of a transitional discharge model for people with serious mental illness . Journal of Psychiatric and Mental Health Nursing 12:51–56, 2005 Crossref , Medline ,  Google Scholar

47 Lee BR , Munson MR , Ware NR , et al. : Experiences of and attitudes toward mental health services among older youths in foster care . Psychiatric Services 57:487–492, 2006 Link ,  Google Scholar

48 Becker D , Whitley R , Bailey EL , et al. : Long-term employment trajectories among participants with severe mental illness in supported employment . Psychiatric Services 58:922–928, 2007 Link ,  Google Scholar

49 Wetherell JL , Ayers CR , Sorrell JT , et al. : Modular psychotherapy for anxiety in older primary care patients . American Journal of Geriatric Psychiatry 17:483–492, 2009 Crossref , Medline ,  Google Scholar

Figures and Tables

Figure 1 Common mixed-methods designs used in mental health services research

Table 1 Journals in which the 50 articles reviewed were published, with number published and 2008 impact factor

Table 2 Year of publication or of project initiation of articles and projects reviewed

Table 3 Characteristics of 50 published studies and 60 funded projects that used mixed-methods designs, by study aims

• A multi- and mixed-method adaptation study of a patient-centered perioperative mental health intervention bundle 27 October 2023 | BMC Health Services Research, Vol. 23, No. 1
• Physician Assistant Student Attitudes About People With Serious Mental Illness 21 November 2023 | Journal of Physician Assistant Education, Vol. 66
• Educators’ Perspectives on Training Mechanisms That Facilitate Evidence-Based Practice Use for Autistic Students in General Education Settings: A Mixed-Methods Analysis 2 July 2023 | Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, Vol. 46, No. 4
• Community-led identification of mental health support, challenges, and needs among Ethiopian immigrants to the U.S.: opportunities for partnership with faith communities 15 January 2024 | Mental Health, Religion & Culture, Vol. 26, No. 9
• Social network and mental health of Chinese immigrants in affordable senior housing during the COVID-19 pandemic: a mixed-methods study 22 May 2023 | Aging & Mental Health, Vol. 27, No. 10
• Incazelo nomlando oqukethwe emagameni aqanjwe abesifazane abashade ngaphambi konyaka we-1990 esigodini sakaGcaliphiwe eMaphephetheni 22 December 2023 | South African Journal of African Languages, Vol. 43, No. 3
• Implementation Science and Practice-Oriented Research: Convergence and Complementarity 30 August 2023 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 27
• Adapting to Unprecedented Times: Community Clinician Modifications to Parent–Child Interaction Therapy During COVID-19 11 August 2023 | Evidence-Based Practice in Child and Adolescent Mental Health, Vol. 8, No. 3
• Evaluating the validity of depression-related stigma measurement among diabetes and hypertension patients receiving depression care in Malawi: A mixed-methods analysis 17 May 2023 | PLOS Global Public Health, Vol. 3, No. 5
• Potential advantages of combining randomized controlled trials with qualitative research in mood and anxiety disorders - A systematic review Journal of Affective Disorders, Vol. 325
• Mental Health Therapist Perspectives on the Role of Executive Functioning in Children’s Mental Health Services 10 January 2022 | Evidence-Based Practice in Child and Adolescent Mental Health, Vol. 8, No. 1
• Therapist and supervisor perspectives about two train-the-trainer implementation strategies in schools: A qualitative study 3 August 2023 | Implementation Research and Practice, Vol. 4
• Efficacy of Therapist Guided Internet Based Cognitive Behavioural Therapy for Depression: A Qualitative Exploration of Therapists and Clients Experiences 31 December 2022 | Journal of Professional & Applied Psychology, Vol. 3, No. 4
• Prevalence of Research Designs and Efforts at Integration in Mixed Methods Research: A Systematic Review 31 December 2022 | International Journal of Multiple Research Approaches, Vol. 14, No. 3
• The measurement-based care to opioid treatment programs project (MBC2OTP): a study protocol using rapid assessment procedure informed clinical ethnography 19 August 2022 | Addiction Science & Clinical Practice, Vol. 17, No. 1
• Barbershops as a setting for supporting men's mental health during the COVID-19 pandemic: a qualitative study from the UK 27 June 2022 | BJPsych Open, Vol. 8, No. 4
• A mixed methods study of provider factors in buprenorphine treatment retention International Journal of Drug Policy, Vol. 105
• Evaluation of a systems-level technical assistance program to support youth with complex behavioral health needs Evaluation and Program Planning, Vol. 92
• Barriers to students opting-in to universities notifying emergency contacts when serious mental health concerns emerge: A UK mixed methods analysis of policy preferences Journal of Affective Disorders Reports, Vol. 7
• Development of an Online Resource for People Bereaved by Suicide: A Mixed-Method User-Centered Study Protocol 21 December 2021 | Frontiers in Psychiatry, Vol. 12
• Protocol for a hybrid type 2 cluster randomized trial of trauma-focused cognitive behavioral therapy and a pragmatic individual-level implementation strategy 7 January 2021 | Implementation Science, Vol. 16, No. 1
• Understanding adaptations in the Veteran Health Administration’s Transitions Nurse Program: refining methodology and pragmatic implications for scale-up 13 July 2021 | Implementation Science, Vol. 16, No. 1
• Defining effective care coordination for mental health referrals of refugee populations in the United States 19 November 2018 | Ethnicity & Health, Vol. 26, No. 5
• A Mixed-method Evaluation of the Behavioral Health Integration and Complex Care Initiative Using the Consolidated Framework for Implementation Research 13 May 2021 | Medical Care, Vol. 59, No. 7
• Parent Training for Youth with Autism Served in Community Settings: A Mixed-Methods Investigation Within a Community Mental Health System 2 September 2020 | Journal of Autism and Developmental Disorders, Vol. 51, No. 6
• Client, clinician, and administrator factors associated with the successful acceptance of a telehealth comprehensive recovery service: A mixed methods study Psychiatry Research, Vol. 300
• “Don’t … Break Down on Tuesday Because the Mental Health Services are Only in Town on Thursday”: A Qualitative Study of Service Provision Related Barriers to, and Facilitators of Farmers’ Mental Health Help-Seeking 15 September 2020 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 48, No. 3
• Social media and community-oriented policing: examining the organizational image construction of municipal police on Twitter and Facebook 9 November 2020 | Police Practice and Research, Vol. 22, No. 1
• The ‘shift reflection’ model of group reflective practice: a pilot study in an acute mental health setting Mental Health Practice, Vol. 24, No. 1
• Challenges Experienced by Behavioral Health Organizations in New York Resulting from COVID-19: A Qualitative Analysis 23 October 2020 | Community Mental Health Journal, Vol. 57, No. 1
• Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives 24 February 2021 | DIGITAL HEALTH, Vol. 7
• An eHealth Intervention for Promoting COVID-19 Knowledge and Protective Behaviors and Reducing Pandemic Distress Among Sexual and Gender Minorities: Protocol for a Randomized Controlled Trial (#SafeHandsSafeHearts) 10 December 2021 | JMIR Research Protocols, Vol. 10, No. 12
• Promotion of mental health in young adults via mobile phone app: study protocol of the ECoWeB (emotional competence for well-being in Young adults) cohort multiple randomised trials 22 September 2020 | BMC Psychiatry, Vol. 20, No. 1
• Adaption and pilot implementation of an autism executive functioning intervention in children’s mental health services: a mixed-methods study protocol 27 April 2020 | Pilot and Feasibility Studies, Vol. 6, No. 1
• Improving the implementation and sustainment of evidence-based practices in community mental health organizations: a study protocol for a matched-pair cluster randomized pilot study of the Collaborative Organizational Approach to Selecting and Tailoring Implementation Strategies (COAST-IS) 25 February 2020 | Implementation Science Communications, Vol. 1, No. 1
• Using mixed methods in health services research: A review of the literature and case study 21 September 2020 | Journal of Health Services Research & Policy, Vol. 4
• Healthcare attendance styles among long-term unemployed people with substance-related and mood disorders Public Health, Vol. 186
• Mixed-Methods-Studien in der Gesundheitsförderung. Ergebnisse eines systematischen Reviews deutschsprachiger Publikationen Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, Vol. 153-154
• Mixed method study of workforce turnover and evidence-based treatment implementation in community behavioral health care settings Child Abuse & Neglect, Vol. 102
• Mixing Beyond Measure: Integrating Methods in a Hybrid Effectiveness–Implementation Study of Operating Room to Intensive Care Unit Handoffs 4 May 2019 | Journal of Mixed Methods Research, Vol. 14, No. 2
• The search for the ejecting chair: a mixed-methods analysis of tool use in a sedentary behavior intervention 25 November 2018 | Translational Behavioral Medicine, Vol. 10, No. 1
• SIPsmartER delivered through rural, local health districts: adoption and implementation outcomes 18 September 2019 | BMC Public Health, Vol. 19, No. 1
• An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance 10 October 2019 | Archives of Public Health, Vol. 77, No. 1
• Five Challenges in the Design and Conduct of IS Trials for HIV Prevention and Treatment JAIDS Journal of Acquired Immune Deficiency Syndromes, Vol. 82, No. 3
• Mental health recovery narratives: their impact on service users and other stakeholder groups Mental Health and Social Inclusion, Vol. 23, No. 4
• A Mixed Methods Study of Organizational Readiness for Change and Leadership During a Training Initiative Within Community Mental Health Clinics 19 June 2019 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 46, No. 5
• Associations Among Job Role, Training Type, and Staff Turnover in a Large-Scale Implementation Initiative 3 January 2019 | The Journal of Behavioral Health Services & Research, Vol. 46, No. 3
• American Journal of Community Psychology
• Internet Interventions, Vol. 18
• Journal of Public Child Welfare, Vol. 13, No. 3
• Method Sequence and Dominance in Mixed Methods Research: A Case Study of the Social Acceptance of Wind Energy Literature 12 April 2019 | International Journal of Qualitative Methods, Vol. 18
• JMIR Research Protocols, Vol. 8, No. 1
• Sundhedsprofessionelles begejstringfor fortællinger fra levet erfaring Tidsskrift for psykisk helsearbeid, Vol. 15, No. 4
• Availability of comprehensive services in permanent supportive housing in Los Angeles 6 October 2017 | Health & Social Care in the Community, Vol. 26, No. 2
• Nursing Outlook, Vol. 66, No. 2
• Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, Vol. 133
• Social Work in Mental Health, Vol. 16, No. 4
• International Journal of Family & Community Medicine, Vol. 2, No. 4
• A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives 7 December 2017 | Health Research Policy and Systems, Vol. 15, No. 1
• Fostering Psychotropic Medication Oversight for Children in Foster Care: A National Examination of States’ Monitoring Mechanisms 10 February 2016 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 44, No. 2
• Beliefs and Behaviors of Pregnant Women with Addictions Awaiting Treatment Initiation 17 November 2016 | Child and Adolescent Social Work Journal, Vol. 34, No. 1
• Psychiatric Quarterly, Vol. 88, No. 3
• Quality & Quantity, Vol. 51, No. 1
• Translational Behavioral Medicine, Vol. 7, No. 3
• Psychology, Health & Medicine, Vol. 22, No. 5
• Use of Mixed Methods Research in Research on Coronary Artery Disease, Diabetes Mellitus, and Hypertension Circulation: Cardiovascular Quality and Outcomes, Vol. 10, No. 1
• Changes in Social Networks and HIV Risk Behaviors Among Homeless Adults Transitioning Into Permanent Supportive Housing 8 July 2016 | Journal of Mixed Methods Research, Vol. 11, No. 1
• Victoria D. Ojeda , Ph.D., M.P.H. ,
• Sarah P. Hiller , M.P.I.A. ,
• Samantha Hurst , Ph.D. ,
• Nev Jones , Ph.D. ,
• Sara McMenamin , Ph.D. ,
• James Burgdorf , Ph.D. ,
• Todd P. Gilmer , Ph.D.
• Mixed-Methods Research in the Discipline of Nursing Advances in Nursing Science, Vol. 39, No. 3
• Impact of Caregiver Factors on Youth Service Utilization of Trauma-Focused Cognitive Behavioral Therapy in a Community Setting 27 February 2016 | Journal of Child and Family Studies, Vol. 25, No. 6
• Measuring Current Drug Use in Female Sex Workers and Their Noncommercial Male Partners in Mexico: Concordance Between Data Collected From Surveys Versus Semi-Structured Interviews 18 December 2015 | Substance Use & Misuse, Vol. 51, No. 1
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 43, No. 4
• Implementation and Outcomes of Forensic Housing First Programs 5 October 2015 | Community Mental Health Journal, Vol. 52, No. 1
• Academic Psychiatry, Vol. 40, No. 4
• Journal of Psychoactive Drugs, Vol. 48, No. 5
• Rural Society, Vol. 25, No. 2
• The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services 9 March 2016 | Implementation Science, Vol. 11, No. 1
• Perceptions of clinicians treating young people with first‐episode psychosis for post‐traumatic stress disorder 27 June 2013 | Early Intervention in Psychiatry, Vol. 9, No. 1
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 42, No. 2
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 42, No. 5
• Journal of Religion and Health, Vol. 54, No. 1
• The Journal of Behavioral Health Services & Research, Vol. 42, No. 4
• BMC Palliative Care, Vol. 14, No. 1
• Implementation Science, Vol. 11, No. 1
• International Journal of Environmental Research and Public Health, Vol. 12, No. 5
• Evidence-Based Programs in “Real World” Settings: Finding the Best Fit
• Ana Stefancic , M.A.
• Marian L. Katz , Ph.D.
• Marisa Sklar , M.S.
• Sam Tsemberis , Ph.D.
• Lawrence A. Palinkas , Ph.D.
• Causality and Causal Inference in Social Work 22 May 2014 | Research on Social Work Practice, Vol. 24, No. 5
• A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions 8 October 2013 | Research on Social Work Practice, Vol. 24, No. 2
• Critical Care Medicine, Vol. 42, No. 4
• Implementation Science, Vol. 9, No. 1
• Implementation Science, Vol. 8, No. 1
• Topics in research Current Opinion in Supportive & Palliative Care, Vol. 6, No. 4
• Mixed Methods for Implementation Research 5 December 2011 | Child Maltreatment, Vol. 17, No. 1
• RO1 Funding for Mixed Methods Research 2 September 2011 | Journal of Mixed Methods Research, Vol. 5, No. 4

An integrative review on methodological considerations in mental health research - design, sampling, data collection procedure and quality assurance

Affiliations.

• 1 1School of Nursing and Midwifery, The University of Newcastle, Callaghan, Australia.
• 2 2Faculty of Health and Medicine, School Nursing and Midwifery, University of Newcastle, Callaghan, Australia.
• 3 3Faculty of Business and Economics, Macquarie University, North Ryde, Australia.
• PMID: 31624592
• PMCID: PMC6785873
• DOI: 10.1186/s13690-019-0363-z

Background: Several typologies and guidelines are available to address the methodological and practical considerations required in mental health research. However, few studies have actually attempted to systematically identify and synthesise these considerations. This paper provides an integrative review that identifies and synthesises the available research evidence on mental health research methodological considerations.

Methods: A search of the published literature was conducted using EMBASE, Medline, PsycINFO, CINAHL, Web of Science, and Scopus. The search was limited to papers published in English for the timeframe 2000-2018. Using pre-defined inclusion and exclusion criteria, three reviewers independently screened the retrieved papers. A data extraction form was used to extract data from the included papers.

Results: Of 27 papers meeting the inclusion criteria, 13 focused on qualitative research, 8 mixed methods and 6 papers focused on quantitative methodology. A total of 14 papers targeted global mental health research, with 2 papers each describing studies in Germany, Sweden and China. The review identified several methodological considerations relating to study design, methods, data collection, and quality assurance. Methodological issues regarding the study design included assembling team members, familiarisation and sharing information on the topic, and seeking the contribution of team members. Methodological considerations to facilitate data collection involved adequate preparation prior to fieldwork, appropriateness and adequacy of the sampling and data collection approach, selection of consumers, the social or cultural context, practical and organisational skills; and ethical and sensitivity issues.

Conclusion: The evidence confirms that studies on methodological considerations in conducting mental health research largely focus on qualitative studies in a transcultural setting, as well as recommendations derived from multi-site surveys. Mental health research should adequately consider the methodological issues around study design, sampling, data collection procedures and quality assurance in order to maintain the quality of data collection.

Keywords: Data collection; Mental health; Methodological approach; Mixed methods; Sampling.

© The Author(s). 2019.

Publication types

Complexity in Mental Health Research: Theory, Method, and Empirical Contributions

Guest edited by Dr Eiko I. Fried and Dr Donald Robinaugh

Mental disorders are dynamic, heterogeneous, and multicausal phenomena. Despite increasingly widespread recognition of this inherent complexity, progress in understanding mental disorders as complex biopsychosocial systems has been limited.

To advance our understanding of the etiology, prevention, and treatment of mental disorders, it is critical that both our theories and our research methods reflect the complex reality of psychopathology. In this collection, BMC Medicine will present a series of theoretical, methodological, and empirical papers that embrace complexity and chart a path forward for investigating mental disorders as complex systems.

We are seeking submissions in three domains:

• Empirical research. Example topics include causal relations among features of psychopathology, vicious cycles, emergence, attractor states of health and illness, phase transitions, early warning signals, resilience, adaptation, and bridging the gap between biological, psychological and social levels of analysis.
• Methodological contributions that either introduce newly developed methods for investigating mental disorders as complex systems; or that describe applications of methods drawn from other fields (network science, dynamic systems theory) to mental health research.
• Theoretical contributions that adopt a complex systems perspective, especially theories formalized as mathematical or computational models.

Importantly, while the subject of this collection is complexity, we are principally interested in contributions that can be readily understood by a broad audience, with implications not only for researchers, but also clinical practitioners, policy makers, and public health.

We welcome direct submission of original research within the article collection's scope. Please submit directly to BMC Medicine , indicating in your cover letter that you are targeting this collection. Alternatively, you can email a pre-submission query to the editorial team at [email protected] . The collection will remain open and accept submissions until July 2021.

Guest Editors provided guidance on the scope of this collection and advised on commissioned content. However, they are not involved in editorial decision-making on papers submitted to this collection. All final editorial decisions are with the Editor-in-Chief, Dr. Lin Lee.​

Revisiting the seven pillars of RDoC

In 2013, a few years after the launch of the National Institute of Mental Health’s Research Domain Criteria (RDoC) initiative, Cuthbert and Insel published a paper titled “Toward the future of psychiatric diag...

• View Full Text

Understanding the impact of exposure to adverse socioeconomic conditions on chronic stress from a complexity science perspective

Chronic stress increases chronic disease risk and may underlie the association between exposure to adverse socioeconomic conditions and adverse health outcomes. The relationship between exposure to such condit...

The importance of transdiagnostic symptom level assessment to understanding prognosis for depressed adults: analysis of data from six randomised control trials

Depression is commonly perceived as a single underlying disease with a number of potential treatment options. However, patients with major depression differ dramatically in their symptom presentation and comor...

Reducing youth suicide: systems modelling and simulation to guide targeted investments across the determinants

Reducing suicidal behaviour (SB) is a critical public health issue globally. The complex interplay of social determinants, service system factors, population demographics, and behavioural dynamics makes it ext...

Understanding personalized dynamics to inform precision medicine: a dynamic time warp analysis of 255 depressed inpatients

Major depressive disorder (MDD) shows large heterogeneity of symptoms between patients, but within patients, particular symptom clusters may show similar trajectories. While symptom clusters and networks have ...

The long road from person-specific models to personalized mental health treatment

The original article was published in BMC Medicine 2020 18 :345

Using person-specific networks in psychotherapy: challenges, limitations, and how we could use them anyway

The complexity of psychopathology is evident from its multifactorial etiology and diversity of symptom profiles and hampers effective treatment. In psychotherapy, therapists approach this complexity by using c...

The Commentary to this article has been published in BMC Medicine 2020 18 :365

How important are parents in the development of child anxiety and depression? A genomic analysis of parent-offspring trios in the Norwegian Mother Father and Child Cohort Study (MoBa)

Many studies detect associations between parent behaviour and child symptoms of anxiety and depression. Despite knowledge that anxiety and depression are influenced by a complex interplay of genetic and enviro...

Early warning signals in psychopathology: what do they tell?

Despite the increasing understanding of factors that might underlie psychiatric disorders, prospectively detecting shifts from a healthy towards a symptomatic state has remained unattainable. A complex systems...

On the validity of the centrality hypothesis in cross-sectional between-subject networks of psychopathology

In the network approach to psychopathology, psychiatric disorders are considered networks of causally active symptoms (nodes), with node centrality hypothesized to reflect symptoms’ causal influence within a n...

Complexity in psychological self-ratings: implications for research and practice

Psychopathology research is changing focus from group-based “disease models” to a personalized approach inspired by complex systems theories. This approach, which has already produced novel and valuable insigh...

Comorbidity between depression and anxiety: assessing the role of bridge mental states in dynamic psychological networks

Comorbidity between depressive and anxiety disorders is common. A hypothesis of the network perspective on psychopathology is that comorbidity arises due to the interplay of symptoms shared by both disorders, ...

Towards formal models of psychopathological traits that explain symptom trajectories

A dominant methodology in contemporary clinical neuroscience is the use of dimensional self-report questionnaires to measure features such as psychological traits (e.g., trait anxiety) and states (e.g., depres...

Systems all the way down: embracing complexity in mental health research

In this editorial for the collection on complexity in mental health research, we introduce and summarize the inaugural contributions to this collection: a series of theoretical, methodological, and empirical p...

A complex systems approach to the study of change in psychotherapy

A growing body of research highlights the limitations of traditional methods for studying the process of change in psychotherapy. The science of complex systems offers a useful paradigm for studying patterns o...

Bridging the gap between complexity science and clinical practice by formalizing idiographic theories: a computational model of functional analysis

The past decades of research have seen an increase in statistical tools to explore the complex dynamics of mental health from patient data, yet the application of these tools in clinical practice remains uncom...

Mechanisms linking childhood trauma exposure and psychopathology: a transdiagnostic model of risk and resilience

Transdiagnostic processes confer risk for multiple types of psychopathology and explain the co-occurrence of different disorders. For this reason, transdiagnostic processes provide ideal targets for early inte...

Measuring resilience prospectively as the speed of affect recovery in daily life: a complex systems perspective on mental health

There is growing evidence that mental disorders behave like complex dynamic systems. Complex dynamic systems theory states that a slower recovery from small perturbations indicates a loss of resilience of a sy...

The complex neurobiology of resilient functioning after childhood maltreatment

Childhood maltreatment has been associated with significant impairment in social, emotional and behavioural functioning later in life. Nevertheless, some individuals who have experienced childhood maltreatment...

The Correction to this article has been published in BMC Medicine 2020 18 :202

Comparison of brain connectomes by MRI and genomics and its implication in Alzheimer’s disease

The human brain is complex and interconnected structurally. Brain connectome change is associated with Alzheimer’s disease (AD) and other neurodegenerative diseases. Genetics and genomics studies have identifi...

Unravelling the complex nature of resilience factors and their changes between early and later adolescence

Childhood adversity (CA) is strongly associated with mental health problems. Resilience factors (RFs) reduce mental health problems following CA. Yet, knowledge on the nature of RFs is scarce. Therefore, we ex...

Psychological primitives can make sense of biopsychosocial factor complexity in psychopathology

Many agree that the biopsychosocial contributions to psychopathology are complex, yet it is unclear how we can make sense of this complexity. One approach is to reduce this complexity to a few necessary and su...

• Open access
• Published: 26 June 2023

Methodological procedures for priority setting mental health research: a systematic review summarising the methods, designs and frameworks involved with priority setting

• Kris Deering   ORCID: orcid.org/0000-0001-9723-5524 1 ,
• Neil Brimblecombe 2 ,
• Jane C. Matonhodze 3 ,
• Fiona Nolan 4 ,
• Daniela A. Collins 2 &
• Laoise Renwick 5  

Health Research Policy and Systems volume  21 , Article number:  64 ( 2023 ) Cite this article

2013 Accesses

1 Citations

1 Altmetric

Metrics details

Research priority setting aims to identify research gaps within particular health fields. Given the global burden of mental illness and underfunding of mental health research compared to other health topics, knowledge of methodological procedures may raise the quality of priority setting to identify research with value and impact. However, to date there has been no comprehensive review on the approaches adopted with priority setting projects that identify mental health research, despite viewed as essential knowledge to address research gaps. Hence, the paper presents a summary of the methods, designs, and existing frameworks that can be adopted for prioritising mental health research to inform future prioritising projects.

A systematic review of electronic databases located prioritisation literature, while a critical interpretive synthesis was adopted whereby the appraisal of methodological procedures was integrated into the synthesis of the findings. The synthesis was shaped using the good practice checklist for priority setting by Viergever and colleagues drawing on their following categories to identify and appraise methodological procedures: (1) Comprehensive Approach—frameworks/designs guiding the entire priority setting; (2) Inclusiveness –participation methods to aid the equal contribution of stakeholders; (3) Information Gathering—data collecting methods to identify research gaps, and (4) Deciding Priorities—methods to finalise priorities.

In total 903 papers were located with 889 papers removed as either duplicates or not meeting the inclusion and exclusion criteria. 14 papers were identified, describing 13 separate priority setting projects. Participatory approaches were the dominant method adopted but existing prioritisation frameworks were modified with little explanation regarding the rationale, processes for adaptation and theoretical foundation. Processes were predominately researcher led, although with some patient involvement. Surveys and consensus building methods gathered information while ranking systems and thematic analysis tend to generate finalised priorities. However, limited evidence found about transforming priorities into actual research projects and few described plans for implementation to promote translation into user-informed research.

Prioritisation projects may benefit from justifying the methodological approaches taken to identify mental health research, stating reasons for adapting frameworks alongside reasons for adopting particular methods, while finalised priorities should be worded in such a way as to facilitate their easy translation into research projects.

Peer Review reports

Introduction

There is urgency to prioritise mental health research and undertake studies given the scale of international mental health problems, not only in terms of rising mental illness since the Covid pandemic, but also considering the early mortality rates of approximately 20 years for people with serious mental health conditions [ 1 , 2 ]. It is now recognised that the importance of mental health research is equal to other health topics, including the prioritising of mental health studies [ 3 ]. Prioritising mental health research tends to adopt multidimensional approaches given the diversity in what impacts on mental health [ 4 ]. Methodological heterogeneity is common owing to different purposes, aims and contextual factors, alongside vast agendas about which research to prioritise from estimating the magnitude of mental illness burden to identifying gaps with care delivery [ 5 ]. However, Wykes et al. [ 6 ] highlights around a 20-year gap for research to be implemented, and to address specific mental health problems in society, the targeting of research needs to improve.

The World Health Organisation (WHO) [ 7 ] describes priority setting as an interpersonal activity to identify research questions and/or topics with the greatest potential for public benefit. Priority setting may commence with the reviewing of existing studies, alongside guidelines and policies to determine knowledge gaps within a research field [ 7 ]. The importance of these gaps is then refined, and prioritised in order of importance, with ideally the top priority put forward as a research project [ 8 ]. Prioritising of mental health research is argued to take a holistic view including intersecting social issues such as unemployment and mental health seen important to patients [ 9 ]. Nevertheless, questions are raised about a bio-pharmacological focus, suggesting social issues can be overlooked as scientific views might take precedence over patients given the social standing of their expertise concerning mental illness [ 10 , 11 ].

In terms of health research, it has been long recognised that evidence is needed to support the use of methodological processes with priority setting, as well as the procedures involved to identify the studies [ 12 ]. Yet understanding their use with mental health research remains underexplored [ 13 ]. Potential reasons for this are a propensity for priority setting to generate and report on priorities rather than the methods to obtain the results, and lack of funding compared to other areas of healthcare suggesting this too impacts on what priorities are decided [ 14 , 15 ]. In a study of European countries, the share of funded health research dedicated to mental health ranged from 4·0% in the United Kingdom (UK) to 9·7% in Finland [ 16 ], while Woelbert et al. [ 17 ] noted a flat and stable trend in funding over the years 2015–19 and unequal geographical distribution. Even with underfunding, the obligation to prioritise mental health research cannot be overstated. Over 1 billion people are affected by mental disorders globally, bringing about 7% of all global burden of disease with 19% of all years lived with some incapacity owing to mental illness [ 18 ].

No consensus on the optimum model for best practice appears to exist, or what constitutes high quality in developing priorities for mental health research despite growing mental health problems [ 14 ]. This is a knowledge-gap that requires attention given the efficacy of priority setting is “determined by the use of systematic, explicit and transparent processes to increase research funding” ([ 8 ], p.2), while funding for mental health research is disproportionate to other health topics. Methodological procedures are preferably evidence-based to be a vehicle to generate robust results, since mental health research requires to have the greatest potential public health benefit while proficient and fair with use of constrained resources [ 8 , 12 , 13 ]. Explicit procedures may also contribute to an inclusiveness of different voices within projects, rather than the tradition of only academics deliberating what research is prioritised. Namely, patients and their significant others who are ultimately impacted by the changes from research, while procedural transparency can help these groups to assess the rigour in how research was prioritised [ 10 ]. To that end, procedural knowledge that contributes to effective priority setting is essential, and to date, there appears no comprehensive review in what approaches can be adopted and why, with prioritising mental health research [ 13 ].

Rationale for review

Given the factors involving underfunding and burden of mental illness, it is important that priority setting adopts evidence based approaches to identify research with value and impact. In keeping with such conscientiousness, the review aim was to summarise methodological procedures located within current and relevant literature identifying mental health research. Hence, provide a flexibility and critical guide of methodological procedures available for mental health stakeholders who wish to undertake a prioritisation project. The review was supported by a preliminary search of databases such as the Cochrane library Footnote 1 to ensure that a litrature review covering the same topic was not published in some form. Adopting the definition of priority setting as the targeting of research with potential public benefit [ 19 ]; the central question and sub question of the review were as follows:

What methods, designs and frameworks are implemented with priority setting mental health research?

What are the characteristics and purposes of these methodological procedures?

Since the field appeared underexplored, the objective was also to locate and critically evaluate the methodological procedures employed with prioritising mental health research, to inform the discussion about the considerations for future projects later in the paper.

A systematic review of published literature was selected as the best method to address the review questions, in terms of providing a structured process that limits selection bias and generates reliable results [ 20 ]. The latest PRISMA guidance was followed to ensure accurate reporting and rigour in the process of identifying and analysing literature [ 21 , 22 ]. A review protocol was not published on Prospero Footnote 2 as standard practice is not to publish a protocol without patient outcomes; however, the originality of the review was supported by the aforementioned preliminary search.

Search strategy

Frameworks and designs were defined as pre-existing guidance or a methodological approach informing the overall priority setting process, while methods were steps to achieve pertinent stages of prioritisation, such as ranking of priorities [ 12 ]. Mental health was defined in terms of psychological and emotional wellbeing or degree of lacking these when involving illness [ 23 ].

An initial search between 1st July 2020 to 1st November 2020 identified papers limited to scholarly and peer reviewed journal articles for the time period of 1st January 2012 to 1st July 2020. A subsequent search in January 2022 updated the results of papers published between 1st July 2020 to December 31st, 2021, to ensure contemporary findings and the reviewed literature was from the last 10 years (2012–2022). A senior university librarian provided guidance to develop the accuracy of searches, while the following health and social care databases were searched as these potentially hold relevant papers: The Allied and Complementary Medicine Database; CINAHL Plus; MEDLINE; APA PsycArticles; Applied Social Sciences Index and Abstracts; International Bibliography of the Social Sciences; PTSDpubs; Scopus and Social Policy and Practice.

The full text of papers within databases were scanned in case that the abstract or title did not contain the key search terms [ 24 ], while Boolean Operators (AND/OR) were employed to generate search term combinations and Truncations [*] to find variations of the root of a word to expand the search. The following keyword combinations were searched: [“mental health” OR “psychiatry” AND “research priority setting”], [“mental healt*” OR psychiatr* AND “resear* priorit* Sett*”] and [“mental health” AND “decid* sett* AND “resear*”].

Inclusion/exclusion criteria

All retrieved papers were screened for eligibility against the inclusion and exclusion criteria in Table 1 . To not limit findings, there was no exclusion of papers based on priority setting participants or priority setting topic if following the aforementioned definition of mental health. International papers were also accepted in view these may expand the identification and knowledge of methodological procedures adopted with priority setting, though the papers required to be written in English to ensure the literature could be understood.

Data extraction

For both searches, two researchers (K.D. and J.C.M) separately considered all papers for inclusion, discussing any discrepant views together with a third researcher (N.B) to reach a consensus. Identifying papers involved removing duplications through an automated process, then the two researchers (K.D and J.C.M) excluding irrelevant titles and abstracts. The full screening of the remaining papers included checking independently that methodological procedures were clearly explained and present in the articles (K.D, L.R, J.C.M and D. A.C.). To aid this process, recommendations by Tong et al. [ 25 ] The REporting guideline for PRIority SEtting of health research (REPRISE) were followed, and this involved checking if the papers (1) demonstrated the aim of priority setting; (2) highlighted the recruitment strategy; (3) illustrated the participants and (4) presented descriptors of methods. See Fig.  1 for a PRISMA summary of the filtering process.

PRISMA flow diagram of the article search process

Quality appraisal

Despite the apparent paucity of frameworks specifically designed to evaluate the quality of priority setting procedures, an assessment was undertaken to inform the considerations for priority setting section later in the paper, while such appraisal is an expected component of PRISMA guidelines [ 21 , 22 ]. Priority setting procedures may vary greatly from research methodologies and methods [ 26 ]. This can diminish the accuracy of the appraisal using tools to evaluate research, for example the Critical Appraisal Skills Programme (CASP) [ 27 ]. However, a critical interpretive synthesis informed the analysis whereby the appraisal of methodological procedures integrated into the synthesis of the findings [ 28 ]. To promote objectivity, the critical synthesis also adopted the categories from the good practice checklist by Viergever et al. [ 12 ] as recommended by Mador et al. [ 26 ], explained in further detail below.

Data-synthesis

A convergent qualitative design was employed to transform results into a qualitative format, with the method reporting statistics using words rather than figures. This allowed for heterogeneous results to be synthesised into the same review [ 29 ]. The synthesis was informed by abduction, involving the interplay of deduction and induction. Inductively, the checklist by Viergever et al. [ 12 ] guided what constituted methods, designs, and frameworks to find, while induction involved locating these within the priority setting literature selected for the review. The last step was categorising the methodological procedures located using a spreadsheet with columns advised by the checklist, adding rigour to the synthesis by applying a reliable approach to shape the critical outline of findings.

Not all nine categories were utilised from the checklist by Viergever et al. [ 12 ], notably actions following priority setting were omitted as not seen relevant to the review. In addition, the research team discerned that several categories from the checklist could be amalgamated for the purpose of the critical synthesis, involving: (1) Comprehensive Approach —frameworks/designs guiding the entire priority setting, including preparatory work, and reasons for the project; (2) Inclusiveness —participation methods; (3) Information Gathering —data collecting methods to identify research gaps, and; (4) Deciding Priorities —methods involved with finalising priorities [ 12 ].

The findings section outlines the key review results. The characteristics of the priority setting are provided before presenting the main findings synthesised through the good practice checklist. Table 2 presents a summary of the forthcoming synthesis highlighting the typical methodological procedures found tabulated through the four checklist categories.

Priority setting characteristics

Thirteen priority setting projects were described in fourteen separate papers (two of the fourteen described the same project and therefore used the same project) [ 30 , 31 ]. Priorty setting projects were conducted in the United Kingdom ( n  = 3) [ 32 , 33 , 34 ], Australia ( n  = 3) [ 35 , 36 , 37 ], Canada ( n  = 2) [ 30 , 31 ], Canada, Sweden, United Kingdom, and the United States ( n  = 1) [ 38 ], Brazil ( n  = 1) [ 39 ], Chile ( n  = 1) [ 40 ] and Germany ( n  = 1) [ 41 ]. The remaining two papers, one described prioritisation to develop a Roadmap for Mental Health Research in Europe (ROAMER project) [ 42 ] and another developed priority areas across humanitarian settings in low and middle-income countries [ 43 ]. Mental health disorder-specific priorities were identified for depression ( n  = 1) [ 30 , 31 ], depression and bipolar disorder ( n  = 1) [ 35 ], eating disorders ( n  = 1) [ 37 ], obsessive–compulsive disorder ( n  = 1) [ 41 ] or broadly for long-term conditions for older people ( n  = 1) [ 38 ] and mental health in terms of dementia [ 33 ], while research was prioritised for psychosocial interventions in areas of humanitarian need ( n  = 1) [ 43 ].

Critical synthesis

The following is the synthesis of findings informed by the checklist categories. Focus is on the variable ways methodological procedures were employed to guide priority setting projects, while a more detailed account of methods, design and frameworks is provided in Table 3 .

Comprehensive approach

The first category explores frameworks/designs guiding the priority setting including preparatory work, and underpinning reasons for the project. Raising the profile of mental health research (e.g., Aboaja et al. [ 32 ]) and exploring the use of finite resources for service provision (e.g., Zitko et al. [ 40 ]) were common motives to conduct priority setting. However, while limited resources for mental health research, and generating research suitable for funding appeared to be reasons for the projects, no project limited their final priorities based on the rationing of research costs. Alternatively, the majority aimed to document patient and healthcare professional views to inform future research agendas, while two individual projects confined their evaluation to eliciting patient views alone [ 32 , 35 ].

The use of frameworks and designs to guide priority setting was limited, though demarcation existed between aiming to promote public involvement, such as identifying patient and caregiver informed research, and health policy approaches to deciding investment priorities. The latter focused specifically on reducing disease burden and inequity [ 35 , 39 , 40 , 42 ]. Aboaja et al. [ 32 ] and Hart and Wade [ 37 ] employed a modified Delphi approach for their priority setting design involving rounds of questions discussed in groups, then aggregated to reach consensus [ 44 ]. Well-known frameworks for priority setting were identified, notably the Child Health and Nutrition Research Initiative (CHNRI) and the James Lind Alliance (JLA). Defined as an interpersonal framework to build consensus, the JLA aims to generate a top 10-priority list [ 45 ] and four projects used the JLA approach [ 30 , 31 , 38 , 41 ].

The CHNRI employed by Gregório et al. [ 39 ] and Zitko et al. [ 40 ] was based on determining five components: population, disease burden, geographic limits, timescale, and investment [ 46 ]. To fulfil this brief, projects using the CHNRI recruited subject and scientific experts alongside advocates, mid-level implementers and key, strategic, decision-makers at policy level to inform national priority-based resource allocation agendas [ 39 , 40 ]. When the JLA and CHNRI were applied, modifications were made to both frameworks. Attempts were made to improve quality and suitably accommodate the parameters of specific projects, by augmenting structured stages with additional processes and tasks. For example, Breault et al. [30,31:E399] added two additional stages to the JLA partnership model referred to a “funnel approach” to channel patient participation and home in on the generating questions. Conversely, other projects were inspired by the frameworks but omitted key phases of best practice due to what appeared to be a limitation with resourcing [ 33 , 41 ], or making use of existing data [ 38 ].

The JLA [ 47 ] suggests that final priority lists have an existing, adequate evidence base to support adoption and implementation, and comprises of the extensive reviewing of the literature alongside expert checking. This phase appeared omitted by some of the selected projects in the review [ 34 , 35 , 36 , 37 , 38 , 42 ], and may reflect a process issue whereby the finalised priorities are not sufficiently supported by the evidence base [ 12 ]. Two papers suggested that using experts as participants justified not checking whether research existed to answer identified questions [ 39 , 40 ]. However, the researchers focused on ensuring contextual relevance of the final list of priorities by utilising existing policy documents to shape key informant’s discussions in the initial information gathering stages. For example, Zitko et al. [ 40 ] analysed clinical guidelines and national health strategies to identify specific research questions for prioritisation, while Gregório et al. [ 39 ]. directed key informants to guide their deliberations using a national clinical strategy.

International priority-setting projects performed more robust, systematic mapping and syntheses of existing evidence for prioritisation. It is unclear whether systematic mapping influenced the development of priorities in the ROAMER project [ 42 ] though reference to other work packages to document the perspectives of patients, carers, clinicians, and policymakers suggests the researchers aimed to develop a harmonised research priority agenda [ 48 ]. Similarly, in setting global priorities for humanitarian interventions, Lee et al. [ 43 ] considered these complementary processes, inviting 160 key ( n  = 109 accepted) informants for individual consultations to ensure that the information gathered represented international perspectives on important research areas.

Inclusiveness

Inclusiveness identifies participatory methods to aid joint decision-making, and whilst few papers reported operationalised objectives underpinning the methods selected; the majority adopted participatory methods of some form stressing the importance of stakeholder involvement in determining priorities. However, it was not clear how all participants were recruited in some projects [ 30 , 35 , 39 , 42 , 43 , 48 ], although in other projects stakeholders were contacted using databases or patient data held by the lead organisation [ 32 , 33 , 34 , 38 , 41 ] or relevant advocacy groups [ 36 , 37 , 38 ], and social media advertising [ 30 , 31 ].

The aim of the priority setting appeared to impact on participant selection, notably to promote patient involvement and identify their views about beneficial research [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 41 , 42 ], alongside draw on more traditional expertise involving researchers and clinicians [ 33 , 37 , 39 , 40 , 41 , 42 ]. In other projects, ‘users’ were considered as a range of stakeholders of healthcare research and in some included patients, caregivers, and healthcare professionals [ 30 , 31 , 34 , 35 , 41 ] and in others, wider groups included advocates, managers, and administrators [ 35 , 36 , 37 , 38 ]. The extent these priority setting projects enlisted stakeholders to define the parameters of the exercise and mobilise their own communities to produce priorities varied substantially. All except two exercises [ 35 , 36 ], were initiated and led by researchers. Some engaged patients to comment on processes [ 33 , 36 ], appointed steering groups comprising of patients, advocates, professionals, and academics [ 34 , 37 ], or developed partnerships who assumed responsibility for key decisions such as deciding on the scope and overseeing the conduct of successive phases of the projects [ 30 , 31 ].

Information gathering

The following examines the methods to collect relevant data such as research gaps to determine priorities. A mixture of online surveys [ 30 , 31 , 33 , 37 , 38 , 41 , 42 ], structured group discussions [ 32 , 34 , 35 , 36 , 40 , 42 ], stakeholder engagement and systematic review [ 43 ] alongside individual participants listing research gaps [ 39 ] were used to develop initial key questions/topics that needed to be addressed by research. These were prefaced with evidence-based knowledge of emerging research areas, meta-reviews, or existing available databases in some projects [ 37 , 38 , 41 ] to inform the development of surveys.

Information-gathering methods within priority setting included qualitative focus groups—assembling participants to discuss priorities [ 35 , 36 ], nominal group technique (NGT)—structured small-group discussions involving deliberating and voting [ 34 , 43 , 44 ] and modified Delphi exercises [ 32 , 37 , 42 ]. Group discussions were used to bring stakeholders together to identify priorities in some approaches [ 32 , 34 , 35 , 36 ] two of which generated and ranked priorities at the same meeting [ 35 , 36 ] and one used existing patient community meetings within hospitals [ 32 ].

In addition to the three consensus building methods described, surveys and online consultations were also used. One project engaged members of a steering group to codesign questionnaires [ 31 ], one engaged researchers and patients [ 33 ] and in another, researchers worked with wider advocacy or patient groups [ 37 ]. However, one project designed the survey without stakeholder participation though it was evidence-informed in which priorities were cross-referenced with the literature [ 41 ]. Measures were taken to enhance the relevance of survey questions to potential participants including providing examples and definitions of research [ 30 , 31 , 36 ], categorising research areas in advance of the survey [ 33 , 37 , 38 ], and utilising evidence and policy to inform the design [ 37 , 39 , 41 ]. However, no projects reported piloting or refining the questionnaire before commencing the survey.

Deciding priorities

The last section considers the methods to finalise priorities presented in two-parts; refinement/ranking and finalisation of priorities.

Refining and ranking generated priorities

The key task of refining stakeholder-generated priorities is formulating questions that conform to searchable frameworks while retaining the intended meaning of the respondent. In some instances, projects sought to identify thematic areas of topics that circumvented the need to identify specific questions [ 32 , 35 , 37 ] and were derived through qualitative analysis of responses, such as workshops [ 30 , 31 ], including “dot-mocracy”, using adhesive dots on a flipchart to vote for research topics ([ 36 ], p. 2). Other refining methods involved online surveys [ 33 ], ranking [ 39 , 41 ], and expert analysis without patients [ 37 , 40 ]. Metric based ranking and obtaining final priority lists were also merged into one exercise in some projects such that ranking and gaining consensus was merged into one activity, e.g., Forsman et al. [ 42 ].

Finalising priorities

Group consensus approaches were used in several projects, although as highlighted priority metric-based ranking was also employed which resulted in final priority lists [ 32 , 36 , 42 ]. The outcome for priority setting included valuable lists of research gaps without necessarily agreement on which should be prioritised [ 35 ]. In other projects, respondents identified their top three priorities and frequency counts were obtained, without always explaining whether these responses were weighted. Aboaja et al. [ 32 ] identified weighting of 10.7% with patient responses, whereas Breault et al., [ 30 , 31 ] provided little detail in terms of responses though presented the demographics of the participants who responded. Ranking of priorities also varied, with several projects distributing successive phases of ranked data for further refinement based on sophisticated criteria [ 39 , 42 , 43 ], while one project took a percentage of endorsements of broad research priorities [ 37 ]. Collaborative workshops based on consensus-methods were also utilised, employing NGTs or adapted versions of these [ 30 , 31 , 33 , 38 ] which are strengthened by the iterative nature of gaining consensus on priorities through active discussion and participation. However, only one project selected a top research priority using participant voting in workshops [ 34 ].

Priority setting frameworks predominately employed within the sample of fourteen papers were the JLA and CHNRI. The JLA was the most used although often in modified form, and whilst not always clear as to why these adaptions occurred, Kühne et al. [ 41 ] reported this was owing somewhat to financial constraints. Not only can cost potentially impact on the way frameworks are adopted but also patient involvement, notably Boivin et al. [ 49 ] identified a 17% increased cost for involving patients, suggesting such stakeholders can be priced out of participation. The other notable framework identified was the CHNRI, also modified, with apparent focus on some of its categories to collate research topics involving symptomology, illness burden, equality, and budgetary impact [ 39 , 40 ]. Some papers did attempt to explain adaptations made to frameworks by signposting to other articles, although not necessarily fully clarifying the reasons for changes. Amongst motives for such signposting, may involve ‘Salami Slicing’ whereby the project is published over several articles to increase citations, lessening understanding of methodological procedures, as not presented as a cohesive whole in one article [ 50 ].

In addition to the JLA and CHNRI frameworks, it was also found that the papers used two objectives to inform the priority setting projects:

Generate research topics in terms of available or limited resources, for example the affordability of research [ 39 ], efficient use of limited research funding [ 37 ], the cost effectiveness of research [ 40 ] and/or

Capture the voices of living experiences, for example, from patients and caregivers to inform care [ 36 ].

Barra et al. [ 51 ] characterises these two points as a likely politicising amongst stakeholder views between generating meaningful research and research rationing, given finite resources. Rationing, in terms of identifying research based on cost effectiveness alone was not overly apparent, though as point A. highlights, rationing of mental health research in some way was reason to why some projects occurred. Hence, when influenced by what can be realistically funded, a politically charged terrain does seem inescapable, especially as such restrictions may potentially shape priorities not necessarily addressing patient concerns, or insufficiently substantial to initiate policy changes that improve mental health [ 14 ].

Methodological procedures in the papers were also found to be somewhat directed by the priority setting aim. Preferences included consensus building, particularly when the aim was to enrich the patient voice, symbolic of going to the heart of mental health care involving coproducing knowledge through some interpersonal connection [ 52 ]. These resonated with democratic group methods such as the NGT to ensure all voices were heard, but also, not necessarily concurrently, discursive methods like qualitative focus groups, at times involving policymakers and budget holders when seemingly tied to seeking value for money to inform national policy [ 40 ]. Some projects ranked and engaged in discursive exercises to gather uncertainties simultaneously, e.g., Forsman et al. [ 42 ]. Whilst the approach may lessen the dominance of individuals and reduce cost, it could result in a common representation of research priorities to ensure participants make an agreement. This could impact on the originality of the priorities, and without necessarily addressing a knowledge gap, may limit the implementation as a research project [ 53 ].

Several groups recruited for the priority setting projects appeared to represent the target stakeholder population, whilst the recruitment process of other projects lacked clarity. For example, priority setting considered the mental health of young people [ 34 , 42 , 43 ], though the reporting of involving young people as participants was not clear, and if not involved, suggests a possible disparity with prioritising research enlightened by the views of children and adolescents. In general, greater opportunities for participation existed for those from professional backgrounds, raising philosophical questions in what constitutes expert knowledge with some priority setting projects [ 54 ]. Professionals such as policymakers and scientists may have better vantage points given their expertise and experience about the feasibility of priority setting and ways to reach the endpoint of funded research [ 55 ]. Cost of research training might also have implications about who can participate [ 49 ]. However, living experiences of care are attributes to identify meaningful research topics, signifying the importance of patient and caregiver views, and whilst training cost is an issue, it may simply involve raising awareness about the parameters of prioritising research to ensure its success [ 56 , 57 ].

Not all projects started with a clear scope or terms of reference. Whereas some commenced with literature reviews, systematic mapping reviews were an alternative. Although use of review mapping in one project was unclear in how it impacted on the priority setting process [ 42 ], the method can aid prioritising by mapping research gaps within a given research field, providing further evidence to implement the identified priorities as research projects [ 58 ]. While bringing about an evidence-informed approach, identifying priorities from available databases or research may narrow patient choices. Final priority lists could potentially omit research areas that are both important to patients and neglected by research reducing the potential impact of priority-setting project to address gaps in the evidence base. Alternatively, different forms of surveys were adopted to commence priority setting, drawing on wider and on occasion more ambiguous research terrain.

Overlooking the gaps and needs of the research field, makes priority setting difficult to achieve [ 25 ]. Reviewing the literature suggests that a pragmatic approach is needed in preparation for a prioritisation project, to improve its focus with mapping out research gaps, but combined with gathering diverse expertise, such as from patients when concerning care, to improve the understanding of research needs [ 47 ]. This appeared within a contextual focus concerning particular mental health conditions or other relevant care factors aligning to the participant expertise. For example, when seeking to make use of resources in some way, budget holders appeared more recruited for priority setting projects [ 40 ].

Having a clearly defined aim is likely to help inform the methodological procedures to be taken in a prioritisation process. The aim should take account of the complex context, including funding, resources, and feasibility and other factors influencing mental health research [ 3 , 47 ]. Clear and precise project aims may be less likely to produce broad themes that appear too ambiguous to be financed [ 26 ]. Given the limited research funding available, methodological procedures must be such that the endpoint of priority setting are research topics that easily translate into actual investigations.

Although themes might not always convert well into specific research projects, limitations with funding also play a role in skewing research priorities towards those involving hypothesis testing. This may not always correspond with what patients find useful, for example, understanding experiences of care to develop practice [ 59 ]. Despite the aforementioned risk of politicising, without taking funding into consideration, priority setting might give the impression of appearing superfluous if not leading to substantial investigations. When involving patients, priority setting in such circumstances could appear tokenistic, and reaffirm a sense of underrepresentation, by patient views not transforming into actual research projects [ 56 ]. The same could be proposed with lists without obvious ranking, suggesting a further step is required to home in on a specific priority, in consideration of the competitiveness, and limited funding available for mental health research. The JLA [ 47 ] somewhat echoes this view, in which priority setting results in the top 10 priorities in order of importance.

Considerations for priority setting

The critical analysis of priority setting procedures seems a fledgling field. However, the checklist by Viergever et al. [ 12 ] not only supported the synthesis of findings, but alongside the discussion of the paper, helped to develop the following considerations to inform future priority setting projects specific to mental health research.

Priority setting appeared beneficial when involving a range of expertise, as highlighted by Foresman et al. [ 42 ], aligning patients, scientists, and policymakers to subgroups in which they may have greater knowledge, while subgroup views were reviewed by other participants [ 42 ]. Given priority setting may examine mental health concepts that are broad in nature, the above approach might be considered for it allows a deep dive into specific parts that make up the vast mental health field under exploration [ 60 ].

Despite the review highlighting inclusivity of patients and caregiver views, there was little evidence of co-producing the priority setting project with these participants. Hence suggested is that such involvement improves to enhance the identifying of research relevant to those in receipt of care and their significant others.

The papers reviewed invariably reported the adoption of recommendations or good practice guidance such as Viergever et al. [ 12 ], and given the importance of rigour with identifying priorities, such guidance is ideally utilised to shape the priority setting project.

When adapting frameworks for example as provided by the JLA, consideration is given to these adaptions as part of writing up, alongside stating why these adaptions were made. This can help to understand methodological congruence, and although predominately applied to research, the WHO [ 7 ] alludes to the approach when planning the coherence of projects, so that the priority setting aim(s) aligns to the purposes amongst its methodological parts. Thus, provide the rationale for adaptions and why methods were employed, also acknowledging the shaping of methodological procedures via limitations such as funding and feasibility [ 25 ].

The aim(s) and approach of the final research priorities needs to be explained to aid their funding. Priorities otherwise may not develop into research projects and may reaffirm that some participants are less likely to have their voices heard, notable with patients [ 61 ].

Given the diversity of mental health research, the final consideration is for priority setting to go beyond only illness. Problematising mental health appeared evident with the literature, loosely tied to mental illness and mental health problems. Research about mitigating illness may receive more funding over maintaining and promoting mental health [ 10 ]. However, consideration should also be given in how research can enrich the lives of people, so they may thrive and thereby lessen the prevalence of mental health difficulties [ 62 , 63 ].

Review limitations

The review was limited by challenges with identifying search terms for prioritisation, which potentially may have excluded papers otherwise meeting the inclusion criteria. The lack of a standardised approach to the critical appraisal was also a limitation, for such appraisal is the cornerstone of systematic reviews to assess the quality of investigative methods and inform the direction of future research [ 20 ]. However, to apply a critical approach, the review drew on the seminal work of Viergever et al. [ 12 ] to guide the synthesis and inform the above considerations. Whilst perhaps not providing the depth of critique such as employing the CASP [ 27 ] with reviewing research, a recognised approach was nevertheless utilised to identify and review the methodological procedures located within priority-setting projects.

This systematic review summarised frameworks, designs and methods adopted with priority setting for mental health research, to inform stakeholders in mental health about the methodological procedures to conduct priority setting, be it from grassroot levels to more national approaches. The findings highlighted that while a growing trend with involving participation from experts by experience such as patients, there is room to improve their leadership roles where feasible. Prioritisation frameworks, notably the JLA and the CHNRI were utilised but were adapted in practice, potentially impacting on methodological quality. Generally, greater clarity in defining the aims of priority setting would support the appropriate selection of methodological procedures that may lead to the creation of actual research projects.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

The Cochrane Library is a database containing high-quality health related systematic reviews https://www.cochranelibrary.com/ .

The International Prospective Register of Systematic Reviews (PROSPERO) is database to register a systematic review protocol to raise awareness of the work and limit duplications https://www.crd.york.ac.uk/prospero/ .

Holmes EA, O’Connor RC, Perry VH, Tracey I, Wessely S, Arseneault L, Ballard C, Christensen H, Silver RC, Everall I, Ford T, John A, Kabir T, King K, Madan I, Michie S, Przybylski AK, Shafran R, Sweeney A, Worthman CM, Yardley L, Cowan K, Cope C, Hotopf M, Bullmore E. Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry. 2020;7(6):547–60.

Article   PubMed   PubMed Central   Google Scholar  

Ferrari AJ, Santomauro DF, Mantilla Herrera AM, Shadid J, Ashbaugh C, Erskine H E, Charlson FJ, Degenhardt L, Scott JG, McGrath JJ, Allebeck P, Benjet C, Breitborde NJK, Brugha T, Dai X, Dandona l, Dandona R, Fischer F, Haagsma JA, Haro JM, Kieling C, Skrindo Knudsen AK, Kumar GA, Leung J, Majeed A, Mitchell PB, Moitra M, Mokdad AH, Molokhia M, Patten SB, Patton GC, Phillips MR, Soriano JB, Stein DJ, Stein MB, Szoeke CE., Mohsen Naghavi M, Hay SI, Christopher JL, Murray CJL, Vos T, Whiteford HA. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Psychiatry. 2022;9(2):137–150.

Department of Health. A Framework for mental health research. 2017. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/665576/A_framework_for_mental_health_research.pdf . Accessed 18 Apr 2021.

Alegria M, Atkins M, Farmer E, Slaton E, Stelk W. One size does not fit all: taking diversity, culture and context seriously. Adm Policy Ment Health. 2010;37(1):48–60.

Noorain S, Paola Scaparra M, Kotiadis K. Mind the gap: a review of optimisation in mental healthcare service delivery. Health Syst. 2022;13:1–34.

Google Scholar  

Wykes T, Bell A, Carr S, Coldham T, Gilbody S, Hotopf M, Johnson S, Kabir T, Pinfold V, Sweeney A, Jones PB. Shared goals for mental health research: what, why and when for the 2020s. Jo Mental Health. 2021; ahead-of-print.

World Health Organisation. A systematic approach for undertaking a research priority-setting exercise: guidance for WHO staff. 2020. https://apps.who.int/iris/handle/10665/334408 . Accessed 11 Feb 2021.

Bryant J, Sanson-Fisher R, Walsh J, Stewart J. Health research priority setting in selected high income countries: a narrative review of methods used and recommendations for future practice. Cost Eff Resour Alloc. 2014;12(1):1–11.

Article   Google Scholar  

Academy of Medical Sciences. Progress and priorities for mental health sciences research since COVID-19. Summary report of a joint Academy of Medical Sciences/MQ virtual workshop on 23 April 2021. 2021 https://acmedsci.ac.uk/file-download/96569754 . Accessed 7 Jan 2022.

Patel V. Why mental health matters to global health. Transcult Psychiatry. 2014;51(6):777–89.

Article   PubMed   Google Scholar  

Levelink M, Voigt-Barbarowicz M, Bruett AL. Priorities of patients, caregivers and health-care professionals for health research—a systematic review. Health Expect. 2020;23(5):992.

Viergever RF, Olifson S, Ghaffar A, Terry RF. A checklist for health research priority setting: nine common themes of good practice. Health Res Policy Syst. 2010;8(1):1–9.

Pringle B, Williams M, Collins PY. Evidence for action: Stimulating research to address the global mental health treatment gap. In: Okpaku S, editor. Innovations in global mental health. Switzerland: Springer; 2021. p. 63–88.

Chapter   Google Scholar  

Patel V. Mental health research funding: too little, too inequitable, too skewed. Lancet Psychiatry. 2020;8(3):171–2.

Christensen H, Batterham PJ, Hickie IB, McGorry PD, Mitchell PB, Kulkarni J. Funding for mental health research: The gap remains. Med J Aust. 2011;195(11–12):681–4.

Wong EC, Maher AR, Motala A, Ross R, Akinniranye O, Larkin J, Hempel S. Methods for identifying health research gaps, needs, and priorities: a scoping review. J Gen Intern Med. 2021;37(1):198–205.

Hazo JB, Gandré C, Leboyer M, Obradors-Tarragó C, Belli S, McDaid D, Park AL, Maliandi MV, Wahlbeck K, Wykes T, van Os J. National funding for mental health research in Finland, France, Spain and the United Kingdom. Eur Neuropsychopharmacol. 2017;27(9):892–9.

Article   CAS   PubMed   Google Scholar  

Woelbert E, Lundell-Smith K, White R, Kemmer D. Accounting for mental health research funding: developing a quantitative baseline of global investments. The Lancet Psychiatry. 2021;8(3):250–8.

Rehm J, Shield KD. Global burden of disease and the impact of mental and addictive disorders. Curr Psychiatry Rep. 2019;21(2):1–7.

Montorzi G, de Haan S, IJsselmuiden C. Priority Setting for Research for Health: a management process for countries. 2010. http://www.cohred.org/downloads/-Priority_Setting_COHRED_approach_August_2010.pdf . Accessed 10 Jun 2021.

Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):1–7.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow, CD, Shamseer L, Tetzlaff, JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco A C, Welch VA, Whiting P, Moher, D. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. 2020. https://www.equator-network.org/reporting-guidelines/prisma/ . Accessed 3 Jan 2021.

Manwell LA, Barbic SP, Roberts K, Durisko Z, Lee C, Ware E, McKenzie K. What is mental health? Evidence towards a new definition from a mixed methods multidisciplinary international survey. BMJ Open. 2015;5(6):1–11.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016;5(1):1–23.

Tong A, Synnot A, Crowe S, Hill S, Matus A, Scholes-Robertson N, Oliver S, Cowan K, Nasser M, Bhaumik S, Gutman T. Reporting guideline for priority setting of health research (REPRISE). BMC Med Res Methodol. 2019;19(1):1–11.

Article   CAS   Google Scholar  

Mador RL, Kornas K, Simard A, Haroun V. Using the Nine Common Themes of Good Practice checklist as a tool for evaluating the research priority setting process of a provincial research and program evaluation program. Health Res Policy Syst. 2016;14(1):1–9.

Critical Appraisal Skills Programme (CASP)UK. CASP checklists. 2022. https://casp-uk.net/casp-tools-checklists/ . Accessed 5 May 2022.

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6(1):1–3.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35(1):29–45.

Breault LJ, Rittenbach K, Hartle K, Babins-Wagner R, de Beaudrap C, Jasaui Y, Ardell E, Purdon SE, Michael A, Sullivan G, Vandall-Walker L. People with lived experience (PWLE) of depression: describing and reflecting on an explicit patient engagement process within depression research priority setting in Alberta, Canada. Res Involv Engagem. 2018;4(1):1–10.

Breault LJ, Rittenbach K, Hartle K, Babins-Wagner R, de Beaudrap C, Jasaui Y, Ardell E, Purdon SE, Michael A, Sullivan G, Vandall-Walker L. The top research questions asked by people with lived depression experience in Alberta: a survey. Can Med Assoc Open Access J]. 2018;6(3):E398-405.

Aboaja A, Forsyth B, Bates H, Wood R. Involving service users to identify research priorities in a UK forensic mental health service. BJPsych Bull. 2021;45(6):321–6.

Emrich-Mills L, Hammond LL, Rivett E, Rhodes T, Richmond P, West J. Identifying research priorities for older people’s mental health services. Ment Health Soc Incl. 2019;23(2):89–100.

Ghisoni M, Wilson CA, Morgan K, Edwards B, Simon N, Langley E, Rees H, Wells A, Tyson PJ, Thomas P, Meudell A. Priority setting in research: user led mental health research. Res Involv Engagem. 2017;3(1):1–7.

Banfield MA, Barney LJ, Griffiths KM, Christensen HM. Australian mental health consumers’ priorities for research: qualitative findings from the SCOPE for Research project. Health Expect. 2014;17(3):365–75.

Banfield MA, Morse AR, Gulliver A, Griffiths KM. Mental health research priorities in Australia: a consumer and carer agenda. Health Res Policy Syst. 2018;16(1):1–11.

Hart LM, Wade T. Identifying research priorities in eating disorders: A Delphi study building consensus across clinicians, researchers, consumers, and carers in Australia. Int J Eat Disord. 2020;53(1):31–40.

Chamberlain SA, Estabrooks CA, Keefe JM, Hoben M, Berendonk C, Corbett K, Gruneir A. Citizen and stakeholder led priority setting for long-term care research: identifying research priorities within the Translating Research in Elder Care (TREC) Program. Research involvement and engagement. 2020;6(1):1–9.

Gregório G, Tomlinson M, Gerolin J, Kieling C, Moreira HC, Razzouk D, Mari JD. Setting priorities for mental health research in Brazil. Braz J Psychiatry. 2012;34:434–9.

Zitko P, Borghero F, Zavala C, Markkula N, Santelices E, Libuy N, Pemjean A. Priority setting for mental health research in Chile. Int J Ment Heal Syst. 2017;11(1):1–8.

Kühne F, Brütt AL, Otterbeck MJ, Weck F. Research priorities set by people with OCD and OCD researchers: Do the commonalities outweigh the differences? Health Expect. 2021;24:40–6.

Forsman AK, Wahlbeck K, Aarø LE, Alonso J, Barry MM, Brunn M, Cardoso G, Cattan M, De Girolamo G, Eberhard-Gran M, Evans-Lacko S, Fiorillo A, Hansson L, Haro JM, Hazo JB, Hegerl U, Katschnig H, Knappe S, Luciano M, Miret M, Nordentoft M, Obradors-Tarragó C, Pilgrim D, Ruud T, Salize HJ, Stewart-Brown SL, Tómasson K, van der Feltz-Cornelis CM, Ventus DB, Vuori J, Värnik A. Research priorities for public mental health in Europe: recommendations of the ROAMER project. Eur J Public Health. 2015;25(2):249–54.

Lee C, Nguyen AJ, Haroz E, Tol W, Aules Y, Bolton P. Identifying research priorities for psychosocial support programs in humanitarian settings. Global Mental Health. 2019;6(e23):1–10.

McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38(3):655–62.

PubMed   PubMed Central   Google Scholar  

Rowbotham NJ, Smith SJ, Elliott ZC, Leighton PA, Rayner OC, Morley R, Smyth AR. Adapting the James Lind Alliance priority setting process to better support patient participation: an example from cystic fibrosis. Res Involv Engagem. 2019;5(1):1–8.

Rudan I. Setting health research priorities using the CHNRI method: IV. Key conceptual advances. J Glob Health. 2016;6(1):1–8.

James Lind Alliance. The James Lind Alliance Guidebook version 10 [updated March 2021]. 2021. https://www.jla.nihr.ac.uk/jla-guidebook/downloads/JLA-Guidebook-Version-10-March-2021.pdf . Accessed 2 Feb 2022.

Fiorillo A, Luciano M, Del Vecchio V, Sampogna G, Obradors-Tarragó C, Maj M. Priorities for mental health research in Europe: A survey among national stakeholders’ associations within the ROAMER project. World Psychiatry. 2013;12(2):165–70.

Boivin A, Lehoux P, Lacombe R, Burgers J, Grol R. Involving patients in setting priorities for healthcare improvement: a cluster randomized trial. Implement Sci. 2014;9(1):1–10.

Tolsgaard MG, Ellaway R, Woods N, Norman G. Salami-slicing and plagiarism: How should we respond? Adv Health Sci Educ. 2019;24(1):3–14.

Barra M, Broqvist M, Gustavsson E, Henriksson M, Juth N, Sandman L, Solberg CT. Severity as a priority setting criterion: setting a challenging research agenda. Health Care Anal. 2020;28(1):25–44.

Ehrlich C, Kendall E, Frey N, Denton M, Kisely S. Consensus building to improve the physical health of people with severe mental illness: a qualitative outcome mapping study. BMC Health Serv Res. 2015;15(1):1–9.

Sibbald SL, Singer PA, Upshur R, Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res. 2009;9(1):1–2.

Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, Brito JP, Boehmer K, Hasan R, Firwana B, Erwin P. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14(1):1–9.

George AS, Mehra V, Scott K, Sriram V. Community participation in health systems research: a systematic review assessing the state of research, the nature of interventions involved and the features of engagement with communities. PLoS ONE. 2015;10(10):1–25.

Rand L, Dunn M, Slade I, Upadhyaya S, Sheehan M. Understanding and using patient experiences as evidence in healthcare priority setting. Cost Eff Resour Alloc. 2019;17(1):1–3.

Pratt B. Achieving inclusive research priority-setting: what do people with lived experience and the public think is essential? BMC Med Ethics. 2021;22(1):1–4.

Hawwash D, Pinxten W, Bonn NA, Verstraeten R, Kolsteren P, Lachat C. Perspective: consideration of values when setting priorities in nutrition research: guidance for transparency. Adv Nutr. 2018;9(6):671–87.

Haufe C. Why do funding agencies favor hypothesis testing? Stud Hist Philos Sci. 2013;44(3):363–74.

Tan A, Nagraj SK, Nasser M, Sharma T, Kuchenmüller T. What do we know about evidence-informed priority setting processes to set population-level health-research agendas: an overview of reviews. Bull Natl Res Cent. 2022;46(1):1–20.

Grill C. Involving stakeholders in research priority setting: a scoping review. Res Involv Engagem. 2021;7(1):1–8.

Rose D, Fleischman P, Wykes T. What are mental health service users’ priorities for research in the UK? J Ment Health. 2008;17(5):520–30.

Wahlbeck K. European comparisons between mental health services. Epidemiol Psychiatr Sci. 2011;20(1):15–8.

Download references

Acknowledgements

Mental Health Nurse Academics of United Kingdom— https://mhnauk.org/ .

No funding utilised.

Author information

Authors and affiliations.

University of Exeter Medical School, St Luke’s Campus, 79 Heavitree Rd, Exeter, EX1 2LT, United Kingdom

Kris Deering

London South Bank University, 103 Borough Road, London, SE1 0AA, United Kingdom

Neil Brimblecombe & Daniela A. Collins

University of Greenwich, Avery Hill Campus, Southwood Site, Avery Hill Road, London, SE9 2UG, United Kingdom

Jane C. Matonhodze

Anglia Ruskin University, Chelmsford Campus Bishop Hall Lane, Chelmsford, CM1 1SQ, United Kingdom

Fiona Nolan

The University of Manchester, Oxford Rd, Manchester, M13 9PL, United Kingdom

Laoise Renwick

You can also search for this author in PubMed   Google Scholar

Contributions

KD led on the writing, searching and analysis of the literature; NB assisted with analysis and writing the paper; JCM assisted with identifying literature and contributed to the abstract and conclusion of the paper; FN assisted with the writing of paper; DAC assisted with the analysis of findings; LR assisted with writing the paper with KD, designed the process of analysis, and producing findings. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kris Deering .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Deering, K., Brimblecombe, N., Matonhodze, J.C. et al. Methodological procedures for priority setting mental health research: a systematic review summarising the methods, designs and frameworks involved with priority setting. Health Res Policy Sys 21 , 64 (2023). https://doi.org/10.1186/s12961-023-01003-8

Download citation

Received : 07 November 2022

Accepted : 18 May 2023

Published : 26 June 2023

DOI : https://doi.org/10.1186/s12961-023-01003-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research priority setting
• Mental health research
• Priority setting frameworks
• Priority setting designs
• Priority setting methods
• Methodological procedures

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Advertisement

Methodology Considerations in School Mental Health Research

• Original Paper
• Published: 04 January 2014
• Volume 6 , pages 68–83, ( 2014 )

Cite this article

• Gregory A. Fabiano 1 ,
• Sandra M. Chafouleas 2 ,
• Mark D. Weist 3 ,
• W. Carl Sumi 4 &
• Neil Humphrey 5  

1243 Accesses

21 Citations

1 Altmetric

Explore all metrics

Research in the area of school mental health (SMH) has undergone rapid evolution and expansion, and as such, studies require the use of diverse and emerging methodologies. In parallel with the increase in SMH research studies has been greater realization of the complex research methods needed for the optimal measurement, design, implementation, analysis, and presentation of results. This paper reviews key steps needed to effectively study SMH research questions. Considerations around research designs, methods for describing effects and outcomes, issues in measurement of process and outcomes, and the foundational role of school and community research partnerships are discussed within the context of SMH research studies. Ongoing developments within SMH research methods are presented as illustrative examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

A worked example of Braun and Clarke’s approach to reflexive thematic analysis

David Byrne

How to use and assess qualitative research methods

Loraine Busetto, Wolfgang Wick & Christoph Gumbinger

Academic Stress Interventions in High Schools: A Systematic Literature Review

Tess Jagiello, Jessica Belcher, … Viviana M. Wuthrich

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101 , 213–232.

Article   PubMed   Google Scholar  

Agency for Healthcare Research. (2013). Methods guide for effectiveness and comparative effectiveness reviews . Rockville, MD: Agency for Healthcare Research.

Google Scholar  

Andis, P., Cashman, J., Praschil, R., Oglesby, D., Adelman, H., Taylor, L., et al. (2002). A strategic and shared agenda to advance mental health in schools through family and system partnerships. International Journal of Mental Health Promotion, 4 , 28–35.

Article   Google Scholar  

Atkins, M. S., Hoagwood, K. E., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 37 , 40–47.

Article   PubMed Central   PubMed   Google Scholar  

Berkowitz, H. (1968). A preliminary assessment of the extent of interaction between child psychiatric clinics and public schools. Psychology in the Schools, 5 , 291–295.

Brandt, N. E., Glimpse, C., Fette, C., Lever, N. A., Cammack, N. L., & Cox, J. (2014). Advancing effective family–school–community partnerships. In M. Weist, N. Lever, C. Bradshaw, & J. Owens (Eds.), Handbook of school mental health: Research, training, practice, and policy (pp. 209–222). New York: Springer.

Chapter   Google Scholar  

Briesch, A. M., Chafouleas, S. M., Neugebauer, S. R., & Riley-Tillman, T. C. (2013). Assessing influences on intervention use: Revision of the usage rating profile-intervention. Journal of School Psychology, 51 , 81–96.

Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research . Chicago, IL: Rand-McNally.

Campbell, M. K., Thomson, S., Ramsay, C. R., MacLennan, G. S., & Grimshaw, J. M. (2004). Sample size calculator for cluster randomized trials. Computers in Biology and Medicine, 34 , 113–125.

Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., Christ, T. C., Black, A. C., & Kilgus, S. P. (2010a). An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology, 48 , 219–246. doi: 10.1016/j.jsp.2010.02.001 .

Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., Christ, T. J., Volpe, R. J., & Gresham, F. (2009, August). Review of methods for formative assessment of child social behavior . In Symposium presented at the meeting of the American Psychological Association, Toronto, Canada.

Chafouleas, S. M., Kilgus, S. P., Riley-Tillman, T. C., Jaffery, R., & Harrison, S. (2012). Preliminary evaluation of various training components on accuracy of direct behavior ratings. Journal of School Psychology. doi: 10.1016/j.jsp.2011.11.007 .

Chafouleas, S. M., Volpe, R. J., Gresham, F. M., & Cook, C. (2010b). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review, 34 , 343–349.

Clauss-Ehlers, C., Serpell, Z., & Weist, M. D. (2013). Handbook of culturally responsive school mental health: Advancing research, training, practice, and policy . New York: Springer.

Book   Google Scholar  

Cohen, J. (1992). A power primer. Psychological Bulletin, 112 , 155–159.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings . Boston, MA: Houghton Mifflin.

Cordray, D. S., & Pion, G. M. (2006). Treatment strength and integrity: Models and methods. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, D.C.: American Psychological Association.

Craig, W. M., & Pepler, D. J. (1998). Observations of bullying and victimization in the school yard. Canadian Journal of School Psychology, 13 , 41–59.

Cunningham, C. E., Vaillancourt, T., Rimas, H., Deal, K., Cunningham, L., Short, K., et al. (2009). Modeling the bullying prevention program preferences of educators: A discrete choice conjoint experiment. Journal of Abnormal Child Psychology, 37 , 929–943.

Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18 , 23–45.

Donner, A., & Klar, N. (2004). Pitfalls of and controversies in cluster randomization trials. American Journal of Public Health, 94 , 416–422.

Dumas, J. E., Lynch, A. M., Laughlin, J. E., Phillips Smith, E., & Prinz, R. J. (2001). Promoting intervention fidelity: Conceptual issues, methods, and preliminary results from the Early Alliance prevention trial. American Journal of Preventive Medicine, 20 , 38–47.

Duncan, G. J., & Magnuson, K. (2007). Penny wise and effect size foolish. Child Development Perspectives, 1 , 46–51.

DuPaul, G. J., Eckert, T. L., & Vilardo, B. (2012). The effects of school-based interventions for attention deficit hyperactivity disorder: A meta-analysis 1996–2010. School Psychology Review, 41 , 387–412.

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41 , 327–350.

Evans, S. W., Allen, J., Moore, S., & Strauss, V. (2005). Measuring symptoms and functioning of youth with ADHD in middle schools. Journal of Abnormal Child Psychology, 33 , 695–706.

Evans, S. W., Brady, C. E., Harrison, J. R., Bunford, N., Kern, L., State, T., et al. (2013). Measuring ADHD and ODD symptoms and impairment using high school teachers’ ratings. Journal of Clinical Child and Adolescent Psychology, 42 , 197–207.

Evans, C., Margison, F., & Barkham, M. (1998). The contribution of reliable and clinically significant change methods to evidence-based mental health. Evidence-Based Mental Health, 1 , 70–73.

Fabiano, G. A., Pelham, W. E., Gnagy, E. M., Burrows-MacLean, L., Coles, E. K., Chacko, A., et al. (2007). The single and combined effects of multiple intensities of behavior modification and multiple intensities of methylphenidate in a classroom setting. School Psychology Review, 36 , 195–216.

Fabiano, G. A., Pelham, W. E., Waschbusch, D., Gnagy, E. M., Lahey, B. B., Chronis, A. M., et al. (2006). A practical impairment measure: Psychometric properties of the impairment rating scale in samples of children with attention-deficit/hyperactivity disorder and two school-based samples. Journal of Clinical Child and Adolescent Psychology, 35 , 369–385.

Fabiano, G. A., Vujnovic, R., Pelham, W. E., Waschbusch, D. A., Massetti, G. M., Yu, J., et al. (2010). Enhancing the effectiveness of special education programming for children with ADHD using a daily report card. School Psychology Review, 39 , 219–239.

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature . (FMHI Publication #231). Tampa, FL: University of South Florida, National Implementation Research Network.

Fonagy, P., Twemlow, S. W., Vernberg, E. M., Nelson, J. M., Dill, E. J., Little, T. D., et al. (2009). A cluster randomized controlled trial of child-focused psychiatric consultation and a school systems-focused intervention to reduce aggression. Journal of Child Psychology and Psychiatry, 50 , 607–616.

Frazier, S. L., Formoso, D., Birman, D., & Atkins, M. S. (2008). Closing the research to practice gap: Redefining feasibility. Clinical Psychology: Science and Practice, 15 , 125–129.

Gottfredson, G. D., & Gottfredson, D. C. (2001). What schools do to prevent problem behavior and promote safe environments. Journal of Educational and Psychological Consultation, 12 , 313–344.

Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18 , 37–50.

Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38 , 445–459.

Hill, C., Bloom, H., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2 , 172–177.

Humphrey, N. (2013). Social and emotional learning: A critical appraisal . London: Sage.

Humphrey, N., Hanley, T., Lendrum, A., & Wigelsworth, M. (2012). Assessing therapeutic outcomes. In T. Hanley, N. Humphrey, & C. Lennie (Eds.),  Adolescent counselling psychology: Theory, research and practice (pp. 157–170). London: Routledge.

Humphrey, N., Kalambouka, A., Wigelsworth, M., Lendrum, A., Lennie, C., & Farrell, P. (2010). New beginnings: Evaluation of a short social–emotional intervention for primary-aged children. Educational Psychology, 30 , 513–532.

Humphrey, N., Lendrum, A., Barlow, A., Wigelsworth, M., & Squires, G. (2013). Achievement for All: Improving psychosocial outcomes for students with special educational needs and disabilities. Research in Developmental Disabilities, 34 , 1210–1225.

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59 , 12–19.

Kataoka, S., Jaycox, L. H., Wong, M., Nadeem, E., Langley, A., Tang, L., et al. (2011). Effects on school outcomes in low-income minority youth: Preliminary findings from a community-partnered study of a school-based trauma intervention. Ethnicity and Disease, 21 , S1-71–S1-77.

Kataoka, S., Stein, B. D., Jaycox, L. H., Wong, M., Escudero, P., Tu, W., et al. (2003). A school-based mental health program for traumatized Latino immigrant children. Journal of the American Academy of Child and Adolescent Psychiatry, 42 (3), 311–318.

Kazdin, A. E. (1982). Single-case research designs: Methods for clinical and applied settings . New York, NY: Oxford University Press.

Kazdin, A. E. (2005). Evidence-based assessment for children and adolescents: Issues in measurement development and clinical application. Journal of Clinical Child and Adolescent Psychology, 34 , 548–558.

Kendall, P. C., & Grove, W. M. (1988). Normative comparisons in therapy outcome. Psychological Assessment, 10 , 147–158.

Kerry, S., & Bland, J. (1998). Statistics notes: Sample size in cluster randomisation. British Medical Journal, 316 , 549.

Killip, S., Mahfoud, Z., & Pearce, K. (2004). What is an intracluster correlation coefficient? Crucial concepts for researchers. The Annals of Family Medicine, 3 , 204–208.

Kolko, D. J., Bukstein, O. G., & Barron, J. (1999). Methylphenidate and behavior modification in children with ADHD and comorbid ODD and CD: Main and incremental effects across settings. Journal of the American Academy of Child and Adolescent Psychiatry, 38 , 578–586.

Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., et al. (2010). Single-case designs technical documentation. Retrieved from: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf .

Lane, K. L., Beebe-Frankenberger, M. E., Lambros, K. M., & Pierson, M. (2001). Designing effective interventions for children at risk for antisocial behavior: An integrated model of components necessary for making valid inferences. Psychology in the Schools, 38 , 365–379.

Langberg, J. M., Vaughn, A. J., Williamson, P., Epstein, J. N., Girio-Herrera, E., & Becker, S. P. (2011). Refinement of an organizational skills intervention for adolescents with ADHD for implementation by school mental health providers. School Mental Health, 3 , 143–155.

Lendrum, A., & Humphrey, N. (2012). The importance of studying the implementation of school-based interventions. Oxford Review of Education, 38 , 635–652.

Lever, N. A., Adelsheim, S., Prodente, C., Christodulu, K. V., Ambrose, M. G., Schlitt, J., et al. (2003). System, agency and stakeholder collaboration to advance mental health programs in schools. In M. D. Weist, S. W. Evans, & N. A. Lever (Eds.), Handbook of school mental health: Advancing practice and research (pp. 149–162). New York, NY: Springer.

Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., et al. (2012). Translating the statistical representation of effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, D.C.: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.

Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998). Empirically supported psychosocial interventions for children: An overview. Journal of Clinical Child Psychology, 27 , 138–145.

Losen, D. J., & Gillespie, J. (2012, August). Opportunities suspended: The disparate impact of disciplinary exclusion from school . University of California, Los Angeles: The Civil Rights Project. Retrieved from http://civilrightsproject.ucla.edu/resources/projects/center-for-civil-rights-remedies/school-to-prison-folder/federal-reports/upcoming-ccrr-research .

Martens, B. K. (1993). Social labeling, precision of measurement, and problem-solving: Key issues in the assessment of children’s emotional problems. School Psychology Review, 22 , 308–312.

Mash, E. J., & Hunsley, J. (2005). Evidence-based assessment of child and adolescent disorders: Issues and challenges. Journal of Clinical Child and Adolescent Psychology, 34 , 362–379.

Miller, F. G., Johnson, A. H., Welsh, M., Chafouleas, S. M., Riley-Tillman, T. C., & Fabiano, G. (2013, July). Evaluation of universal screening methods to identify behavioral risk . In Poster presented at the meeting of the American Psychological Association, Honolulu, Hawaii.

Miller, F. G., Neugebauer, S. R., Chafouleas, S. M., Briesch, A. M., & Riley-Tillman, T. C. (2013, July). Examining innovation usage: Construct validation of the Usage Rating Profile - Assessment . In Poster presented at the meeting of the American Psychological Association, Honolulu, Hawaii.

Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11 , 247–266.

Muijs, D. (2010). Doing quantitative research in education with SPSS (2nd ed.). London: Sage.

Murphy, S. A. (2005). An experimental design for the development of adaptive treatment strategies. Statistics in Medicine, 24 , 1455–1481.

Muthén, B., Brown, C. H., Masyn, K., Jo, B., Khoo, S.-T., Yang, C.-C., et al. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3 , 459–475.

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, E. M., Fabiano, G. A., et al. (2012a). Q-learning: A data analysis method for constructing adaptive interventions. Psychological Methods, 17 , 478–494.

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W. E., Gnagy, E. M., Fabiano, G. A., et al. (2012b). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17 , 457–477.

Nelson, J. R., Benner, G. J., Reid, R. C., Epstein, M. H., & Currin, D. (2002). The convergent validity of office discipline referrals with the CBCL-TRF. Journal of Emotional and Behavioral Disorders, 10 , 181–188.

O’Connell, M., Boat, T., & Warner, K. E. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities . Washington, DC: The National Academies Press.

O’Leary, K. D., & O’Leary, S. G. (1982). Classroom management: The successful use of behavior modification . New York, NY: Pergamon Press Inc.

Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31 , 109–118.

Pelham, W. E., Carlson, C., Sams, S. E., Vallano, G., Dixon, J., & Hoza, B. (1993). Separate and combined effects of methylphenidate and behavior modification on boys with ADHD in the classroom. Journal of Consulting and Clinical Psychology, 61 , 506–515.

Pelham, W. E., Fabiano, G. A., & Massetti, G. M. (2005). Evidence-based assessment of attention-deficit/hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34 , 449–476.

Pfiffner, L. J., Kaiser, N. M., Burner, C., Zalecki, C., Rooney, M., Setty, P., et al. (2011). From clinic to school: Translating a collaborative school–home behavioral intervention for ADHD. School Mental Health, 3 , 127–142.

Pfiffner, L. J., & O’Leary, S. G. (1987). The efficacy of all-positive management as a function of the prior use of negative consequences. Journal of Applied Behavior Analysis, 20 , 265–271.

Pfiffner, L. J., Rosen, L. A., & O’Leary, S. G. (1985). The efficacy of an all-positive approach to classroom management. Journal of Applied Behavior Analysis, 18 , 257–261.

Power, T. J., Blom-Hoffman, J., Clarke, A. T., Riley-Tillman, T. C., Kelleher, C., & Manz, P. H. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42 , 495–507.

Power, T. J., Mautone, J. A., Soffer, S. L., Clarke, A. T., Marshall, S. A., Sharman, J., et al. (2012). A family–school intervention for children with ADHD: Results of a randomized clinical trial. Journal of Consulting and Clinical Psychology, 80 , 611–623.

Puffer, S., Torgerson, D. J., & Watson, J. (2005). Cluster randomized controlled trials. Journal of Evaluation in Clinical Practice, 11 , 479–483.

Raudenbusch, S. W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., et al. (2011). Optimal design plus empirical evidence ( version 3.0). Retrieved from:  http://www.wtgrantfoundation.org/resources/consultation-service-and-optimal-design .

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models . Thousands Oaks, CA: Sage.

Riley-Tillman, T. R., & Burns, M. K. (2009). Evaluating educational interventions: Single-case designs for measuring response to intervention . New York, NY: The Guilford Press.

Robb, J. A., Sibley, M. H., Pelham, W. E., Foster, E. M., Molina, B. S. G., Gnagy, E. M., et al. (2011). The estimated annual cost of ADHD to the U.S. education system. School Mental Health, 3 , 169–177.

Schnoes, C., Reid, R., Wagner, M., & Marder, C. (2006). ADHD among students receiving special education services: A national survey. Exceptional Children, 72 , 483–496.

Schulte, S. C., Easton, J. E., & Parker, J. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38 , 460–475.

Sellström, E., & Bremberg, S. (2006). Is there a “school effect” on pupil outcomes? A review of multilevel studies. Journal of Epidemiology and Community Health, 60 , 149–155.

Shernoff, E. S., Marinez-Lora, A. M., Frazier, S. L., Jakobsons, L. J., Atkins, M. S., & Bonner, D. (2011). Teachers supporting teachers in urban schools: What iterative research designs can teach us. School Psychology Review, 40 , 465–485.

PubMed Central   PubMed   Google Scholar  

Skiba, R. J., Horner, R. H., Chung, C. G., Rausch, M. K., May, S. L., & Tobin, T. (2011). Race is not neutral: A national investigation of African American and Latino disproportionality in school discipline. School Psychology Review, 40 , 85–107.

Social and Character Development Research Consortium. (2010). Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary school children (NCER 2011–2001). Washington, D.C.: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education.

Spoth, R., & Redmond, C. (1993). Identifying program preferences through conjoint analysis: Illustrative results from a parent sample. American Journal of Health Promotion, 8 , 124–133.

Stein, B. D., Jaycox, L. H., Kataoka, S. H., Wong, M., Tu, W., Elliott, M. N., et al. (2003). A mental health intervention for schoolchildren exposed to violence: A randomized controlled trial. Journal of the American Medical Association, 290 (5), 603–611.

Sugai, G., Horner, R., & Walker, H. (2000). Preventing school violence: The use of office referral to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8 , 94–101.

U.S. Department of Education, Institute of Education Sciences, & What Works Clearinghouse. (2012, June). WWC review of report: Enhancing the effectiveness of special education programming for children with attention - deficit/hyperactivity disorder using a daily report card. Retrieved from http://whatworks.ed.gov .

Volpe, R., & Fabiano, G. A. (2013). Daily behavior report cards: An evidence-based system of assessment and intervention . New York: The Guilford Press.

Wagner, M., Kutash, K., Duchnowski, A. J., Epstein, M. H., & Sumi, W. C. (2005). The children and youth we serve: A national picture of the characteristics of students with emotional disturbances receiving special education. Journal of Emotional and Behavioral Disorders, 13 , 79–96.

Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61 , 620–630.

Weist, M. D., Lever, N., Bradshaw, C., & Owens, J. S. (2014). Further advancing the field of school mental health. In M. Weist, N. Lever, C. Bradshaw, & J. S. Owes (Eds.), Handbook of school mental health: Research, training, practice, and policy (pp. 1–16). New York: Springer.

Weist, M. D., Lever, N., Stephan, S., Youngstrom, E., Moore, E., Harrison, B., et al. (2009). Formative evaluation of a framework for high quality, evidence-based services in school mental health. School Mental Health., 1 (3), 196–211.

Weisz, J. R., Thurber, C. A., Sweeney, L., Proffitt, V. D., & LeGagnoux, G. L. (1997). Brief treatment of mild-to-moderate child depression using primary and secondary control enhancement training. Journal of Consulting and Clinical Psychology, 65 , 703–707.

What Works Clearinghouse. (2010). What Works Clearinghouse: Procedures and standards handbook ( version 2.1). Retrieved from: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf .

What Works Clearinghouse. (2013). What Works Clearinghouse: Procedures and standards handbook ( version 3.0). Retrieved from: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_handbook.pdf .

Wigelsworth, M., Humphrey, N., Kalambouka, A., & Lendrum, A. (2010). A review of key issues in the measurement of children’s social and emotional skills. Educational Psychology in Practice, 26 , 173–186.

Download references

Author information

Authors and affiliations.

Department of Counseling, School and Educational Psychology, University at Buffalo, SUNY, Buffalo, NY, 14260, USA

Gregory A. Fabiano

Department of Educational Psychology, Neag School of Education, University of Connecticut, Storrs, CT, USA

Sandra M. Chafouleas

Department of Psychology, University of South Carolina, Columbia, SC, USA

Mark D. Weist

SRI International, Menlo Park, CA, USA

W. Carl Sumi

University of Manchester, Manchester, UK

Neil Humphrey

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Gregory A. Fabiano .

Rights and permissions

Reprints and permissions

About this article

Fabiano, G.A., Chafouleas, S.M., Weist, M.D. et al. Methodology Considerations in School Mental Health Research. School Mental Health 6 , 68–83 (2014). https://doi.org/10.1007/s12310-013-9117-1

Download citation

Published : 04 January 2014

Issue Date : June 2014

DOI : https://doi.org/10.1007/s12310-013-9117-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• School mental health
• Research methods
• Research design
• Outcome measurement
• Find a journal
• Publish with us
• Track your research

Stanford University

Transforming Mental Health Implementation Research

The Lancet Psychiatry Commission has released a report that examines the effective approaches to prevent and treat mental illness, particularly in parts of the world where marginalized communities could most benefit from evidence-based interventions.

“Too often, research produces interventions and implementation strategies that are difficult to scale owing to misalignment with the political, cultural, policy, system, community, provider, and individual realities of real-world settings,” reads the executive summary of the commission report. “Most mental health implementation research has been done in high-income countries, but the Commission’s recommendations incorporate research from low-income and middle-income countries and call for strategies to expand mental health implementation research globally.”

More than 970 million worldwide live with a mental illness worldwide—almost one in eight people—and nearly 18% of the overall global burden of disease is attributable to mental health conditions. Yet mental health has historically been set apart from the public health and health care sectors and many interventions are not fully tested in the real world.

“So much of what we know works doesn’t work in practice because it’s not implemented or scaled well,” said Sara Singer , PhD, a professor of health policy at Stanford Health Policy and a member of the Lancet commission. “To do better, we need research that informs how to close these knowing-doing gaps. I’m excited about this Lancet Psychiatry Commission report because it provides a clear path for doing better on mental health, an area of critical need in the US and globally.”

The commission members call for the following strategies: 

• Replace the research-to-implementation pathway with an integrated approach.
• Embed equity in mental health intervention and implementation research.
• Approach the implementation gap with a complexity science lens.
• Expand the use of non-experimental approaches to establish causality.

Use a transdisciplinary approach to generate actionable knowledge to close the mental health implementation gap.

See the Full Report & Recommendations

This paper is in the following e-collection/theme issue:

Published on 29.3.2024 in Vol 13 (2024)

Design of a Remote Multiparametric Tool to Assess Mental Well-Being and Distress in Young People (mHealth Methods in Mental Health Research Project): Protocol for an Observational Study

Authors of this article:

• Thais Castro Ribeiro 1, 2 , PhD   ; 
• Esther García Pagès 1, 2 , MSc   ; 
• Laura Ballester 3, 4 , PhD   ; 
• Gemma Vilagut 3, 4 , PhD   ; 
• Helena García Mieres 3, 4 , PhD   ; 
• Víctor Suárez Aragonès 5 , MSc   ; 
• Franco Amigo 3, 4 , MSc   ; 
• Raquel Bailón 1, 6 , PhD   ; 
• Philippe Mortier 3, 4 , PhD   ; 
• Víctor Pérez Sola 7, 8, 9, 10 , PhD   ; 
• Antoni Serrano-Blanco 3, 11 , PhD   ; 
• Jordi Alonso 3, 4, 10 , PhD   ; 
• Jordi Aguiló 1, 2 , PhD  

1 CIBER de Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Instituto de Salud Carlos III, Madrid, Spain

2 Departament of Microelectronics and Electronic Systems, Autonomous University of Barcelona, Bellaterra, Spain

3 CIBER de Epidemiología y Salud Pública (CIBERESP), Instituto de Salud Carlos III, Madrid, Spain

4 Health Services Research Group, Hospital del Mar Research Institute, Barcelona, Spain

5 Department of Clinical Psychology and Psychobiology, University of Barcelona, Barcelona, Spain

6 Aragón Institute of Engineering Research (I3A), University of Zaragoza, Zaragoza, Spain

7 CIBER en Salud Mental (CIBERSAM), Instituto de Salud Carlos III, Madrid, Spain

8 Institute of Neuropsychiatry and Addictions (INAD), Parc de Salut Mar (PSMAR), Barcelona, Spain

9 Neurosciences Research Group, Hospital del Mar Research Institute, Barcelona, Spain

10 Department of Medicine and Life Sciences, Universitat Pompeu Fabra, Barcelona, Spain

11 Institut de Recerca Sant Joan de Déu, Parc Sanitari Sant Joan de Déu, Barcelona, Spain

Corresponding Author:

Thais Castro Ribeiro, PhD

CIBER de Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN)

Instituto de Salud Carlos III

Calle Monforte de Lemos, 3-5, Pabellón 11

Madrid, 28029

Phone: 34 935868430

Email: [email protected]

Background: Mental health conditions have become a substantial cause of disability worldwide, resulting in economic burden and strain on the public health system. Incorporating cognitive and physiological biomarkers using noninvasive sensors combined with self-reported questionnaires can provide a more accurate characterization of the individual’s well-being. Biomarkers such as heart rate variability or those extracted from the electrodermal activity signal are commonly considered as indices of autonomic nervous system functioning, providing objective indicators of stress response. A model combining a set of these biomarkers can constitute a comprehensive tool to remotely assess mental well-being and distress.

Objective: This study aims to design and validate a remote multiparametric tool, including physiological and cognitive variables, to objectively assess mental well-being and distress.

Methods: This ongoing observational study pursues to enroll 60 young participants (aged 18-34 years) in 3 groups, including participants with high mental well-being, participants with mild to moderate psychological distress, and participants diagnosed with depression or anxiety disorder. The inclusion and exclusion criteria are being evaluated through a web-based questionnaire, and for those with a mental health condition, the criteria are identified by psychologists. The assessment consists of collecting mental health self-reported measures and physiological data during a baseline state, the Stroop Color and Word Test as a stress-inducing stage, and a final recovery period. Several variables related to heart rate variability, pulse arrival time, breathing, electrodermal activity, and peripheral temperature are collected using medical and wearable devices. A second assessment is carried out after 1 month. The assessment tool will be developed using self-reported questionnaires assessing well-being (short version of Warwick-Edinburgh Mental Well-being Scale), anxiety (Generalized Anxiety Disorder-7), and depression (Patient Health Questionnaire-9) as the reference. We will perform correlation and principal component analysis to reduce the number of variables, followed by the calculation of multiple regression models. Test-retest reliability, known-group validity, and predictive validity will be assessed.

Results: Participant recruitment is being carried out on a university campus and in mental health services. Recruitment commenced in October 2022 and is expected to be completed by June 2024. As of July 2023, we have recruited 41 participants. Most participants correspond to the group with mild to moderate psychological distress (n=20, 49%), followed by the high mental well-being group (n=13, 32%) and those diagnosed with a mental health condition (n=8, 20%). Data preprocessing is currently ongoing, and publication of the first results is expected by September 2024.

Conclusions: This study will establish an initial framework for a comprehensive mental health assessment tool, taking measurements from sophisticated devices, with the goal of progressing toward a remotely accessible and objectively measured approach that maintains an acceptable level of accuracy in clinical practice and epidemiological studies.

Trial Registration: OSF Registries N3GCH; https://doi.org/10.17605/OSF.IO/N3GCH

International Registered Report Identifier (IRRID): DERR1-10.2196/51298

Introduction

Mental health conditions are one of the leading causes of disability worldwide and are estimated to reduce life expectancy by 10 years [ 1 ]. For instance, depressive disorders were considered the eighth cause of disability in Spain in 2000, rising to fifth in 2019 [ 2 ]. Depression, anxiety, and stress-related disorders impose a major economic impact and burden on the public health system. To date, the prevalence of depression in the Spanish population is close to 5%, and the annual cost is estimated at €6145 (US $6648) million [ 3 ].

Young people and university students are populations of particular interest. Approximately 75% of mental health conditions have an early onset before the age of 24 years, and several risk factors (including genetic, early life adversity, family, community, and environmental factors) are involved in the development and course of these conditions [ 4 ]. Moreover, the recent COVID-19 pandemic has aggravated this situation [ 5 ]. Several systematic reviews and meta-analyses have indicated a high prevalence of mental health conditions among young people, with a pooled prevalence for depression of 31% to 33.6%, anxiety of 28% to 39%, sleep problems of 40%, and suicidal ideation of 12.3% [ 6 - 8 ], in line with the results of longitudinal studies suggesting a possible worsening of mental health in this population in recent years [ 9 ]. There is a need for early identification and prevention of mental health conditions, which includes the design and implementation of mental health promotion activities that lead to an increase in emotional well-being [ 10 ].

In recent decades, interest in mental health research has been steadily increasing, recognizing it a crucial aspect of overall health, rather than simply the absence of related conditions [ 11 ]. According to the World Health Organization [ 12 ], mental health is characterized by individuals’ ability to effectively manage typical stressful situations, develop their potential and skills, and contribute productively to both themselves and the community. This comprises an adequate stress response and recovery as well as maintaining cognitive abilities such as attentional level and proper time of response. Stress reactivity is this capacity to respond to a stressor. It is a disposition that underlies individual differences in response to stressors and is assumed to be a vulnerability factor for the development of mental health conditions [ 13 ]. In this context, monitoring physiological response during a stress-inducing task could yield different reactivity patterns, offering valuable insights to differentiate between mental well-being and distress.

In clinical practice, self-reported questionnaires are commonly used to assess the severity of mental health symptoms, quality of life, and mental well-being. Nevertheless, several studies have reported limitations of these tests related to memory biases and distortions in retrospective recall [ 14 , 15 ]. To expand such assessments, including physiological biomarkers information will improve the characterization of endophenotypes (research domain criteria) [ 16 ]. Owing to technological advances, small sensors can measure physiological data for behavioral health, interventions, and outcomes (digital phenotyping) [ 17 , 18 ]. Given the significance of stress reactions as complex phenomena encompassing psychological, cognitive, and physiological reactions involving the autonomic nervous system (ANS) and the neuroendocrine system, which, in turn, can affect other bodily systems, exploring these dynamics could enhance our comprehension of mental distress. Hence, physiological data monitoring including a stress-eliciting task may have an important role in early detection and intervention in mental health care. Heart rate variability (HRV), pulse arrival time (PAT), breathing parameters, electrodermal activity (EDA), and skin temperature (ST) are physiological variables broadly used to study the stress response and gather information about ANS functioning [ 19 - 24 ].

To progress in this field, the use of wearables in mental health research shows promise, offering increased accuracy in data collection and reduced participant burden. Wearable devices allow researchers to passively monitor individuals in real time and gather data outside of traditional laboratory settings, that is, along with everyday life situations, providing a more holistic understanding of mental health status [ 25 , 26 ]. Currently, many studies on stress detection are conducted in controlled environments because accuracy decreases when conducted in real-time environments [ 27 ]. In addition, different instruments to measure perceived stress are used, which hinders the comparability of results [ 28 ], or a small number of signals are usually collected [ 29 - 31 ]. However, previous studies have shown optimistic results for further advancement in the field for the objective assessment of mental health status and stress. A study analyzing data from 510 participants wearing a Fitbit device during a 2-year follow-up [ 32 ] showed a correlation between decreased resting heart rate variation during the day and the severity of depression, whereas the mean heart rate at night was higher in participants with more severe depressive symptoms. In line with these results, a decreased autonomic reactivity measured through dynamic changes in photoplethysmography (PPG) waveform morphology was associated with a higher degree of depression in the study by Kontaxis et al [ 33 ]. Sano et al [ 30 ] conducted an observational study among university students using wearable sensors that collected EDA and ST, and using psychometric questionnaires as reference, they found an accuracy of 78% and 87% to classify into high or low stress groups and high or low mental health groups, respectively. Similarly, Sano et al [ 34 ] found an accuracy of 90% in classifying stress and mental health groups. From a literature review [ 27 ], it was observed that heart rate and EDA are the most regularly used sensory signals, offering the most promising results and high accuracy for detecting stress.

Effective prevention interventions require strategies to identify early risk groups according to risk factors through the development of predictive models. In addition, from a mental health promotion perspective, effectively assessing mental well-being would help identify the right time to intervene, evaluate the efficacy of the therapy applied, empower the citizens, offer stress-reducing programs, and prevent negative consequences. Here, we present the development and evaluation of a novel multiparametric tool to improve mental health assessments and to facilitate the evaluation of risk and protective factors as well as the effectiveness of promotion and prevention interventions.

This study aims to design and validate a remote multiparametric tool, including several physiological and cognitive variables, to objectively assess mental well-being as well as mental distress (ie, symptoms of depression and anxiety) among young people for epidemiologic and clinical studies.

The specific objectives of this study are (1) to develop an assessment tool for mental well-being and distress based on the most relevant physiological and cognitive variables; (2) to validate the assessment tool using self-reported measures and evaluate the tool reliability and accuracy; and (3) to develop and establish a protocol to automate the measurement process, ensuring that it can be reproduced in large populations.

Study Design and Setting

This is a multicenter observational study of the mHealth Methods in Mental Health Research (M&M) project, currently ongoing, being conducted by the Autonomous University of Barcelona (UAB) and Parc de Salut Mar (PSMAR).

Three different mental health states will be studied: (1) high mental well-being, (2) presenting mild to moderate psychological distress, and (3) depressive or anxiety disorder (diagnosed by a mental health professional). For the high mental well-being and the mild to moderate psychological distress groups, a web-based mental health questionnaire is being distributed among UAB students for screening and analyzed to determine the participant’s eligibility. Participants who meet the selection criteria are consecutively included. For the mental health condition group, the patients are being referred from the Institute of Neuropsychiatry and Addictions-PSMAR, the Hospital Sant Joan de Déu, and the Psychology and Speech Therapy Service of the UAB. The assessments are planned at 2 time points and are being conducted at the site of recruitment (UAB, Institute of Neuropsychiatry and Addictions-PSMAR, or Hospital Sant Joan de Déu). The second assessment takes place after 1 month of the first assessment.

Participants and Eligibility Criteria

The 3 abovementioned participant groups are being recruited according to the inclusion and exclusion criteria described in detail in Table 1 . To ensure a homogeneous sample in terms of age, participants aged between 18 and 34 years are being recruited in all 3 groups.

a PHQ-4: Patient Health Questionnaire-4.

b SWEMWBS: short version of Warwick-Edinburgh Mental Well-being Scale.

c UAB: Autonomous University of Barcelona.

d INAD-PSMAR: Institute of Neuropsychiatry and Addictions-Parc de Salut Mar.

e HSJD: Hospital Sant Joan de Déu.

For the high mental well-being and mild to moderate psychological distress groups, inclusion criteria are assessed for eligibility through a web-based questionnaire that contains questions about mental health history (eg, “Have you ever experienced any mental health issue?”). The Patient Health Questionnaire (PHQ; PHQ-4) [ 35 ] is used to screen for anxiety and depression symptoms, and the short version of Warwick-Edinburgh Mental Well-being Scale (SWEMWBS) [ 36 , 37 ] is used to evaluate mental well-being. The cutoff points to be considered as high mental well-being are based on data from a representative sample of young adults of Catalonia from the Catalonia Health Survey conducted in 2016 [ 38 ], in which a median score of 30 points in SWEMWBS was found. Individuals with SWEMWBS well-being score ≥30 and PHQ-4 <3 points are classified in the high well-being group. Individuals with SWEMWBS score between 20 and 29 points or a PHQ-4 score between 3 and 8 are classified into the mild to moderate psychological distress group.

Recruitment

The primary recruitment pathway for nonpatients is the dissemination of the study through institutional mail or social media and the placement of posters in public areas of UAB. The information includes a link or QR code to answer a web-based questionnaire. To facilitate the recruitment of the mild to moderate psychological distress group, the Psychology and Speech Therapy Service of the UAB is collaborating by inviting students who attended the service to participate in this study. In both cases, once the responsible researcher confirms the eligibility criteria, the participant is contacted to schedule the first assessment.

For the mental health condition group, the patients who meet the criteria are identified at the consultation with the psychologist or psychiatrist, who briefly informs them about the study and suggests participation. The research assistant contacts the interested patients by phone and makes an appointment for the first assessment. Written informed consent is provided by all participants before starting the first assessment interview.

Study Procedure

All participants who agree to participate are asked to abstain from tobacco, alcohol, caffeine, or any other beverage or stimulating substance for 2 hours before the study. Figure 1 shows the complete schematic of the experimental procedure.

At the first assessment, the participants are fully informed about the study procedure and are requested to sign the informed consent form. This visit includes an ad hoc interview conducted by a qualified examiner to collect individuals’ sociodemographic, modifiable lifestyle factors, health-related variables, and clinical data through the management software Qualtrics (Silver Lake). Subsequently, a psychological assessment is carried out. The participants respond to 7 self-reported questionnaires using the same software. These questionnaires aim to estimate the current mental well-being, stress perception, symptoms of anxiety and depression, physical activity, sleep quality, and substance use. All these measures are described in the Study Variables section.

The physiological assessment consists of recording different stress-related physiological signals using (1) the medical-graded device NeXus-10 MKII (Mind Media BV) and (2) the wearable E4 Empatica wristband (Empatica Inc). PPG, EDA, and ST will be the physiological signals recorded simultaneously by both devices. The electrocardiogram (ECG) and respiration can only be measured using a medical-graded device. This device is used to obtain a more accurate measure for preliminary analysis and, thereafter, validate the predictive model with the wearable device.

The wristband is placed on the nondominant wrist and the PPG (middle finger), EDA (middle phalanges of the second and fourth digits), and ST (fingertip of the fifth finger) sensors are placed on the nondominant hand to avoid excessive movement artifacts. An adjustable elastic band is placed over the abdomen to measure the respiration signal. For lead 1 of the ECG signal, electrodes are positioned below the right collarbone and below the left rib cage, whereas for lead 2, electrodes are positioned on the fifth intercostal space along the midaxillary line on the left side and symmetrically on the right side. The reference electrode is placed on the left collarbone.

This part of the procedure lasts approximately 15 minutes and is divided into three different stages: (1) baseline (green block in Figure 1 ): participant in a resting state, sitting comfortably with eyes open; (2) cognitive task (red block in Figure 1 ): corresponds to the stress-inducing stage, when the individual is submitted to a cognitive task, the Stroop Test [ 39 ]; and (3) recovery (yellow block in Figure 1 ): when the individual’s physiological responses are expected to return to the baseline levels. All physiological signals and variables of interest will be detailed in the Physiological Variables section.

A second assessment is then scheduled 1 month apart and includes the same psychological and physiological assessments. This follow-up session is intended to allow test-retest reliability and account for random errors that could occur in a single session.

Study Variables

Outcome measures.

The following outcome measures are used:

• Depression: It is evaluated using the PHQ-9 [ 40 , 41 ]. It is a Likert-type scale used to screen the severity of depressive symptoms according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition . All 9 items are rated from 0 (not at all) to 3 (nearly every day). Total scores can range from 0 to 27, with higher scores indicating more severe depression. Furthermore, 5, 10, 15, and 20 represent the cutoff points for mild, moderate, moderately severe, and severe depression, respectively [ 42 ].
• Anxiety: The Generalized Anxiety Disorder-7 [ 43 , 44 ] is an instrument for screening the presence of symptoms of anxiety as listed in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition . It is a 1-dimensional scale with scores for all 7 items ranging from 0 (not at all) to 3 (nearly every day). The total score was categorized into 4 severity groups according to the original authors: minimal (0-4), mild (5-9), moderate (10-14), and severe (>15).
• Mental well-being: The SWEMWBS [ 36 , 37 ] is an instrument used to assess mental well-being. This unidimensional scale comprises 7 items ranging from 1 (none of the time) to 5 (all of the time). The higher the total score, the greater the perception of well-being.

Sociodemographic variables include age, gender, nationality, marital status, living status, educational level, and occupation.

Current physical and mental health conditions, previous mental health treatments, medication, and suicidal thoughts and behaviors are evaluated through items in the ad hoc interview. Perceived stress is evaluated using the 10-item Perceived Stress Scale [ 45 , 46 ]. It is a 5-point Likert scale with questions about the frequency of feelings and thoughts during the last month, with each item ranging from 0 (never) to 4 (very often). Higher scores indicate higher levels of perceived stress. To assess substance use problems, the CAGE-Adapted to Include Drugs [ 47 ] scale is used. It is an adaptation of the original CAGE questionnaire [ 48 ] for conjointly screening for alcohol and drug problems based on lifetime. The scale contains 4 yes or no questions, and a higher score indicates substance use problems.

Modifiable lifestyle variables are assessed using an ad hoc interview, including coffee consumption, cigarette smoking, alcohol and drug consumption, and BMI. Physical activity is measured using the short form of the International Physical Activity Questionnaire [ 49 , 50 ]. This questionnaire comprises 7 open-ended questions about individuals’ 7-day recall of physical activity. According to the total energy expenditure in metabolic equivalent of task (ie, 1 metabolic equivalent of task is the energy cost of sitting quietly) in minutes per week, the physical activity level is determined as low or inactive, moderate, or high.

Sleep is evaluated using the Medical Outcomes Study Sleep Scale [ 51 , 52 ]. This questionnaire contains 12 items about a 4-week recall, divided into 8 subscales (sleep adequacy, optimal sleep, quantity of sleep, awakening shortness of breath or with headache, snoring, sleep disturbance, somnolence, and global index of sleep interference). In general, higher scores indicate greater sleep problems. The quantity of sleep score is the patient-reported number of hours of sleep per night, and optimal sleep is scored as 1 (7 or 8 h of sleep per night) or 0 (any different response).

The comfort level is evaluated by asking participants if they are currently experiencing higher stress than usual and identifying its potential causes.

Stress-Inducing Cognitive Task

The Spanish version of the standard Stroop Color and Word Test (SCWT) is applied (originally [ 53 ] and Spanish version [ 39 ]) as a cognitive stress-inducing task. This test is extensively used to assess cognitive inhibition and processing speed. Furthermore, it has also been shown to be a reliable method to induce mental stress in experimental settings [ 54 ]. The individuals are required to read the 3-color cards as fast as possible in a fixed time of 45 seconds each. The stimuli presented on the first 2 cards are congruent, that is, read names of colors or name different colors. In contrast, the last card represents the incongruent stimuli, that is, name the color of the ink instead of reading the color.

Three direct scores are derived by tallying the correct responses for each condition: (1) W (word) represents the number of colors read on the first card (where colors are written in black ink), (2) C (color) represents the number of elements identified on the card of colors (where name colors are represented with strings of XXXXs), and (3) CW (color word) represents the number of items correctly identified on the third card (where colors are printed in an ink that does not correspond to the color name, requiring participants to say the colors of the ink). Two other scores will be calculated from these: (1) predicted CW ( PCW ): ( W × C / W + C ) and (2) interference: ( CW − PCW ). A higher score indicates a greater ability to inhibit interference.

Direct scores are then converted to T scores, with a preset mean of 50 and SD of 10, so that they can be more easily compared in similar age ranges (in this study, young adults aged between 18 and 44 years). The limits considered normal are between 35 and 65 T points in any of the scores (for details, refer the study by Golden [ 39 ]).

Physiological Variables

We are following a methodology already used and validated as stress assessment for healthy students and principal caregivers [ 20 , 55 ]; the electrophysiological raw signals recorded with NeXus-10 MKII (ie, ECG, PPG, EDA, respiration, and ST) are analyzed using BioSigBrowser [ 56 ] in MATLAB software (The MathWorks Inc); and several groups of variables are extracted, as described in Table 2 . A literature review was conducted to select the most relevant variables for assessing stress response and mental health. A summary of the findings can be found in Multimedia Appendix 1 [ 19 - 24 , 57 - 65 ].

a HRV: heart rate variability.

b PRV: pulse rate variability.

c ECG: electrocardiogram.

d PPG: photoplethysmography.

e HR: heart rate.

f These will also be extracted from the recordings of the Empatica E4 wearable device.

g bpm: beats per minute.

h IBI: interbeat interval.

i VLF: very low frequency

j LF: low frequency.

k HF: high frequency.

l PAT: pulse arrival time.

m RR: respiratory rate.

n EDA: electrodermal activity.

o SCL: skin conductance level.

p SCR: skin conductance response.

q ST: skin temperature.

The raw signals recorded with the Empatica E4 wearable device (ie, EDA, PPG, and ST) will be also analyzed in MATLAB (The MathWorks Inc) using a similar procedure, given the different format files. The variables intended to be explored in this case are indicated in the footnotes in Table 2 . Furthermore, the stress reactivity, that is, the difference between the stress-inducing cognitive task stage and the baseline stage (stress−baseline), and the stress recovery, that is, the difference between the stress and the posterior recovery stage (stress−recovery), will also be computed for each variable to determine the most relevant set of variables to be considered to design the final model.

Physiological Data Processing

For processing the ECG signal, beat detection is performed through a discrete wavelet transform [ 66 ]. Afterward, the existence of ectopic beats or false QRS detections will be verified and corrected using the algorithm reported by Mateo and Laguna [ 67 ] before the computation of the interbeat interval series. Segments of up to 3 interpolated or corrected beats are accepted and assumed to be normal. Following this, the HRV parameters are calculated by a time-domain analysis and a frequency-domain analysis by Fourier transform of the heart rate signal.

PPG signal is preprocessed using a low-pass finite impulse response (FIR) filter with a cutoff frequency of 35 Hz (order 50) and then a high-pass FIR filter with a cutoff frequency of 0.3 Hz (order 5000). PPG artifacts are suppressed using a Hjorth parameter–based PPG artifact detector described by Gil et al [ 68 ]. Pulses are detected from the PPG signal on those time slots without artifacts using an algorithm based on the study by Lázaro et al [ 69 ]. The same ECG parameters are also extracted in PPG, in this case, referred to as pulse rate variability. Subsequently, the mean time difference between the R peak in the ECG signal and the point of 50% increase, corresponding to the pulse detected on the finger by the PPG signal, is considered as the PAT, and its SD (SD of PAT) is also calculated.

The respiration wave is filtered with an FIR passband filter with cutoff frequencies of 0.03 and 0.9 Hz. The respiratory rate is estimated as the frequency to which the maximum peak of the power density spectrum corresponds, estimated using a fast Fourier transform [ 70 ]. When the peak is >65%, then respiratory rate is considered valid.

The EDA signal is visually inspected to remove motion artifacts and linearly interpolated. First, a time-domain analysis is performed using a convex optimization model, called cvxEDA [ 57 ], to calculate the tonic and phasic components. The second procedure is a frequency-domain analysis, proposed to assess sympathetic tone through a parameter named EDASymp, described in the study by Posada-Quintero et al [ 71 ].

Finally, for the ST signal, a visual inspection is carried out to look for possible large artifacts. These segments are discarded before proceeding with the calculation of the parameters.

Statistical Analysis Plan

An initial descriptive analysis will be conducted for all study variables, for the overall sample and stratified by study group. The quantitative variables will be summarized, assuming normal distribution (Shapiro-Wilk normality test), using the mean and SD. The qualitative variables will be summarized using the relative and absolute frequencies. Physiological variables that present a skewed distribution will be logarithmically transformed.

A parametric test (Pearson correlation) or nonparametric test (Spearman correlation) will be, accordingly, applied as a descriptive measure of the association between quantitative variables.

To develop a useful tool for assessing mental distress and well-being, the initial step of the statistical analysis plan will involve variables reduction using two methods: (1) a correlation analysis to prioritize the most relevant variables for the prediction of the primary outcome measures and (2) a principal component analysis to find the directions of maximum variance in the data and reduce collinearity. Subsequently, the generated components that account for at least 85% (51/60) of the sample’s variability will be included. From these components, 2 separate analyses will be conducted. First, a generalized linear model will be fitted to predict the values of each primary outcome measure. Second, a model will be developed to differentiate the 3 groups identified in the study, each representing a different level of mental health. The models will be fitted using standardized variables and performing k-fold cross-validation to quantify the model’s performance using R 2 for the linear regression model and area under the curve for the classification models. Known-group validity will be assessed by comparing the mean scores of the tool among the preestablished groups at baseline: diagnosed with current depression or anxiety, symptoms of mental distress, and high mental well-being. The predefined hypothesis that higher model scores are predicted for individuals with higher well-being will be evaluated using the Jonckheere-Terpstra test, and Cohen effect sizes will be computed for each category as compared with the lowest category (mental health condition), considering small (0.2), moderate (0.5), and large (0.8) effect sizes [ 72 ].

The model test-retest reliability will be assessed with a 2-way random effect intraclass correlation coefficient (ICC), taking repeated evaluations of the same unchanged individuals, to assess the extent to which measures remain stable. A change will be considered a clinically relevant change in the outcome scores, ie, ≥4 points for the Generalized Anxiety Disorder-7, ≥5 points for PHQ-9, or ≥3 points for SWEMWBS) [ 73 - 75 ].

The significance level will be set at α=.05. Statistical analysis will be performed using SAS (version 9.4; SAS Institute).

Sample Size

As a proof-of-concept study, we plan to include 20 individuals per group (a total of 60 individuals×2 evaluations). For the assessment of known-groups validity of the developed tool, with this sample size of 20 individuals per group, and a type I error rate α=.05 on a 1-sided t test, we will have power of 0.80 to detect a difference between 2 groups corresponding to an effect size of 0.8. For a moderate effect size of 0.5, the power will decrease to 0.47 [ 17 ].

Concerning the assessment of test-retest reliability of the tool, assuming a 15% (9/60) loss in follow-up or nonstable participants from baseline participants, an ICC of 0.6 under the null hypothesis, and a type I error rate of α=.05, a sample size of 51 participants with 2 observations per participant achieves a power of 0.90 to detect a hypothetical ICC value of 0.8 under the alternative hypothesis [ 17 ].

Ethical Considerations

This study protocol was approved by the research ethics committees of both institutions (2021/10163 for PSMAR and 5912 for UAB). This study is in line with the principles established by national and international regulations, including the Declaration of Helsinki and the Code of Ethics. Ethics approval has been obtained in Barcelona from the independent PSMAR Clinical Research Ethics Committee and the Research Ethics Committee of the UAB. Informed consent is requested from all participants before their inclusion in the study. Participants are explained that they can withdraw from the study at any time without giving a reason and that they can request to delete all the data collected from them.

All personal data will be handled following Regulation (European Union) 2016/679 of the European Parliament and the Council on the protection of natural persons concerning the processing of personal data and on the free movement of such data and the National Organic Law 3/2018, of December 5, on Personal Data Protection and the Guarantee of Digital Rights. Physiological and psychometric data will be pseudoanonymized to guarantee privacy in data analysis and will be stored in a research database following the General Data Protection Regulation of the European Union.

Participants receive a €10 (US $11) gift card for enrolling in the study, and participants receive another gift card of the same value if they complete the second assessment.

The project was granted in February 2022 and the approval from ethical committees was obtained between April and May 2022. Participant recruitment started in October 2022 and is expected to continue through June 2024. Different recruitment strategies are implemented, including advertising campaigns and invitation letters at the university and recruitment of patients by psychologists.

As of July 2023, a total of 41 participants completed the first and second assessments. The sample corresponds mainly to the group with mild to moderate psychological distress (n=20, 49%), followed by the group with high mental well-being (n=13, 32%) and, finally, those diagnosed with an anxiety disorder (n=8, 20%). At this point, preprocessing and quality checks of the data are ongoing, and the statistical analysis will subsequently begin. The first results are expected to be published in September 2024.

There is a growing interest in remotely assessing mental health through changes in the ANS functioning and its association with mental health and well-being [ 26 ]. There is significant evidence to support the notion that changes in the ANS can be inferred from changes in physiological variables, such as HRV, EDA response, and peripheral temperature [ 58 , 76 , 77 ]. However, there is a challenge in designing a robust predictive model that allows this assessment to be easy and objective to be systematically used in epidemiology and clinical settings. To fill this gap, it is important to carefully measure electrophysiological signals considering the updated standards of measurement (eg, [ 78 ]), following a structure of 3-time point experimental design: considering a basal condition, during a stress-inducing task to investigate the stress reactivity, and a recovery stage. This study is applying a similar methodology used in previous research that reached good reliability in its predictive models [ 20 , 79 ]. However, in this novel approach, individuals with a clinically diagnosed mental health condition are being enrolled, which may allow for a better distinction between different profiles of mental health through the stress reactivity pattern and cognitive performance.

The SCWT is a well-known neuropsychological test reported as a reliable moderate mental stressor, provoking significant physiological changes such as reduced HRV and increased EDA and blood pressure [ 59 , 80 , 81 ]. Furthermore, it is a useful tool for evaluating cognitive processes and has the advantage of not generating a significant learning effect [ 82 ]. Cognitive inhibition is compromised in multiple mental health conditions; for example, individuals with anxiety disorders tend to have longer reaction times and higher error rates on the SCWT, particularly in the incongruent condition, which suggests difficulties with selective attention and response inhibition [ 83 ]. Thus, although the impaired performance of the task may be indicative of an underlying neurological disorder, a good result performance may add another layer of confidence in evaluating mental well-being.

There is a lot of research committed to investigating biomarkers and stress reactivity in patients diagnosed with depression and anxiety disorders [ 15 , 84 ]. The evidence supports differences in physiological behavior in patients with depression and anxiety compared with healthy individuals. Considering this, the decision to include 3 groups with different mental health states will facilitate the detection of patterns that could discriminate them more accurately. The age range selected is justified by the typical early age of onset of mental health conditions and the need for assertive responses to prevent a poor prognosis. Moreover, the recruitment is planned to be carried out in part at a university, considering that university students have been reported as a susceptible population with a higher risk to manifest mental health symptoms [ 85 ], which enables us to recruit the mild to moderate psychological distress group effortlessly. To minimize interindividual variation in physiological measures and the influence of external factors, we propose 2 different sessions with the same participants, which will also increase the statistical power [ 78 ].

Mobile health (mHealth), which includes physical devices, sensors, software, and other technologies, has been proposed to improve clinical care as it enables data collection, symptom monitoring, and provision of interventions [ 29 ]. In this sense, it could be a valuable resource to reach both regular patients and those who do not receive adequate care [ 12 ]. Specifically, the use of wearables provides a new and unobtrusive way to monitor physiological biomarkers and gather continuous information about individuals’ daily lives and clinical symptoms for both clinical and research purposes. These devices with multiple embedded sensors can be useful in following up patients and remotely assessing their mental well-being through robust models that combine a set of relevant biomarkers [ 31 ].

Nevertheless, to ensure that autonomous nervous system biomarkers are effectively used for objectively measuring mental well-being in health services, it is necessary to undertake substantial work in identifying the most useful biomarkers and comprehending the possible obstacles and enablers of widespread adoption. To advance in this direction, this study intends to carry out a preliminary model validation by enrolling a preselected sample with different states of mental well-being in a laboratory-controlled condition; subsequently, we plan to explore its application in larger populations and in a real-life context.

To summarize, our study aims to design a comprehensive multiparametric model combining physiological and cognitive variables to assess the mental well-being among young people. The self-reported questionnaires currently used in clinical settings will be used as a reference to select the best model fit. This novel approach proposes shifting the paradigm to assessing mental well-being rather than measuring the severity of symptoms or a mental health condition. We believe that this change may allow health professionals to properly recommend prevention strategies and increase the possibility of intervening before the diagnosis of a mental health condition. To effectively develop a model that can be easily calculated by a wearable device, we first take a measurement of a standard medical device to ensure the best quality of physiological signal and then establish the final predictive tool.

Conclusions

This study represents the primary phase in developing a comprehensive mental well-being assessment tool. Our goal is to progress toward remote measurements with acceptable accuracy, using sophisticated devices as a benchmark for comparison. Developing a robust predictive model will facilitate objective assessment that can be systematically used in epidemiological and clinical settings. Further research is required to explore the full potential of this technology in mental health research and clinical practice.

Acknowledgments

This study is receiving funding from the Centro de Investigación Biomédica en Red (CIBER) de Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN, Instituto de Salud Carlos III [ISCIII], Madrid, Spain) and the CIBER de Epidemiología y Salud Pública (CIBERESP, ISCIII, Madrid, Spain) for CIBERESP-BBN Collaboration Projects and the Departament de Recerca i Universitats of the Generalitat de Catalunya (AGAUR 2021 SGR 00624). This work is also partially funded by Ministerio de Ciencia e Innovación and European Social Fund, Spain (TED2021-131106B-I00), and the European Social Fund (European Union). PM received funding from the ISCIII-FSE (Fondo Social Europeo; Miguel Servet grant CP21/00078). RB received funding from the BSICoS (Biomedical Signal Interpretation and Computational Simulation) group (T39 23R) by the Aragon Government.

Data Availability

The data sets generated during this study will be deposited in the Open Science Framework public repository.

Authors' Contributions

TCR, EGP, and J Aguiló drafted the first version of this manuscript. LB, J Alonso, and GV provided critical revision of further drafts of the manuscript. J Alonso, J Aguiló, TCR, EGP, HGM, GV, and FA made substantial contributions to the conception and design of the study protocol. RB and PM contributed to the scientific knowledge of the project. VPS and ASB contributed to patients’ referral. TCR, EGP, and VSA were involved with the clinical interviews and psychophysiological assessments. J Alonso and J Aguiló are the principal investigator and coinvestigator of the study, respectively. All authors reviewed the latest version of the paper draft, provided comments, and approved the final manuscript.

Conflicts of Interest

None declared.

Summary of the literature review of the physiological variables assessed in the M&M mHealth Methods in Mental Health Research (M&M) Project

• Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden implications: a systematic review and meta-analysis. JAMA Psychiatry. Apr 2015;72(4):334-341. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Global health estimates. World Health Organization. 2019. URL: https://www.who.int/data/global-health-estimates [accessed 2023-07-11]
• Vieta E, Alonso J, Pérez-Sola V, Roca M, Hernando T, Sicras-Mainar A, et al. Epidemiology and costs of depressive disorder in Spain: the EPICO study. Eur Neuropsychopharmacol. Sep 2021;50:93-103. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fusar-Poli P. Integrated mental health services for the developmental period (0 to 25 years): a critical review of the evidence. Front Psychiatry. Jun 7, 2019;10:355. [ FREE Full text ] [ CrossRef ] [ Medline ]
• de Sousa GM, Tavares VD, de Meiroz Grilo ML, Coelho ML, de Lima-Araújo GL, Schuch FB, et al. Mental health in COVID-19 pandemic: a meta-review of prevalence meta-analyses. Front Psychol. Sep 21, 2021;12:703838. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dragioti E, Li H, Tsitsas G, Lee KH, Choi J, Kim J, et al. A large-scale meta-analytic atlas of mental health problems prevalence during the COVID-19 early pandemic. J Med Virol. May 2022;94(5):1935-1949. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li W, Zhao Z, Chen D, Peng Y, Lu Z. Prevalence and associated factors of depression and anxiety symptoms among college students: a systematic review and meta-analysis. J Child Psychol Psychiatry. Nov 2022;63(11):1222-1230. [ CrossRef ] [ Medline ]
• Farooq S, Tunmore J, Wajid Ali M, Ayub M. Suicide, self-harm and suicidal ideation during COVID-19: a systematic review. Psychiatry Res. Dec 2021;306:114228. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lipson SK, Zhou S, Abelson S, Heinze J, Jirsa M, Morigney J, et al. Trends in college student mental health and help-seeking by race/ethnicity: findings from the national healthy minds study, 2013-2021. J Affect Disord. Jun 01, 2022;306:138-147. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Solmi M, Radua J, Olivola M, Croce E, Soardo L, Salazar de Pablo G, et al. Age at onset of mental disorders worldwide: large-scale meta-analysis of 192 epidemiological studies. Mol Psychiatry. Jan 2022;27(1):281-295. [ FREE Full text ] [ CrossRef ] [ Medline ]
• World Health Organization. The World health report : 2001 : mental health : new understanding, new hope. World Health Organization. 2001. URL: https://iris.who.int/handle/10665/42390 [accessed 2021-11-15]
• Promoting mental health: concepts, emerging evidence, practice. World Health Organization. Jan 01, 2005. URL: https://www.who.int/publications/i/item/9241562943 [accessed 2021-11-15]
• Kiecolt-Glaser JK, Renna ME, Shrout MR, Madison AA. Stress reactivity: what pushes us higher, faster, and longer - and why it matters. Curr Dir Psychol Sci. Oct 01, 2020;29(5):492-498. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rodrigues S, Kaiseler M, Queirós C. Psychophysiological assessment of stress under ecological settings. Eur Psychol. Jul 2015;20(3):204-226. [ CrossRef ]
• Ahmed T, Qassem M, Kyriacou PA. Physiological monitoring of stress and major depression: a review of the current monitoring techniques and considerations for the future. Biomed Signal Process Control. May 2022;75:103591. [ CrossRef ]
• Cuthbert BN. Research domain criteria (RDoC): progress and potential. Curr Dir Psychol Sci. Apr 2022;31(2):107-114. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Visintainer PF. SAMPICC: stata module to compute sample size for an intra-class correlation (ICC). Boston College Department of Economics. 2008. URL: https://ideas.repec.org/c/boc/bocode/s456978.html [accessed 2023-12-14]
• Insel TR. Digital phenotyping: technology for a new science of behavior. JAMA. Oct 03, 2017;318(13):1215-1216. [ CrossRef ] [ Medline ]
• Kim HG, Cheon EJ, Bai DS, Lee YH, Koo BH. Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investig. Mar 2018;15(3):235-245. [ FREE Full text ] [ CrossRef ] [ Medline ]
• García Pagès E, Arza A, Lazaro J, Puig C, Castro T, Ottaviano M, et al. Psychosomatic response to acute emotional stress in healthy students. Front Physiol. Jan 9, 2023;13:960118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tipton MJ, Harper A, Paton JF, Costello JT. The human ventilatory response to stress: rate or depth? J Physiol. Sep 01, 2017;595(17):5729-5752. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ghiasi S, Greco A, Barbieri R, Scilingo EP, Valenza G. Assessing autonomic function from electrodermal activity and heart rate variability during cold-pressor test and emotional challenge. Sci Rep. Mar 25, 2020;10(1):5406. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vinkers CH, Penning R, Hellhammer J, Verster JC, Klaessens JH, Olivier B, et al. The effect of stress on core and peripheral body temperature in humans. Stress. Sep 2013;16(5):520-530. [ CrossRef ] [ Medline ]
• Pisanski K, Kobylarek A, Jakubowska L, Nowak J, Walter A, Błaszczyński K, et al. Multimodal stress detection: testing for covariation in vocal, hormonal and physiological responses to trier social stress test. Horm Behav. Nov 2018;106:52-61. [ CrossRef ] [ Medline ]
• Matcham F, Leightley D, Siddi S, Lamers F, White KM, Annas P, et al. Remote Assessment of Disease and Relapse in Major Depressive Disorder (RADAR-MDD): recruitment, retention, and data availability in a longitudinal remote measurement study. BMC Psychiatry. Feb 21, 2022;22(1):136. [ FREE Full text ] [ CrossRef ] [ Medline ]
• W Adams Z, McClure EA, Gray KM, Danielson CK, Treiber FA, Ruggiero KJ. Mobile devices for the remote acquisition of physiological and behavioral biomarkers in psychiatric clinical research. J Psychiatr Res. Feb 2017;85:1-14. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gedam S, Paul S. A review on mental stress detection using wearable sensors and machine learning techniques. IEEE Access. Jun 02, 2021;9:84045-84066. [ CrossRef ]
• Klimek A, Mannheim I, Schouten G, Wouters EJ, Peeters MW. Wearables measuring electrodermal activity to assess perceived stress in care: a scoping review. Acta Neuropsychiatr. Mar 24, 2023.:1-11. [ CrossRef ] [ Medline ]
• Gaskin J, Jenkins J, Meservy T, Steffen J, Payne K. Using wearable devices for non-invasive, inexpensive physiological data collection. In: Proceedings of the 50th Hawaii International Conference on System Sciences. 2017. Presented at: HICSS 2017; January 4-7, 2017; Hilton Waikoloa Village, Hawaii. [ CrossRef ]
• Sano A, Taylor S, McHill AW, Phillips AJ, Barger LK, Klerman E, et al. Identifying objective physiological markers and modifiable behaviors for self-reported stress and mental health status using wearable sensors and mobile phones: observational study. J Med Internet Res. Jun 08, 2018;20(6):e210. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shanmugasundaram G, Yazhini S, Hemapratha E, Nithya S. A comprehensive review on stress detection techniques. In: Proceedings of the 2019 IEEE International Conference on System, Computation, Automation and Networking (ICSCAN). 2019. Presented at: ICSCAN 2019; March 29-30, 2019; Pondicherry, India. [ CrossRef ]
• Siddi S, Bailon R, Giné-Vázquez I, Matcham F, Lamers F, Kontaxis S, PAB members, et al. The usability of daytime and night-time heart rate dynamics as digital biomarkers of depression severity. Psychol Med. Jun 2023;53(8):3249-3260. [ CrossRef ] [ Medline ]
• Kontaxis S, Gil E, Marozas V, Lazaro J, Garcia E, Posadas-de Miguel M, et al. Photoplethysmographic waveform analysis for autonomic reactivity assessment in depression. IEEE Trans Biomed Eng. Apr 2021;68(4):1273-1281. [ CrossRef ]
• Sano A, Phillips AJ, Yu AZ, McHill AW, Taylor S, Jaques N, et al. Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones. In: Proceedings of the 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN). 2015. Presented at: BSN 2015; June 9-12, 2015; Cambridge, MA. [ CrossRef ]
• Kroenke K, Spitzer RL, Williams JB, Löwe B. An ultra-brief screening scale for anxiety and depression: the PHQ-4. Psychosomatics. 2009;50(6):613-621. [ CrossRef ] [ Medline ]
• Stewart-Brown S, Tennant A, Tennant R, Platt S, Parkinson J, Weich S. Internal construct validity of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS): a Rasch analysis using data from the Scottish Health Education Population Survey. Health Qual Life Outcomes. Feb 19, 2009;7:15. [ FREE Full text ] [ CrossRef ] [ Medline ]
• López MA, Gabilondo A, Codony M, García-Forero C, Vilagut G, Castellví P, et al. Adaptation into Spanish of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS) and preliminary validation in a student sample. Qual Life Res. Jun 2013;22(5):1099-1104. [ CrossRef ] [ Medline ]
• Medina A, Schiaffino A. Enquesta de salut de Catalunya, 2016. Direcció General de Planificació en Salut. 2017. URL: https://hdl.handle.net/11351/3205 [accessed 2023-07-23]
• Golden CJ. Stroop. Test de Colores y Palabras – Edición Revisada (B. Ruiz-Fernández, T. Luque y F. Sánchez-Sánchez, adaptadores). TEA Ediciones. Madrid. TEA Ediciones; 2020. URL: https://web.teaediciones.com/Ejemplos/STROOP-extracto-Web.pdf [accessed 2023-04-27]
• Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary care evaluation of mental disorders. Patient health questionnaire. JAMA. Nov 10, 1999;282(18):1737-1744. [ CrossRef ] [ Medline ]
• Diez-Quevedo C, Rangil T, Sanchez-Planell L, Kroenke K, Spitzer RL. Validation and utility of the patient health questionnaire in diagnosing mental disorders in 1003 general hospital Spanish inpatients. Psychosom Med. 2001;63(4):679-686. [ CrossRef ] [ Medline ]
• Levis B, Benedetti A, Thombs BD, DEPRESsion Screening Data (DEPRESSD) Collaboration. Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: individual participant data meta-analysis. BMJ. Apr 09, 2019;365:l1476. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [ CrossRef ] [ Medline ]
• García-Campayo J, Zamorano E, Ruiz MA, Pardo A, Pérez-Páramo M, López-Gómez V, et al. Cultural adaptation into Spanish of the Generalized Anxiety Disorder-7 (GAD-7) scale as a screening tool. Health Qual Life Outcomes. Jan 20, 2010;8:8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. Dec 1983;24(4):385-396. [ CrossRef ]
• Remor E. Psychometric properties of a European Spanish version of the Perceived Stress Scale (PSS). Span J Psychol. May 2006;9(1):86-93. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown RL, Rounds LA. Conjoint screening questionnaires for alcohol and other drug abuse: criterion validity in a primary care practice. Wis Med J. 1995;94(3):135-140. [ Medline ]
• Ewing JA. Detecting alcoholism: the CAGE Questionnaire. JAMA. Oct 12, 1984;252(14):1905-1907. [ CrossRef ]
• Craig CL, Marshall AL, Sjöström M, Bauman AE, Booth ML, Ainsworth BE, et al. International Physical Activity Questionnaire: 12-country reliability and validity. Med Sci Sports Exerc. Aug 2003;35(8):1381-1395. [ CrossRef ] [ Medline ]
• Román Viñas B, Ribas Barba L, Ngo J, Serra Majem L. Validity of the International Physical Activity Questionnaire in the Catalan population (Spain) [Article in Spanish]. Gac Sanit. 2013;27(3):254-257. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stewart AL, Ware JE. Measuring Functioning and Well-being: The Medical Outcomes Study Approach. Durham, NC. Duke University Press; 1992.
• Rejas J, Ribera MV, Ruiz M, Masrramón X. Psychometric properties of the MOS (Medical Outcomes Study) sleep scale in patients with neuropathic pain. Eur J Pain. Apr 2007;11(3):329-340. [ CrossRef ] [ Medline ]
• Stroop JR. Studies of interference in serial verbal reactions. J Exp Psychol. 1935;18(6):643-662. [ CrossRef ]
• Hamid FA, Saad MN, Malik AS. Characterization stress reactions to Stroop Color-Word Test using spectral analysis. Mater Today Proc. 2019;16(4):1949-1958. [ CrossRef ]
• Aguiló Mir S, García Pagès E, López Barbeito B, Ribeiro TC, Garzón-Rey JM, Aguiló Llobet J. Design and validation of an electrophysiological based tool to assess chronic stress. case study: burnout syndrome in caregivers. Stress. Jul 2021;24(4):384-393. [ CrossRef ] [ Medline ]
• Bolea J, Almeida R, Laguna P, Sornmo L, Martinez JP. BioSigBrowser, biosignal processing interface. In: Proceedings of the 2009 9th International Conference on Information Technology and Applications in Biomedicine. 2009. Presented at: ITAB 2009; November 4-7, 2009; Larnaka, Cyprus. [ CrossRef ]
• Greco A, Valenza G, Lanata A, Scilingo E, Citi L. cvxEDA: a convex optimization approach to electrodermal activity processing. IEEE Trans Biomed Eng. Apr 2016;63(4):797-804. [ CrossRef ]
• Pham T, Lau ZJ, Chen SH, Makowski D. Heart rate variability in psychology: a review of HRV indices and an analysis tutorial. Sensors (Basel). Jun 09, 2021;21(12):3998. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nardelli M, Greco A, Sebastiani L, Scilingo EP. ComEDA: a new tool for stress assessment based on electrodermal activity. Comput Biol Med. Nov 2022;150:106144. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Finnegan E, Davidson S, Harford M, Jorge J, Watkinson P, Young D, et al. Pulse arrival time as a surrogate of blood pressure. Sci Rep. Nov 23, 2021;11(1):22767. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Celka P, Charlton PH, Farukh B, Chowienczyk P, Alastruey J. Influence of mental stress on the pulse wave features of photoplethysmograms. Healthc Technol Lett. Nov 26, 2019;7(1):7-12. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Castro Ribeiro T, Sobregrau Sangrà P, García Pagès E, Badiella L, López-Barbeito B, Aguiló S, et al. Assessing effectiveness of heart rate variability biofeedback to mitigate mental health symptoms: a pilot study. Front Physiol. May 10, 2023;14:1147260. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leyro TM, Versella MV, Yang MJ, Brinkman HR, Hoyt DL, Lehrer P. Respiratory therapy for the treatment of anxiety: meta-analytic review and regression. Clin Psychol Rev. Mar 2021;84:101980. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hernando A, Lazaro J, Gil E, Arza A, Garzon JM, Lopez-Anton R, et al. Inclusion of respiratory frequency information in heart rate variability analysis for stress assessment. IEEE J Biomed Health Inform. Jul 2016;20(4):1016-1025. [ FREE Full text ] [ CrossRef ]
• Gioia F, Callara AL, Bruderer T, Ripszam M, Di Francesco F, Scilingo EP, et al. Potential physiological stress biomarkers in human sweat. In: Proceedings of the 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA). 2022. Presented at: MeMeA 2022; June 22-24, 2022; Messina, Italy. [ CrossRef ]
• Martinez JP, Almeida R, Olmos S, Rocha AP, Laguna P. A wavelet-based ECG delineator: evaluation on standard databases. IEEE Trans Biomed Eng. Apr 2004;51(4):570-581. [ CrossRef ]
• Mateo J, Laguna P. Analysis of heart rate variability in the presence of ectopic beats using the heart timing signal. IEEE Trans Biomed Eng. Mar 2003;50(3):334-343. [ CrossRef ]
• Gil E, Mendez M, Vergara JM, Cerutti S, Bianchi AM, Laguna P. Discrimination of sleep-apnea-related decreases in the amplitude fluctuations of PPG signal in children by HRV analysis. IEEE Trans Biomed Eng. Apr 2009;56(4):1005-1014. [ CrossRef ]
• Lazaro J, Gil E, Vergara JM, Laguna P. Pulse rate variability analysis for discrimination of sleep-apnea-related decreases in the amplitude fluctuations of pulse photoplethysmographic signal in children. IEEE J Biomed Health Inform. Jan 2014;18(1):240-246. [ CrossRef ]
• Lázaro J, Alcaine A, Romero D, Gil E, Laguna P, Pueyo E, et al. Electrocardiogram derived respiratory rate from QRS slopes and R-wave angle. Ann Biomed Eng. Oct 2014;42(10):2072-2083. [ CrossRef ] [ Medline ]
• Posada-Quintero HF, Florian JP, Orjuela-Cañón AD, Aljama-Corrales T, Charleston-Villalobos S, Chon KH. Power spectral density analysis of electrodermal activity for sympathetic function assessment. Ann Biomed Eng. Oct 2016;44(10):3124-3135. [ CrossRef ] [ Medline ]
• Cohen J. Statistical Power Analysis for the Behavioral Sciences. Cambridge, MA. Academic Press; 1969.
• Löwe B, Unützer J, Callahan CM, Perkins AJ, Kroenke K. Monitoring depression treatment outcomes with the Patient Health Questionnaire-9. Med Care. Dec 2004;42(12):1194-1201. [ CrossRef ] [ Medline ]
• Shah N, Cader M, Andrews WP, Wijesekera D, Stewart-Brown SL. Responsiveness of the Short Warwick Edinburgh Mental Well-Being Scale (SWEMWBS): evaluation a clinical sample. Health Qual Life Outcomes. Dec 22, 2018;16(1):239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Toussaint A, Hüsing P, Gumz A, Wingenfeld K, Härter M, Schramm E, et al. Sensitivity to change and minimal clinically important difference of the 7-item Generalized Anxiety Disorder Questionnaire (GAD-7). J Affect Disord. Mar 15, 2020;265:395-401. [ CrossRef ] [ Medline ]
• Amin R, Faghih RT. Physiological characterization of electrodermal activity enables scalable near real-time autonomic nervous system activation inference. PLoS Comput Biol. Jul 28, 2022;18(7):e1010275. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arza A, Garzón-Rey JM, Lázaro J, Gil E, Lopez-Anton R, de la Camara C, et al. Measuring acute stress response through physiological signals: towards a quantitative assessment of stress. Med Biol Eng Comput. Jan 2019;57(1):271-287. [ CrossRef ] [ Medline ]
• Laborde S, Mosley E, Thayer JF. Heart rate variability and cardiac vagal tone in psychophysiological research - recommendations for experiment planning, data analysis, and data reporting. Front Psychol. Feb 20, 2017;8:213. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garzón-Rey JM. Psychosomatic approach to stress measurement. Universitat Autònoma de Barcelona. 2017. URL: https://hdl.handle.net/10803/458661 [accessed 2020-09-17]
• Satish P, Muralikrishnan K, Balasubramanian K, Shanmugapriya. Heart rate variability changes during Stroop Color and Word Test among genders. Indian J Physiol Pharmacol. 2015;59(1):9-15. [ Medline ]
• Boutcher YN, Boutcher SH. Cardiovascular response to Stroop: effect of verbal response and task difficulty. Biol Psychol. Oct 2006;73(3):235-241. [ CrossRef ] [ Medline ]
• MacLeod CM. Half a century of research on the Stroop effect: an integrative review. Psychol Bull. 1991;109(2):163-203. [ CrossRef ]
• Berggren N, Derakshan N. Inhibitory deficits in trait anxiety: increased stimulus-based or response-based interference? Psychon Bull Rev. Oct 2014;21(5):1339-1345. [ CrossRef ] [ Medline ]
• Koch C, Wilhelm M, Salzmann S, Rief W, Euteneuer F. A meta-analysis of heart rate variability in major depression. Psychol Med. Sep 2019;49(12):1948-1957. [ CrossRef ] [ Medline ]
• Hernández-Torrano D, Ibrayeva L, Sparks J, Lim N, Clementi A, Almukhambetova A, et al. Mental health and well-being of university students: a bibliometric mapping of the literature. Front Psychol. Jun 9, 2020;11:1226. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 27.07.23; peer-reviewed by V Sideropoulos; comments to author 09.11.23; revised version received 22.12.23; accepted 11.01.24; published 29.03.24.

©Thais Castro Ribeiro, Esther García Pagès, Laura Ballester, Gemma Vilagut, Helena García Mieres, Víctor Suárez Aragonès, Franco Amigo, Raquel Bailón, Philippe Mortier, Víctor Pérez Sola, Antoni Serrano-Blanco, Jordi Alonso, Jordi Aguiló. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 29.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.

• Program Finder
• Admissions Services
• Course Directory
• Academic Calendar
• Hybrid Campus
• Lecture Series
• Convocation
• Strategy and Development
• Implementation and Impact
• Integrity and Oversight
• In the School
• In the Field
• In Baltimore
• Resources for Practitioners
• Articles & News Releases
• In The News
• Statements & Announcements
• At a Glance
• Student Life
• Strategic Priorities
• Inclusion, Diversity, Anti-Racism, and Equity (IDARE)
• What is Public Health?

Research Identifies Characteristics of Cities That Would Support Young People’s Mental Health

Survey responses from global panel that included young people provide insights into what would make cities mental health-friendly for youth

As cities around the world continue to draw young people for work, education, and social opportunities, a new study identifies characteristics that would support young urban dwellers’ mental health. The findings, based on survey responses from a global panel that included adolescents and young adults, provide a set of priorities that city planners can adopt to build urban environments that are safe, equitable, and inclusive. 

To determine city characteristics that could bolster youth mental health, researchers administered an initial survey to a panel of more than 400, including young people and a multidisciplinary group of researchers, practitioners, and advocates. Through two subsequent surveys, participants prioritized six characteristics that would support young city dwellers’ mental health: opportunities to build life skills; age-friendly environments that accept young people’s feelings and values; free and safe public spaces where young people can connect; employment and job security; interventions that address the social determinants of health; and urban design with youth input and priorities in mind. 

The paper was published online February 21 in  Nature .

The study’s lead author is Pamela Collins, MD, MPH, chair of the Johns Hopkins Bloomberg School of Public Health’s Department of Mental Health. The study was conducted while Collins was on the faculty at the University of Washington. The paper was written by an international, interdisciplinary team, including citiesRISE, a global nonprofit that works to transform mental health policy and practice in cities, especially for young people.

Cities have long been a draw for young people. Research by UNICEF projects that cities will be home to 70 percent of the world’s children by 2050. Although urban environments influence a broad range of health outcomes, both positive and negative, their impacts manifest unequally. Mental disorders are the leading causes of disability among 10- to 24-year-olds globally. Exposure to urban inequality, violence, lack of green space, and fear of displacement disproportionately affects marginalized groups, increasing risk for poor mental health among urban youth.

“Right now, we are living with the largest population of adolescents in the world’s history, so this is an incredibly important group of people for global attention,” says Collins. “Investing in young people is an investment in their present well-being and future potential, and it’s an investment in the next generation—the children they will bear.” 

Data collection for the study began in April 2020 at the start of the COVID-19 pandemic. To capture its possible impacts, researchers added an open-ended survey question asking panelists how the pandemic influenced their perceptions of youth mental health in cities. The panelists reported that the pandemic either shed new light on the inequality and uneven distribution of resources experienced by marginalized communities in urban areas, or confirmed their preconceptions of how social vulnerability exacerbates health outcomes. 

For their study, the researchers recruited a panel of more than 400 individuals from 53 countries, including 327 young people ages 14 to 25, from a cross-section of fields, including education, advocacy, adolescent health, mental health and substance use, urban planning and development, data and technology, housing, and criminal justice. The researchers administered three sequential surveys to panelists beginning in April 2020 that asked panelists to identify elements of urban life that would support mental health for young people.

The top 37 characteristics were then grouped into six domains: intrapersonal, interpersonal, community, organizational, policy, and environment. Within these domains, panelists ranked characteristics based on immediacy of impact on youth mental health, ability to help youth thrive, and ease or feasibility of implementation. 

Taken together, the characteristics identified in the study provide a comprehensive set of priorities that policymakers and urban planners can use as a guide to improve young city dwellers' mental health. Among them: Youth-focused mental health and educational services could support young people’s emotional development and self-efficacy. Investment in spaces that facilitate social connection may help alleviate young people’s experiences of isolation and support their need for healthy, trusting relationships. Creating employment opportunities and job security could undo the economic losses that young people and their families experienced during the pandemic and help cities retain residents after a COVID-era exodus from urban centers.  

The findings suggest that creating a mental health-friendly city for young people requires investments across multiple interconnected sectors like transportation, housing, employment, health, and urban planning, with a central focus on social and economic equity. They also require urban planning policy approaches that commit to systemic and sustained collaboration, without magnifying existing privileges through initiatives like gentrification and developing green spaces at the expense of marginalized communities in need of affordable housing.

The authors say this framework underscores that responses by cities should include young people in the planning and design of interventions that directly impact their mental health and well-being. 

“ Making cities mental health friendly for adolescents and young adults ” was co-authored by an international, interdisciplinary team of 31 researchers led by the University of Washington Consortium for Global Mental Health, Urban@UW, the University of Melbourne, and citiesRISE. Author funding is listed in the Acknowledgements section of the paper.

Related Content

Student Spotlight: Glendedora Dolce

Study Estimates Nearly 70 Percent of Children Under Six in Chicago May Be Exposed to Lead-Contaminated Tap Water

Hidden Food Insecurity: The Adolescents Who Aren’t Getting Enough to Eat

A Zero-Emissions Transition Would Save Kids’ Lives

Child Diarrhea Has a Cheap and Easy Fix—Why Isn’t It Reaching Patients?

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychiatry

Mental Health Prevention and Promotion—A Narrative Review

Associated data.

Extant literature has established the effectiveness of various mental health promotion and prevention strategies, including novel interventions. However, comprehensive literature encompassing all these aspects and challenges and opportunities in implementing such interventions in different settings is still lacking. Therefore, in the current review, we aimed to synthesize existing literature on various mental health promotion and prevention interventions and their effectiveness. Additionally, we intend to highlight various novel approaches to mental health care and their implications across different resource settings and provide future directions. The review highlights the (1) concept of preventive psychiatry, including various mental health promotions and prevention approaches, (2) current level of evidence of various mental health preventive interventions, including the novel interventions, and (3) challenges and opportunities in implementing concepts of preventive psychiatry and related interventions across the settings. Although preventive psychiatry is a well-known concept, it is a poorly utilized public health strategy to address the population's mental health needs. It has wide-ranging implications for the wellbeing of society and individuals, including those suffering from chronic medical problems. The researchers and policymakers are increasingly realizing the potential of preventive psychiatry; however, its implementation is poor in low-resource settings. Utilizing novel interventions, such as mobile-and-internet-based interventions and blended and stepped-care models of care can address the vast mental health need of the population. Additionally, it provides mental health services in a less-stigmatizing and easily accessible, and flexible manner. Furthermore, employing decision support systems/algorithms for patient management and personalized care and utilizing the digital platform for the non-specialists' training in mental health care are valuable additions to the existing mental health support system. However, more research concerning this is required worldwide, especially in the low-and-middle-income countries.

Introduction

Mental disorder has been recognized as a significant public health concern and one of the leading causes of disability worldwide, particularly with the loss of productive years of the sufferer's life ( 1 ). The Global Burden of Disease report (2019) highlights an increase, from around 80 million to over 125 million, in the worldwide number of Disability-Adjusted Life Years (DALYs) attributable to mental disorders. With this surge, mental disorders have moved into the top 10 significant causes of DALYs worldwide over the last three decades ( 2 ). Furthermore, this data does not include substance use disorders (SUDs), which, if included, would increase the estimated burden manifolds. Moreover, if the caregiver-related burden is accounted for, this figure would be much higher. Individual, social, cultural, political, and economic issues are critical mental wellbeing determinants. An increasing burden of mental diseases can, in turn, contribute to deterioration in physical health and poorer social and economic growth of a country ( 3 ). Mental health expenditure is roughly 3–4% of their Gross Domestic Products (GDPs) in developed regions of the world; however, the figure is abysmally low in low-and-middle-income countries (LMICs) ( 4 ). Untreated mental health and behavioral problems in childhood and adolescents, in particular, have profound long-term social and economic adverse consequences, including increased contact with the criminal justice system, lower employment rate and lesser wages among those employed, and interpersonal difficulties ( 5 – 8 ).

Need for Mental Health (MH) Prevention

Longitudinal studies suggest that individuals with a lower level of positive wellbeing are more likely to acquire mental illness ( 9 ). Conversely, factors that promote positive wellbeing and resilience among individuals are critical in preventing mental illnesses and better outcomes among those with mental illness ( 10 , 11 ). For example, in patients with depressive disorders, higher premorbid resilience is associated with earlier responses ( 12 ). On the contrary, patients with bipolar affective- and recurrent depressive disorders who have a lower premorbid quality of life are at higher risk of relapses ( 13 ).

Recently there has been an increased emphasis on the need to promote wellbeing and positive mental health in preventing the development of mental disorders, for poor mental health has significant social and economic implications ( 14 – 16 ). Research also suggests that mental health promotion and preventative measures are cost-effective in preventing or reducing mental illness-related morbidity, both at the society and individual level ( 17 ).

Although the World Health Organization (WHO) defines health as “a state of complete physical, mental, and social wellbeing and not merely an absence of disease or infirmity,” there has been little effort at the global level or stagnation in implementing effective mental health services ( 18 ). Moreover, when it comes to the research on mental health (vis-a-viz physical health), promotive and preventive mental health aspects have received less attention vis-a-viz physical health. Instead, greater emphasis has been given to the illness aspect, such as research on psychopathology, mental disorders, and treatment ( 19 , 20 ). Often, physicians and psychiatrists are unfamiliar with various concepts, approaches, and interventions directed toward mental health promotion and prevention ( 11 , 21 ).

Prevention and promotion of mental health are essential, notably in reducing the growing magnitude of mental illnesses. However, while health promotion and disease prevention are universally regarded concepts in public health, their strategic application for mental health promotion and prevention are often elusive. Furthermore, given the evidence of substantial links between psychological and physical health, the non-incorporation of preventive mental health services is deplorable and has serious ramifications. Therefore, policymakers and health practitioners must be sensitized about linkages between mental- and physical health to effectively implement various mental health promotive and preventive interventions, including in individuals with chronic physical illnesses ( 18 ).

The magnitude of the mental health problems can be gauged by the fact that about 10–20% of young individuals worldwide experience depression ( 22 ). As described above, poor mental health during childhood is associated with adverse health (e.g., substance use and abuse), social (e.g., delinquency), academic (e.g., school failure), and economic (high risk of poverty) adverse outcomes in adulthood ( 23 ). Childhood and adolescence are critical periods for setting the ground for physical growth and mental wellbeing ( 22 ). Therefore, interventions promoting positive psychology empower youth with the life skills and opportunities to reach their full potential and cope with life's challenges. Comprehensive mental health interventions involving families, schools, and communities have resulted in positive physical and psychological health outcomes. However, the data is limited to high-income countries (HICs) ( 24 – 28 ).

In contrast, in low and middle-income countries (LMICs) that bear the greatest brunt of mental health problems, including massive, coupled with a high treatment gap, such interventions remained neglected in public health ( 29 , 30 ). This issue warrants prompt attention, particularly when global development strategies such as Millennium Development Goals (MDGs) realize the importance of mental health ( 31 ). Furthermore, studies have consistently reported that people with socioeconomic disadvantages are at a higher risk of mental illness and associated adverse outcomes; partly, it is attributed to the inequitable distribution of mental health services ( 32 – 35 ).

Scope of Mental Health Promotion and Prevention in the Current Situation

Literature provides considerable evidence on the effectiveness of various preventive mental health interventions targeting risk and protective factors for various mental illnesses ( 18 , 36 – 42 ). There is also modest evidence of the effectiveness of programs focusing on early identification and intervention for severe mental diseases (e.g., schizophrenia and psychotic illness, and bipolar affective disorders) as well as common mental disorders (e.g., anxiety, depression, stress-related disorders) ( 43 – 46 ). These preventive measures have also been evaluated for their cost-effectiveness with promising findings. In addition, novel interventions such as digital-based interventions and novel therapies (e.g., adventure therapy, community pharmacy program, and Home-based Nurse family partnership program) to address the mental health problems have yielded positive results. Likewise, data is emerging from LMICs, showing at least moderate evidence of mental health promotion intervention effectiveness. However, most of the available literature and intervention is restricted mainly to the HICs ( 47 ). Therefore, their replicability in LMICs needs to be established and, also, there is a need to develop locally suited interventions.

Fortunately, there has been considerable progress in preventive psychiatry over recent decades, including research on it. In the light of these advances, there is an accelerated interest among researchers, clinicians, governments, and policymakers to harness the potentialities of the preventive strategies to improve the availability, accessibility, and utility of such services for the community.

The Concept of Preventive Psychiatry

Origins of preventive psychiatry.

The history of preventive psychiatry can be traced back to the early 1900's with the foundation of the national mental health association (erstwhile mental health association), the committee on mental hygiene in New York, and the mental health hygiene movement ( 48 ). The latter emphasized the need for physicians to develop empathy and recognize and treat mental illness early, leading to greater awareness about mental health prevention ( 49 ). Despite that, preventive psychiatry remained an alien concept for many, including mental health professionals, particularly when the etiology of most psychiatric disorders was either unknown or poorly understood. However, recent advances in our understanding of the phenomena underlying psychiatric disorders and availability of the neuroimaging and electrophysiological techniques concerning mental illness and its prognosis has again brought the preventive psychiatry in the forefront ( 1 ).

Levels of Prevention

The literal meaning of “prevention” is “the act of preventing something from happening” ( 50 ); the entity being prevented can range from the risk factors of the development of the illness, the onset of illness, or the recurrence of the illness or associated disability. The concept of prevention emerged primarily from infectious diseases; measures like mass vaccination and sanitation promotion have helped prevent the development of the diseases and subsequent fatalities. The original preventive model proposed by the Commission on Chronic Illness in 1957 included primary, secondary, and tertiary preventions ( 48 ).

The Concept of Primary, Secondary, and Tertiary Prevention

The stages of prevention target distinct aspects of the illness's natural course; the primary prevention acts at the stage of pre-pathogenesis, that is, when the disease is yet to occur, whereas the secondary and tertiary prevention target the phase after the onset of the disease ( 51 ). Primary prevention includes health promotion and specific protection, while secondary and tertairy preventions include early diagnosis and treatment and measures to decrease disability and rehabilitation, respectively ( 51 ) ( Figure 1 ).

The concept of primary and secondary prevention [adopted from prevention: Primary, Secondary, Tertiary by Bauman et al. ( 51 )].

The primary prevention targets those individuals vulnerable to developing mental disorders and their consequences because of their bio-psycho-social attributes. Therefore, it can be viewed as an intervention to prevent an illness, thereby preventing mental health morbidity and potential social and economic adversities. The preventive strategies under it usually target the general population or individuals at risk. Secondary and tertiary prevention targets those who have already developed the illness, aiming to reduce impairment and morbidity as soon as possible. However, these measures usually occur in a person who has already developed an illness, therefore facing related suffering, hence may not always be successful in curing or managing the illness. Thus, secondary and tertiary prevention measures target the already exposed or diagnosed individuals.

The Concept of Universal, Selective, and Indicated Prevention

The classification of health prevention based on primary/secondary/tertiary prevention is limited in being highly centered on the etiology of the illness; it does not consider the interaction between underlying etiology and risk factors of an illness. Gordon proposed another model of prevention that focuses on the degree of risk an individual is at, and accordingly, the intensity of intervention is determined. He has classified it into universal, selective, and indicated prevention. A universal preventive strategy targets the whole population irrespective of individual risk (e.g., maintaining healthy, psychoactive substance-free lifestyles); selective prevention is targeted to those at a higher risk than the general population (socio-economically disadvantaged population, e.g., migrants, a victim of a disaster, destitute, etc.). The indicated prevention aims at those who have established risk factors and are at a high risk of getting the disease (e.g., family history of psychiatric illness, history of substance use, certain personality types, etc.). Nevertheless, on the other hand, these two classifications (the primary, secondary, and tertiary prevention; and universal, selective, and indicated prevention) have been intended for and are more appropriate for physical illnesses with a clear etiology or risk factors ( 48 ).

In 1994, the Institute of Medicine (IOM) Committee on Prevention of Mental Disorders proposed a new paradigm that classified primary preventive measures for mental illnesses into three categories. These are indicated, selected, and universal preventive interventions (refer Figure 2 ). According to this paradigm, primary prevention was limited to interventions done before the onset of the mental illness ( 48 ). In contrast, secondary and tertiary prevention encompasses treatment and maintenance measures ( Figure 2 ).

The interventions for mental illness as classified by the Institute of Medicine (IOM) Committee on Prevention of Mental Disorders [adopted from Mrazek and Haggerty ( 48 )].

Although the boundaries between prevention and treatment are often more overlapping than being exclusive, the new paradigm can be used to avoid confusion stemming from the common belief that prevention can take place at all parts of mental health management ( 48 ). The onset of mental illnesses can be prevented by risk reduction interventions, which can involve reducing risk factors in an individual and strengthening protective elements in them. It aims to target modifiable factors, both risk, and protective factors, associated with the development of the illness through various general and specific interventions. These interventions can work across the lifespan. The benefits are not restricted to reduction or delay in the onset of illness but also in terms of severity or duration of illness ( 48 ).On the spectrum of mental health interventions, universal preventive interventions are directed at the whole population without identifiable risk factors. The interventions are beneficial for the general population or sub-groups. Prenatal care and childhood vaccination are examples of preventative measures that have benefited both physical and mental health. Selective preventive mental health interventions are directed at people or a subgroup with a significantly higher risk of developing mental disorders than the general population. Risk groups are those who, because of their vulnerabilities, are at higher risk of developing mental illnesses, e.g., infants with low-birth-weight (LBW), vulnerable children with learning difficulties or victims of maltreatment, elderlies, etc. Specific interventions are home visits and new-born day care facilities for LBW infants, preschool programs for all children living in resource-deprived areas, support groups for vulnerable elderlies, etc. Indicated preventive interventions focus on high-risk individuals who have developed minor but observable signs or symptoms of mental disorder or genetic risk factors for mental illness. However, they have not fulfilled the criteria of a diagnosable mental disorder. For instance, the parent-child interaction training program is an indicated prevention strategy that offers support to children whose parents have recognized them as having behavioral difficulties.

The overall objective of mental health promotion and prevention is to reduce the incidence of new cases, additionally delaying the emergence of mental illness. However, promotion and prevention in mental health complement each other rather than being mutually exclusive. Moreover, combining these two within the overall public health framework reduces stigma, increases cost-effectiveness, and provides multiple positive outcomes ( 18 ).

How Prevention in Psychiatry Differs From Other Medical Disorders

Compared to physical illnesses, diagnosing a mental illness is more challenging, particularly when there is still a lack of objective assessment methods, including diagnostic tools and biomarkers. Therefore, the diagnosis of mental disorders is heavily influenced by the assessors' theoretical perspectives and subjectivity. Moreover, mental illnesses can still be considered despite an individual not fulfilling the proper diagnostic criteria led down in classificatory systems, but there is detectable dysfunction. Furthermore, the precise timing of disorder initiation or transition from subclinical to clinical condition is often uncertain and inconclusive ( 48 ). Therefore, prevention strategies are well-delineated and clear in the case of physical disorders while it's still less prevalent in mental health parlance.

Terms, Definitions, and Concepts

The terms mental health, health promotion, and prevention have been differently defined and interpreted. It is further complicated by overlapping boundaries of the concept of promotion and prevention. Some commonly used terms in mental health prevention have been tabulated ( Table 1 ) ( 18 ).

Commonly used terms in mental health prevention.

Mental Health Promotion and Protection

The term “mental health promotion” also has definitional challenges as it signifies different things to different individuals. For some, it means the treatment of mental illness; for others, it means preventing the occurrence of mental illness; while for others, it means increasing the ability to manage frustration, stress, and difficulties by strengthening one's resilience and coping abilities ( 54 ). It involves promoting the value of mental health and improving the coping capacities of individuals rather than amelioration of symptoms and deficits.

Mental health promotion is a broad concept that encompasses the entire population, and it advocates for a strengths-based approach and tries to address the broader determinants of mental health. The objective is to eliminate health inequalities via empowerment, collaboration, and participation. There is mounting evidence that mental health promotion interventions improve mental health, lower the risk of developing mental disorders ( 48 , 55 , 56 ) and have socioeconomic benefits ( 24 ). In addition, it strives to increase an individual's capacity for psychosocial wellbeing and adversity adaptation ( 11 ).

However, the concepts of mental health promotion, protection, and prevention are intrinsically linked and intertwined. Furthermore, most mental diseases result from complex interaction risk and protective factors instead of a definite etiology. Facilitating the development and timely attainment of developmental milestones across an individual's lifespan is critical for positive mental health ( 57 ). Although mental health promotion and prevention are essential aspects of public health with wide-ranging benefits, their feasibility and implementation are marred by financial and resource constraints. The lack of cost-effectiveness studies, particularly from the LMICs, further restricts its full realization ( 47 , 58 , 59 ).

Despite the significance of the topic and a considerable amount of literature on it, a comprehensive review is still lacking that would cover the concept of mental health promotion and prevention and simultaneously discusses various interventions, including the novel techniques delivered across the lifespan, in different settings, and level of prevention. Therefore, this review aims to analyze the existing literature on various mental health promotion and prevention-based interventions and their effectiveness. Furthermore, its attempts to highlight the implications of such intervention in low-resource settings and provides future directions. Such literature would add to the existing literature on mental health promotion and prevention research and provide key insights into the effectiveness of such interventions and their feasibility and replicability in various settings.

Methodology

For the current review, key terms like “mental health promotion,” OR “protection,” OR “prevention,” OR “mitigation” were used to search relevant literature on Google Scholar, PubMed, and Cochrane library databases, considering a time period between 2000 to 2019 ( Supplementary Material 1 ). However, we have restricted our search till 2019 for non-original articles (reviews, commentaries, viewpoints, etc.), assuming that it would also cover most of the original articles published until then. Additionally, we included original papers from the last 5 years (2016–2021) so that they do not get missed out if not covered under any published review. The time restriction of 2019 for non-original articles was applied to exclude papers published during the Coronavirus disease (COVID-19) pandemic as the latter was a significant event, bringing about substantial change and hence, it warranted a different approach to cater to the MH needs of the population, including MH prevention measures. Moreover, the COVID-19 pandemic resulted in the flooding of novel interventions for mental health prevention and promotion, specifically targeting the pandemic and its consequences, which, if included, could have biased the findings of the current review on various MH promotion and prevention interventions.

A time frame of about 20 years was taken to see the effectiveness of various MH promotion and protection interventions as it would take substantial time to be appreciated in real-world situations. Therefore, the current paper has put greater reliance on the review articles published during the last two decades, assuming that it would cover most of the original articles published until then.

The above search yielded 320 records: 225 articles from Google scholar, 59 articles from PubMed, and 36 articles from the Cochrane database flow-diagram of records screening. All the records were title/abstract screened by all the authors to establish the suitability of those records for the current review; a bibliographic- and gray literature search was also performed. In case of any doubts or differences in opinion, it was resolved by mutual discussion. Only those articles directly related to mental health promotion, primary prevention, and related interventions were included in the current review. In contrast, records that discussed any specific conditions/disorders (post-traumatic stress disorders, suicide, depression, etc.), specific intervention (e.g., specific suicide prevention intervention) that too for a particular population (e.g., disaster victims) lack generalizability in terms of mental health promotion or prevention, those not available in the English language, and whose full text was unavailable were excluded. The findings of the review were described narratively.

Interventions for Mental Health Promotion and Prevention and Their Evidence

Various interventions have been designed for mental health promotion and prevention. They are delivered and evaluated across the regions (high-income countries to low-resource settings, including disaster-affiliated regions of the world), settings (community-based, school-based, family-based, or individualized); utilized different psychological constructs and therapies (cognitive behavioral therapy, behavioral interventions, coping skills training, interpersonal therapies, general health education, etc.); and delivered by different professionals/facilitators (school-teachers, mental health professionals or paraprofessionals, peers, etc.). The details of the studies, interventions used, and outcomes have been provided in Supplementary Table 1 . Below we provide the synthesized findings of the available research.

The majority of the available studies were quantitative and experimental. Randomized controlled trials comprised a sizeable proportion of the studies; others were quasi-experimental studies and, a few, qualitative studies. The studies primarily focussed on school students or the younger population, while others were explicitly concerned with the mental health of young females ( 60 ). Newer data is emerging on mental health promotion and prevention interventions for elderlies (e.g., dementia) ( 61 ). The majority of the research had taken a broad approach to mental health promotion ( 62 ). However, some studies have focused on universal prevention ( 63 , 64 ) or selective prevention ( 65 – 68 ). For instance, the Resourceful Adolescent Program (RAPA) was implemented across the schools and has utilized cognitive-behavioral and interpersonal therapies and reported a significant improvement in depressive symptoms. Some of the interventions were directed at enhancing an individual's characteristics like resilience, behavior regulation, and coping skills (ZIPPY's Friends) ( 69 ), while others have focused on the promotion of social and emotional competencies among the school children and attempted to reduce the gap in such competencies across the socio-economic classes (“Up” program) ( 70 ) or utilized expressive abilities of the war-affected children (Writing for Recover (WfR) intervention) ( 71 ) to bring about an improvement in their psychological problems (a type of selective prevention) ( 62 ) or harnessing the potential of Art, in the community-based intervention, to improve self-efficacy, thus preventing mental disorders (MAD about Art program) ( 72 ). Yet, others have focused on strengthening family ( 60 , 73 ), community relationships ( 62 ), and targeting modifiable risk factors across the life course to prevent dementia among the elderlies and also to support the carers of such patients ( 61 ).

Furthermore, more of the studies were conducted and evaluated in the developed parts of the world, while emerging economies, as anticipated, far lagged in such interventions or related research. The interventions that are specifically adapted for local resources, such as school-based programs involving paraprofessionals and teachers in the delivery of mental health interventions, were shown to be more effective ( 62 , 74 ). Likewise, tailored approaches for low-resource settings such as LMICs may also be more effective ( 63 ). Some of these studies also highlight the beneficial role of a multi-dimensional approach ( 68 , 75 ) and interventions targeting early lifespan ( 76 , 77 ).

Newer Insights: How to Harness Digital Technology and Novel Methods of MH Promotion and Protection

With the advent of digital technology and simultaneous traction on mental health promotion and prevention interventions, preventive psychiatrists and public health experts have developed novel techniques to deliver mental health promotive and preventive interventions. These encompass different settings (e.g., school, home, workplace, the community at large, etc.) and levels of prevention (universal, selective, indicated) ( 78 – 80 ).

The advanced technologies and novel interventions have broadened the scope of MH promotion and prevention, such as addressing the mental health issues of individuals with chronic medical illness ( 81 , 82 ), severe mental disorders ( 83 ), children and adolescents with mental health problems, and geriatric population ( 78 ). Further, it has increased the accessibility and acceptability of such interventions in a non-stigmatizing and tailored manner. Moreover, they can be integrated into the routine life of the individuals.

For instance, Internet-and Mobile-based interventions (IMIs) have been utilized to monitor health behavior as a form of MH prevention and a stand-alone self-help intervention. Moreover, the blended approach has expanded the scope of MH promotive and preventive interventions such as face-to-face interventions coupled with remote therapies. Simultaneously, it has given way to the stepped-care (step down or step-up care) approach of treatment and its continuation ( 79 ). Also, being more interactive and engaging is particularly useful for the youth.

The blended model of care has utilized IMIs to a varying degree and at various stages of the psychological interventions. This includes IMIs as a supplementary approach to the face-to-face-interventions (FTFI), FTFI augmented by behavior intervention technologies (BITs), BITs augmented by remote human support, and fully automated BITs ( 84 ).

The stepped care model of mental health promotion and prevention strategies includes a stepped-up approach, wherein BITs are utilized to manage the prodromal symptoms, thereby preventing the onset of the full-blown episode. In the Stepped-down approach, the more intensive treatments (in-patient or out-patient based interventions) are followed and supplemented with the BITs to prevent relapse of the mental illness, such as for previously admitted patients with depression or substance use disorders ( 85 , 86 ).

Similarly, the latest research has developed newer interventions for strengthening the psychological resilience of the public or at-risk individuals, which can be delivered at the level of the home, such as, e.g., nurse family partnership program (to provide support to the young and vulnerable mothers and prevent childhood maltreatment) ( 87 ); family healing together program aimed at improving the mental health of the family members living with persons with mental illness (PwMI) ( 88 ). In addition, various novel interventions for MH promotion and prevention have been highlighted in the Table 2 .

Depiction of various novel mental health promotion and prevention strategies.

a/w, associated with; A-V, audio-visual; b/w, between; CBT, Cognitive Behavioral Therapy; CES-Dep., Center for Epidemiologic Studies-Depression scale; CG, control group; FU, follow-up; GAD, generalized anxiety disorders-7; IA, intervention arm; HCWs, Health Care Workers; LMIC, low and middle-income countries; MDD, major depressive disorders; mgt, management; MH, mental health; MHP, mental health professional; MINI, mini neuropsychiatric interview; NNT, number needed to treat; PHQ-9, patient health questionnaire; TAU, treatment as usual .

Furthermore, school/educational institutes-based interventions such as school-Mental Health Magazines to increase mental health literacy among the teachers and students have been developed ( 80 ). In addition, workplace mental health promotional activities have targeted the administrators, e.g., guided “e-learning” for the managers that have shown to decrease the mental health problems of the employees ( 102 ).

Likewise, digital technologies have also been harnessed in strengthening community mental health promotive/preventive services, such as the mental health first aid (MHFA) Books on Prescription initiative in New Zealand provided information and self-help tools through library networks and trained book “prescribers,” particularly in rural and remote areas ( 103 ).

Apart from the common mental disorders such as depression, anxiety, and behavioral disorders in the childhood/adolescents, novel interventions have been utilized to prevent the development of or management of medical, including preventing premature mortality and psychological issues among the individuals with severe mental illnesses (SMIs), e.g., Lets' talk about tobacco-web based intervention and motivational interviewing to prevent tobacco use, weight reduction measures, and promotion of healthy lifestyles (exercise, sleep, and balanced diets) through individualized devices, thereby reducing the risk of cardiovascular disorders ( 83 ). Similarly, efforts have been made to improve such individuals' coping skills and employment chances through the WorkingWell mobile application in the US ( 104 ).

Apart from the digital-based interventions, newer, non-digital-based interventions have also been utilized to promote mental health and prevent mental disorders among individuals with chronic medical conditions. One such approach in adventure therapy aims to support and strengthen the multi-dimensional aspects of self. It includes the physical, emotional or cognitive, social, spiritual, psychological, or developmental rehabilitation of the children and adolescents with cancer. Moreover, it is delivered in the natural environment outside the hospital premises, shifting the focus from the illness model to the wellness model ( 81 ). Another strength of this intervention is it can be delivered by the nurses and facilitate peer support and teamwork.

Another novel approach to MH prevention is gut-microbiota and dietary interventions. Such interventions have been explored with promising results for the early developmental disorders (Attention deficit hyperactive disorder, Autism spectrum disorders, etc.) ( 105 ). It works under the framework of the shared vulnerability model for common mental disorders and other non-communicable diseases and harnesses the neuroplasticity potential of the developing brain. Dietary and lifestyle modifications have been recommended for major depressive disorders by the Clinical Practice Guidelines in Australia ( 106 ). As most childhood mental and physical disorders are determined at the level of the in-utero and early after the birth period, targeting maternal nutrition is another vital strategy. The utility has been expanded from maternal nutrition to women of childbearing age. The various novel mental health promotion and prevention strategies are shown in Table 2 .

Newer research is emerging that has utilized the digital platform for training non-specialists in diagnosis and managing individuals with mental health problems, such as Atmiyata Intervention and The SMART MH Project in India, and The Allillanchu Project in Peru, to name a few ( 99 ). Such frameworks facilitate task-sharing by the non-specialist and help in reducing the treatment gap in these countries. Likewise, digital algorithms or decision support systems have been developed to make mental health services more transparent, personalized, outcome-driven, collaborative, and integrative; one such example is DocuMental, a clinical decision support system (DSS). Similarly, frameworks like i-PROACH, a cloud-based intelligent platform for research outcome assessment and care in mental health, have expanded the scope of the mental health support system, including promoting research in mental health ( 100 ). In addition, COVID-19 pandemic has resulted in wider dissemination of the applications based on the evidence-based psycho-social interventions such as National Health Service's (NHS's) Mind app and Headspace (teaching meditation via a website or a phone application) that have utilized mindfulness-based practices to address the psychological problems of the population ( 101 ).

Challenges in Implementing Novel MH Promotion and Prevention Strategies

Although novel interventions, particularly internet and mobile-based interventions (IMIs), are effective models for MH promotion and prevention, their cost-effectiveness requires further exploration. Moreover, their feasibility and acceptability in LMICs could be challenging. Some of these could be attributed to poor digital literacy, digital/network-related limitations, privacy issues, and society's preparedness to implement these interventions.

These interventions need to be customized and adapted according to local needs and context, for which implementation and evaluative research are warranted. In addition, the infusion of more human and financial resources for such activities is required. Some reports highlight that many of these interventions do not align with the preferences and use the pattern of the service utilizers. For instance, one explorative research on mental health app-based interventions targeting youth found that despite the burgeoning applications, they are not aligned with the youth's media preferences and learning patterns. They are less interactive, have fewer audio-visual displays, are not youth-specific, are less dynamic, and are a single touch app ( 107 ).

Furthermore, such novel interventions usually come with high costs. In low-resource settings where service utilizers have limited finances, their willingness to use such services may be doubtful. Moreover, insurance companies, including those in high-income countries (HICs), may not be willing to fund such novel interventions, which restricts the accessibility and availability of interventions.

Research points to the feasibility and effectiveness of incorporating such novel interventions in routine services such as school, community, primary care, or settings, e.g., in low-resource settings, the resource persons like teachers, community health workers, and primary care physicians are already overburdened. Therefore, their willingness to take up additional tasks may raise skepticism. Moreover, the attitudinal barrier to moving from the traditional service delivery model to the novel methods may also impede.

Considering the low MH budget and less priority on the MH prevention and promotion activities in most low-resource settings, the uptake of such interventions in the public health framework may be lesser despite the latter's proven high cost-effectiveness. In contrast, policymakers may be more inclined to invest in the therapeutic aspects of MH.

Such interventions open avenues for personalized and precision medicine/health care vs. the traditional model of MH promotion and preventive interventions ( 108 , 109 ). For instance, multivariate prediction algorithms with methods of machine learning and incorporating biological research, such as genetics, may help in devising tailored, particularly for selected and indicated prevention, interventions for depression, suicide, relapse prevention, etc. ( 79 ). Therefore, more research in this area is warranted.

To be more clinically relevant, greater biological research in MH prevention is required to identify those at higher risk of developing given mental disorders due to the existing risk factors/prominent stress ( 110 ). For instance, researchers have utilized the transcriptional approach to identify a biological fingerprint for susceptibility (denoting abnormal early stress response) to develop post-traumatic stress disorders among the psychological trauma survivors by analyzing the expression of the Peripheral blood mononuclear cell gene expression profiles ( 111 ). Identifying such biological markers would help target at-risk individuals through tailored and intensive interventions as a form of selected prevention.

Similarly, such novel interventions can help in targeting the underlying risk such as substance use, poor stress management, family history, personality traits, etc. and protective factors, e.g., positive coping techniques, social support, resilience, etc., that influences the given MH outcome ( 79 ). Therefore, again, it opens the scope of tailored interventions rather than a one-size-fits-all model of selective and indicated prevention for various MH conditions.

Furthermore, such interventions can be more accessible for the hard-to-reach populations and those with significant mental health stigma. Finally, they play a huge role in ensuring the continuity of care, particularly when community-based MH services are either limited or not available. For instance, IMIs can maintain the improvement of symptoms among individuals previously managed in-patient, such as for suicide, SUDs, etc., or receive intensive treatment like cognitive behavior therapy (CBT) for depression or anxiety, thereby helping relapse prevention ( 86 , 112 ). Hence, such modules need to be developed and tested in low-resource settings.

IMIs (and other novel interventions) being less stigmatizing and easily accessible, provide a platform to engage individuals with chronic medical problems, e.g., epilepsy, cancer, cardiovascular diseases, etc., and non-mental health professionals, thereby making it more relevant and appealing for them.

Lastly, research on prevention-interventions needs to be more robust to adjust for the pre-intervention matching, high attrition rate, studying the characteristics of treatment completers vs. dropouts, and utilizing the intention-to-treat analysis to gauge the effect of such novel interventions ( 78 ).

Recommendations for Low-and-Middle-Income Countries

Although there is growing research on the effectiveness and utility of mental health promotion/prevention interventions across the lifespan and settings, low-resource settings suffer from specific limitations that restrict the full realization of such public health strategies, including implementing the novel intervention. To overcome these challenges, some of the potential solutions/recommendations are as follows:

• The mental health literacy of the population should be enhanced through information, education, and communication (IEC) activities. In addition, these activities should reduce stigma related to mental problems, early identification, and help-seeking for mental health-related issues.
• Involving teachers, workplace managers, community leaders, non-mental health professionals, and allied health staff in mental health promotion and prevention is crucial.
• Mental health concepts and related promotion and prevention should be incorporated into the education curriculum, particularly at the medical undergraduate level.
• Training non-specialists such as community health workers on mental health-related issues across an individual's life course and intervening would be an effective strategy.
• Collaborating with specialists from other disciplines, including complementary and alternative medicines, would be crucial. A provision of an integrated health system would help in increasing awareness, early identification, and prompt intervention for at-risk individuals.
• Low-resource settings need to develop mental health promotion interventions such as community-and school-based interventions, as these would be more culturally relevant, acceptable, and scalable.
• Utilizing a digital platform for scaling mental health services (e.g., telepsychiatry services to at-risk populations) and training the key individuals in the community would be a cost-effective framework that must be explored.
• Infusion of higher financial and human resources in this area would be a critical step, as, without adequate resources, research, service development, and implementation would be challenging.
• It would also be helpful to identify vulnerable populations and intervene in them to prevent the development of clinical psychiatric disorders.
• Lastly, involving individuals with lived experiences at the level of mental health planning, intervention development, and delivery would be cost-effective.

Clinicians, researchers, public health experts, and policymakers have increasingly realized mental health promotion and prevention. Investment in Preventive psychiatry appears to be essential considering the substantial burden of mental and neurological disorders and the significant treatment gap. Literature suggests that MH promotive and preventive interventions are feasible and effective across the lifespan and settings. Moreover, various novel interventions (e.g., internet-and mobile-based interventions, new therapies) have been developed worldwide and proven effective for mental health promotion and prevention; such interventions are limited mainly to HICs.

Despite the significance of preventive psychiatry in the current world and having a wide-ranging implication for the wellbeing of society and individuals, including those suffering from chronic medical problems, it is a poorly utilized public health field to address the population's mental health needs. Lately, researchers and policymakers have realized the untapped potentialities of preventive psychiatry. However, its implementation in low-resource settings is still in infancy and marred by several challenges. The utilization of novel interventions, such as digital-based interventions, and blended and stepped-care models of care, can address the enormous mental health need of the population. Additionally, it provides mental health services in a less-stigmatizing and easily accessible, and flexible manner. More research concerning this is required from the LMICs.

Author Contributions

VS, AK, and SG: methodology, literature search, manuscript preparation, and manuscript review. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.898009/full#supplementary-material

• Accessibility Options:
• Skip to Content
• Skip to Search
• Skip to footer
• Office of Disability Services
• Request Assistance
• 305-284-2374
• High Contrast
• School of Architecture
• College of Arts and Sciences
• Miami Herbert Business School
• School of Communication
• School of Education and Human Development
• College of Engineering
• School of Law
• Rosenstiel School of Marine, Atmospheric, and Earth Science
• Miller School of Medicine
• Frost School of Music
• School of Nursing and Health Studies
• The Graduate School
• Division of Continuing and International Education
• People Search
• Class Search
• IT Help and Support
• Privacy Statement
• Student Life
• University of Miami
• Division of University Communications
• Office of Media Relations
• Miller School of Medicine Communications
• Hurricane Sports
• UM Media Experts
• Emergency Preparedness

Explore Topics

• Latest Headlines
• Arts and Humanities
• People and Community
• All Topics A to Z

Related Links

• Subscribe to Daily Newsletter
• Social Networks
• For the Media
• Find University Experts
• News and Info
• People and Culture
• Benefits and Discounts
• More Life@TheU Topics
• About Life@the U
• Connect and Share
• Faculty and Staff Events
• Student Events
• TheU Creates (Arts and Culture Events)
• Undergraduate Students: Important Dates and Deadlines
• Submit an Event
• Miami Magazine
• Rosenstiel School of Marine and Atmospheric Science
• Student Affairs
• More News Sites
• UM NEWS HOME

Psychology professor leads field in HIV and mental health research

By Adrianne Gonzalez 03-29-2024

Steven Safren , professor of psychology and director of the Center for HIV and Research in Mental Health (CHARM) at the University of Miami, was recognized by the Faculty Senate with the  2023–24 Distinguished Faculty Scholar Award for a lifetime of distinguished accomplishments in clinical practice and research. 

After joining the University of Miami faculty in 2015, Safren founded CHARM — an interdisciplinary center between the College of Arts and Sciences, Miller School of Medicine, and the School of Nursing and Health Studies. Funded by the National Institute of Mental Health, the center is designated as a full HIV/AIDS facility and one of seven in the nation.

Nominated for the Distinguished Faculty Scholar Award by Philip M. McCabe , professor and chair of the Department of Psychology, Safren is lauded for his exceptional proactivity, extensive funding success, and leadership in the field. “He is a truly exceptional scholar, teacher, and University citizen. He has my highest recommendation,” said McCabe.

Safren earned his Ph.D. in clinical psychology from the University at Albany State University of New York and trained at Massachusetts General Hospital, Harvard Medical School specializing in cognitive behavior therapy. 

Reflecting on his most memorable moments at the University of Miami, Safren emphasizes his pride in witnessing his students' achievements during the commencement ceremonies and believes that both students and graduates must narrow their focus. “ Find a piece of what you are studying that you really enjoy, and do more of that,” he shared.

The 2023–24 Faculty Senate Awards Ceremony will be held in person on Monday, April 8, at 5 p.m. on the Coral Gables Campus.  Learn more about the awards ceremony.

This profile is part of a 2023–24 Faculty Senate Awards series recognizing all awardees.

Faculty Affairs

Office of the provost, information and resources, browse people at um, faculty events, borwse upcoming events, faculty senate.

University of Miami News Sub

• Coral Gables , FL 33124
• 305-284-2211 305-284-2211
• UM News and Events
• Alumni & Friends
• University Hotline

Tools and Resources

• Academic Calendar
• Department Search
• Parking & Transportation
• social-facebook
• social-twitter
• social-youtube
• social-instagram

Copyright: 2024 University of Miami. All Rights Reserved. Emergency Information Privacy Statement & Legal Notices Title IX & Gender Equity Website Feedback

Individuals with disabilities who experience any technology-based barriers accessing the University’s websites or services can visit the Office of Workplace Equity and Inclusion .

This paper is in the following e-collection/theme issue:

Published on 29.3.2024 in Vol 26 (2024)

Effects of a Digital Mental Health Intervention on Perceived Stress and Rumination in Adolescents Aged 13 to 17 Years: Randomized Controlled Trial

Authors of this article:

There are no citations yet available for this article according to Crossref .

This paper is in the following e-collection/theme issue:

Published on 28.3.2024 in Vol 11 (2024)

Translating Suicide Safety Planning Components Into the Design of mHealth App Features: Systematic Review

Authors of this article:

• Kim Gryglewicz 1 * , MSW, PhD   ; 
• Victoria L Orr 2 * , BS   ; 
• Marissa J McNeil 2 * , MA   ; 
• Lindsay A Taliaferro 3 , MPH, PhD, CHES   ; 
• Serenea Hines 2 , MSW   ; 
• Taylor L Duffy 2 , BS   ; 
• Pamela J Wisniewski 4 * , PhD  

1 School of Social Work, University of Central Florida, Orlando, FL, United States

2 Center for Behavioral Health Research & Training, University of Central Florida, Orlando, FL, United States

3 Department of Population Health Sciences, University of Central Florida, Orlando, FL, United States

4 Department of Computer Science, Vanderbilt University, Nashville, TN, United States

*these authors contributed equally

Corresponding Author:

Kim Gryglewicz, MSW, PhD

School of Social Work

University of Central Florida

12805 Pegasus Drive HS I

Orlando, FL, 32816

United States

Phone: 1 14078232954

Email: [email protected]

Background: Suicide safety planning is an evidence-based approach used to help individuals identify strategies to keep themselves safe during a mental health crisis. This study systematically reviewed the literature focused on mobile health (mHealth) suicide safety planning apps.

Objective: This study aims to evaluate the extent to which apps integrated components of the safety planning intervention (SPI), and if so, how these safety planning components were integrated into the design-based features of the apps.

Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we systematically analyzed 14 peer-reviewed studies specific to mHealth apps for suicide safety planning. We conducted an analysis of the literature to evaluate how the apps incorporated SPI components and examined similarities and differences among the apps by conducting a comparative analysis of app features. An independent review of SPI components and app features was conducted by downloading the available apps.

Results: Most of the mHealth apps (5/7, 71%) integrated SPI components and provided customizable features that expanded upon traditional paper-based safety planning processes. App design features were categorized into 5 themes, including interactive features, individualized user experiences, interface design, guidance and training, and privacy and sharing. All apps included access to community supports and revisable safety plans. Fewer mHealth apps (3/7, 43%) included interactive features, such as associating coping strategies with specific stressors. Most studies (10/14, 71%) examined the usability, feasibility, and acceptability of the safety planning mHealth apps. Usability findings were generally positive, as users often found these apps easy to use and visually appealing. In terms of feasibility, users preferred using mHealth apps during times of crisis, but the continuous use of the apps outside of crisis situations received less support. Few studies (4/14, 29%) examined the effectiveness of mHealth apps for suicide-related outcomes. Positive shifts in attitudes and desire to live, improved coping strategies, enhanced emotional stability, and a decrease in suicidal thoughts or self-harm behaviors were examined in these studies.

Conclusions: Our study highlights the need for researchers, clinicians, and app designers to continue to work together to align evidence-based research on mHealth suicide safety planning apps with lessons learned for how to best deliver these technologies to end users. Our review brings to light mHealth suicide safety planning strategies needing further development and testing, such as lethal means guidance, collaborative safety planning, and the opportunity to embed more interactive features that leverage the advanced capabilities of technology to improve client outcomes as well as foster sustained user engagement beyond a crisis. Although preliminary evidence shows that these apps may help to mitigate suicide risk, clinical trials with larger sample sizes and more robust research designs are needed to validate their efficacy before the widespread adoption and use.

Introduction

Suicide is one of the leading causes of death in the United States, accounting for >45,000 deaths annually [ 1 ]. Over the last decade, suicide rates have doubled for youth aged 10 to 24 years [ 2 ] and have steadily increased for racial and ethnic minority youth [ 1 , 3 , 4 ]. Suicide ideation and attempt rates have also risen [ 5 , 6 ], especially among youth and minoritized populations [ 5 , 7 - 11 ]. Numerous studies have shown that untreated mental illness, limited or lack of available care, and low perceived need for mental health treatment are common, yet preventable, suicide risk antecedents [ 12 - 19 ]. Moreover, stigma, difficulties recognizing suicide warning signs, preferences for self-reliance and autonomy, fear of burdening others, and negative treatment experiences can negatively affect help-seeking intentions and engagement in mental health services [ 20 - 24 ].

Researchers have identified various suicide prevention strategies to reduce the public health problem of suicide [ 25 , 26 ]. Safety planning is an integral component of suicide care [ 27 ] and has been empirically validated for reducing suicidality [ 28 , 29 ]. The process of safety planning involves collaboration between a clinical and client, as well as with the at-risk individual and their support network. This means that the support network could also be part of the safety planning process [ 30 ]. Safety planning involves jointly identifying, problem-solving, and communicating strategies to keep an individual safe if a crisis arises. Core strategies focus on uncovering warning signs or triggers that precede an emotional event, identifying and reinforcing the use of healthful self-management strategies to cope with distress, encouraging the use of positive socialization strategies for distraction and support, creating a network of external support and professional contacts to solicit assistance and support, and reducing access to lethal means [ 31 ]. The individualized nature of creating a safety plan (ie, a written document detailing the plan to keep an individual safe during a crisis) allows the person at risk of suicide the ability to incorporate culturally relevant and meaningful strategies, thereby making these plans useful and relevant for diverse populations [ 30 , 32 ].

Suicide safety planning is a brief intervention that has been used in both acute and clinical settings [ 31 , 33 , 34 ] and as a self-help tool [ 35 ]. Overall, researchers have found this intervention to be feasible, acceptable, and useful to facilitate support and reduce suicide risk [ 32 , 33 , 35 - 37 ]. Researchers have found safety plans and related interventions, such as crisis response planning [ 38 ], to be effective in reducing the risk of hospitalization, increasing engagement in mental health treatment, and promoting the use of healthful coping strategies when used alongside other therapeutic approaches [ 33 , 34 , 36 , 39 , 40 ]. Although safety planning has shown initial success in reducing suicidal urges and offering a sense of hope to individuals in crisis [ 41 ], some clinicians and researchers have criticized this process [ 42 , 43 ]. For example, safety planning encourages clinicians to revisit and update safety plans with their clients over time [ 44 ], which can prove challenging if service use barriers prevent clients from reaccessing care or if clients misplace or throw away their paper-based safety plan.

Considering these challenges, mobile health (mHealth) technologies could offer a timely and effective solution to address some of the criticisms directed at traditional safety planning methods. mHealth, particularly the use of apps, represents a common tool used by consumers with access to mobile phones [ 45 , 46 ]. In addition, mHealth has garnered attention as a practical and convenient method for implementing mental health interventions [ 47 ], with increase in the quantity and functionality of applications and tools resulting in increased use [ 48 ]. In general, mHealth apps have been used to effectively help individuals identify and manage symptoms of various mental health problems and conditions such as depression, anxiety, substance abuse, posttraumatic stress, and eating disorders [ 49 , 50 ]. Thus, incorporating mHealth apps into mental health treatment and adjunctive interventions may prove beneficial.

Furthermore, incorporating mHealth apps into established evidence-based interventions may also serve as a culturally inclusive way of disseminating treatment to younger, more technologically savvy generations who also happen to demonstrate higher rates of suicidal thoughts and behaviors than adults [ 6 ]. mHealth apps may also help address service use barriers and risk factors (eg, stigma) that hinder individuals from seeking help and participating in treatment for suicidality. Combining suicide safety planning practices with mHealth apps may combat accessibility concerns as well, including a commonly reported flaw of the traditional intervention—the reliance on a paper format [ 35 ]. Given the widespread proliferation of mHealth apps for suicide prevention, there is a need to examine the components and features that have been incorporated into the design of suicide safety planning apps.

The purpose of this systematic literature review was to first assess the extent to which suicide safety planning mHealth apps integrated the 6 steps or components of a widely used safety planning intervention (SPI) developed by Stanley and Brown [ 31 ] (research question [RQ] 1). Next, we independently reviewed available mHealth suicide safety planning apps via download from iOS and Android app stores to assess the integration of SPI components and to categorize different app design features used to personalize the end users’ experience (RQ2). We also examined the evidence on the effectiveness of these apps in terms of usability, acceptability, app engagement, and suicide-related outcomes (RQ3). This review aims to synthesize the extant research to inform suicide prevention efforts, clinical practice, and future development of suicide safety planning mHealth apps.

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement guidelines ( Multimedia Appendix 1 [ 51 ]), a comprehensive systematic review of existing literature on suicide safety planning via mHealth apps was conducted. The process is described in the following sections.

Systematic Literature Review

Eligibility criteria.

The inclusion criteria for the reviewed research studies were as follows: (1) a primary focus on suicide safety planning involving the use of a mHealth app, (2) publication in a peer-reviewed article written in English, and (3) availability of the full text of the article. Studies were excluded if (1) the word suicide, safety plan, or app was not included in the title; (2) they included other forms of mHealth technologies as the primary focus (eg, web-based applications); (3) the apps were designed with safety planning as a secondary focus (ie, not exclusively for suicide safety planning, not intended as a crisis intervention, or use of safety planning as a secondary tool to other treatment modalities); and (4) they were part of other systematic reviews or meta-analyses. We included studies across the entire system development life cycle (eg, formative evaluations and 1 group pre-posttest designs) owing to limited research on the topic and the relatively recent emergence of such research.

Information Sources

The following 5 bibliographic databases were used to systematically review the literature: PsycINFO, PubMed, ACM Digital Libraries, Academic Search Premier, and ERIC. We limited our results to articles published between January 2000 and May 2023. All databases were last searched on July 2, 2023.

Search Strategy

We used the following keywords to search for the topic of interest in each scientific database: “Safety Plan*” AND (“Applications” OR “Apps”); (“Suicide” OR “Safety Plan*”) AND (“Applications” OR “Apps”); “Suicide Interven*” AND (“Applications” OR “Apps”); “Suicide Prevent*” AND (“Applications” OR “Apps”); “Suicide Contract” AND (“Applications” OR “Apps”); “mHealth” AND “Suicide”; “Crisis Response” AND “Plan*.” Asterisks were added to search for words that began with the preceding letters (eg, prevent*: prevent, prevention, and preventing). An example of the search strategy outlined above is provided in Multimedia Appendix 2 .

Selection Process

Citations obtained from electronic databases were imported into Zotero (version 6.0.16). Two reviewers (KG and VLO) independently screened the articles to remove duplicates and assessed inclusion and exclusion criteria by title and abstract. For articles about which the reviewers were uncertain after the title and abstract review, 4 reviewers independently analyzed the full-text articles to determine whether they met the inclusion criteria. The reviewers discussed discrepancies until they reached a consensus. The references of all articles that met the inclusion criteria were reviewed and cross-referenced for additional relevant articles. We included all eligible studies (N=14) in this systematic review ( Figure 1 [ 51 ]).

Data Collection Process

Data from eligible studies were analyzed using the Cochrane Collaboration’s data extraction template for included studies (version 1.8) [ 52 ]. We added study-specific items to the template to answer RQ1 and RQ2. Specifically, to answer RQ1, we reviewed articles describing each mHealth app and coded, using a dichotomous (yes or no) coding scheme, for the following SPI components: (1) personal warning signs, (2) coping strategies, (3) ways to distract oneself through social activities, (4) identification of and ways to access trusted individuals (eg, family and friends) for support, (5) identification of and ways to access community supports (eg, mental health professionals, nonmental health adult supports, crisis, or emergency services), and (6) information about keeping the environment safe (eg, restricting access to lethal means). To answer RQ2, we downloaded available mHealth apps via the Apple App Store or Google Play Store or contacted app developers to conduct an independent review of SPI components and app features described in the articles. Next, we created codes to describe app features, organized and categorized codes based on similarities, and generated 5 themes to capture the core aspects of features. To answer RQ3, we extracted both qualitative and quantitative findings reported on primary and secondary outcomes. We categorized the study outcomes into 3 main research themes.

Two reviewers coded 2 research articles to assess interrater reliability based on the coding template and made refinements as necessary (eg, added operational definitions to describe SPI components and provided examples of app features). Once finalized, the reviewers used the template to extract the data from the remaining studies. Data items included (1) general article information (eg, author, publication year, and country); (2) study methods (eg, aims and research design); (3) study characteristics (eg, sample size, sample demographics, and setting); (4) SPI intervention characteristics (RQ1); (5) mHealth app design features (RQ2) and primary and secondary outcomes (RQ3); and (6) study implications and future directions ( Multimedia Appendix 3 [ 42 , 43 , 53 - 64 ]). A similar process was used to independently code the SPI components and app features of the mHealth apps available for download.

Risk of Bias Assessment

The risk of bias for each study was assessed by 2 independent reviewers (KG and VLO) using Joanna Briggs Institute (JBI) appraisal tools for quasi-experimental [ 65 ] and qualitative research study designs [ 66 ]. For studies that included mixed methods designs, we used both tools as recommended by the JBI. Each appraisal tool used a rating scale with yes, no, unclear, and nonapplicable responses. The overall appraisal rating was based on the following categories: include, exclude, and seek further information. Disagreements between the reviewers were discussed until they reached a consensus ( Multimedia Appendix 4 [ 65 , 66 ]).

Synthesis of Results

Owing to the heterogeneity of the study designs, participants, and outcomes collected, we could not perform a meta-analysis of the identified studies in this review. Therefore, we present a narrative synthesis of the study findings.

Study Selection

The initial search of electronic databases and hand-searched references resulted in a total of 46,397 peer-reviewed articles. After duplicate records were removed, 21,151 studies remained. Titles were screened for relevancy (eg, relating to suicide, suicide safety planning, and mHealth apps), and 20,970 articles were excluded. A total of 181 abstracts were reviewed. Following full-text reviews of 54 articles, 40 articles were removed (15 studies did not include an mHealth app and 25 were not intended as a suicide safety planning app). A total of 14 articles met the inclusion criteria (refer to Figure 1 for breakdown).

Study Characteristics

The detailed study characteristics of the selected articles (N=14) are presented in Multimedia Appendix 3 . Most studies (12/14, 86%) were conducted outside the United States [ 42 , 53 - 63 ]. The year range of the selected articles was between 2015 and 2023.

Study Design

As shown in Tables 1 and 2 , a total of 7 mHealth suicide safety planning apps were studied across the 14 articles in our data set ( Multimedia Appendix 3 ). We classified the articles based on the research design (ie, formative feedback, usability assessment, single cohort pre-posttest, and random control trial protocol). Formative designs assessed SPI components and features to guide app development [ 43 , 56 , 61 , 64 ], whereas usability designs assessed interface design issues and functionality (eg, task difficulty and time to complete tasks) [ 55 , 60 , 61 , 64 ]. Other studies evaluated the acceptability or feasibility of a fully developed mHealth app [ 54 , 58 - 60 , 62 , 63 ]. Across these studies, participants rated the frequency and duration of app use; ease of navigation; and level of satisfaction, comfort, confidence, or engagement in using the app.

a N/A: not applicable.

Sample Characteristics

Across studies, the study sample varied in age, type of participant (eg, youth or adults at risk of suicide and clinicians collaborating with suicidal clients), and setting (eg, suicide prevention clinic and pediatric inpatient facility). Among studies that recruited participants to inform or evaluate mHealth suicide safety planning apps [ 43 , 54 - 56 , 58 - 64 ], the sample size ranged from 11 to 36 participants. However, after reporting dropout rates, sample sizes dropped to as low as 2 participants and as high as 22 participants.

Integration of SPI Components Within mHealth Apps

Most articles (5/7, 71%) describing the mHealth apps incorporated SPI components into the design of their apps [ 54 , 58 , 61 , 63 , 64 ] ( Table 3 ). Creating a safe environment from lethal means was the missing component in 29% (2/7) of the apps [ 55 , 56 ].

a BoMM: Brake of My Mind.

b SPC: safety planning component.

c SPC or app feature included in the app.

d MHP: mental health professional.

e SPC or app feature missing in the app.

f Denotes innovative app features aligned with SPI components.

g Feature included in the app that was not mentioned in the article.

We used the JBI quasi-experimental appraisal tool [ 65 ] to assess the risk of bias across 5 studies [ 55 , 58 - 60 , 63 ]. These studies did not include a control or comparison group, increasing the threat to internal validity. Pre- and posttest measures were used to assess the immediate effects of the mHealth apps. However, the lack of repeated outcome measures over time, selection bias (nonrandom samples), and small sample sizes pose a risk of bias within and across these studies.

The qualitative appraisal checklist tool [ 66 ] was used to assess the risk of bias in 4 studies [ 43 , 54 , 56 , 62 ]. Across 2 studies [ 43 , 54 ], the cultural or theoretical orientation of the researchers and their influence on the research process was unclear. These issues were noted in the other 2 studies [ 56 , 62 ] as well. In these studies [ 56 , 62 ], it was also difficult to identify the philosophical perspective and congruity between the research methods, data analysis, and interpretation. The studies included more of a description of the design of the apps and included general perceptions from stakeholders.

The remaining studies [ 61 , 64 ] were assessed using both the quasi-experimental and qualitative appraisal tools owing to their mixed methods designs. In both studies, it was unclear whether the researchers’ cultural or theoretical orientation, their influence on the research, and the adequate representation of the participants and their voices were addressed. Other key issues included the lack of a control or comparison group, nonrandom and small sample sizes, and the use of posttest measures to assess usability at only 1 time point. JBI appraisal results are included in Multimedia Appendix 4 .

On the basis of our independent review of available mHealth suicide safety planning apps, SPI components described in each article were verified in 71% (5/7) of the apps [ 54 , 56 , 58 , 61 , 63 ]. The app features described in the articles were also confirmed in these apps. App features not highlighted in the articles but found within the apps are listed in Table 3 . We were unable to verify SPI components and app features in 2 of the reviewed apps in the literature [ 55 , 64 ].

Comparative Analysis of SPI Components and App Features

In our analysis of the literature and available mHealth apps for download, we synthesized the commonalities of app features and categorized them into 5 broad themes: interactive features, individualized user experience, interface design, guidance and training, and privacy and sharing. These features are described in the following sections.

Interactive Features

Three of the suicide safety planning mHealth apps [ 54 , 56 , 61 ] allowed users to associate suicide warning signs or precipitating stressors with their personalized coping strategies (aligns with SPI 1 and 2 in Table 3 ). O’Grady et al [ 61 ] stressed the importance of including this feature in apps, as this functionality can serve to preemptively address an impending crisis before it fully manifests. Most of the suicide safety planning mHealth apps (6/7, 86%) also included social distractor features in which users had access to their phone’s camera with the ability to upload or view media content (eg, pictures, quotes, music, activities, videos, and inspirational stories; SPI 3) [ 54 , 55 , 58 , 61 , 63 , 64 ]. In the BackUp app [ 63 ], loved ones, trusted supports, and suicidal users were able to upload media and share content to inspire hope and distract users from negative thinking.

Each mHealth app also included a built-in feature for users to save and contact trusted individuals within their social support networks (SPI 4). Typically, users entered contact information into the mHealth app directly or linked to their contact directories. A unique feature of the MYPLAN app [ 54 ] allowed users to create prewritten messages that they could send to their social supports during times of distress. Although this feature was created to inform loved ones of the app user’s emotional state during a crisis, participants (ie, app users) noted concerns about messages being misunderstood, whereas relatives felt that messages could minimize emotional states or provide inaccurate information about the app user’s safety. All apps included the ability to access community supports such as mental health professionals (SPI 5). Three apps [ 54 , 55 , 61 ] included GPS capabilities, which enabled users to search for nearby counseling agencies or emergency services, and, after selecting a search result, users received directions for quick access (SPI 5 and 6). The ED-SAFE app [ 64 ] included a referral search engine that allowed users to find behavioral health care by specialty and zip code. Emergency service numbers, mostly displayed via a phone icon or brief words (eg, “Crisis”), were clearly visible (listed on all pages) in 57% (4/7) of the mHealth apps [ 56 , 58 , 61 , 63 ], which is the suggested ethical guideline from prior work [ 67 ]. Three apps did not include access to emergency service numbers on all pages but provided them somewhere else within the app [ 54 , 55 , 64 ].

Individualized User Experience

All apps (7/7, 100%) allowed users to continually add to or revise their safety plans. Examples included the addition of new warning signs, reasons for living, and identifying coping strategies. None of the apps maintained a historical record of the previous safety plans or provided a visual mechanism to track daily, weekly, or monthly patterns based on stressors encountered or coping strategies used. Other personalization aspects included the ability to enable or disable therapeutic modalities [ 61 ], the inclusion of web-based resources to take an aptitude and personality test [ 55 ], exercises to express moods [ 55 ], and mood tracking [ 55 , 56 , 61 ]. In addition, all apps had built-in features to make esthetic customizations, such as personalizing the home screen, changing the color palate, and adding background pictures [ 54 - 56 , 58 , 61 , 63 ]. In 57% (4/7) of the apps, notifications were enabled to remind users about using their safety plan or skills to practice [ 54 , 56 , 61 , 63 ].

Interface Design

Several studies used iterative feedback from content and app design experts to create easy-to-navigate interfaces [ 58 , 61 , 63 ]. To enhance the navigation experience, a simple layout, clear or user-friendly language, and accessibility features were important design considerations included in some mHealth apps [ 54 , 58 , 61 , 64 ]. For example, SafePlan ’s layout mimicked the paper version of the safety plan to better transition users from using the paper version to the app [ 61 ].

Guidance and Training

In-app tutorials or instructional videos were included in 86% (6/7) of the suicide safety planning mHealth apps [ 54 - 56 , 58 , 63 , 64 ]. Some of these tutorials focused on how to use the app, whereas others explained the safety planning process. For example, the BeyondNow app [ 58 ] included a video outlining the process of safety planning and links to other helpful information. The most extensive tutorials were seen in the companion app to ED-SAFE [ 64 ], where tutorials could be received from a female provider, a male community member, or an avatar. The mHealth suite of apps also included self-care education materials about suicidality, safety plans, and life plans. In addition, the BackUp app [ 63 ] provided supportive contacts with web-based information on ways to identify warning signs and strategies to talk with suicidal individuals. The Brake of My Mind app [ 55 ] included an introduction from the developer with additional web-based resources to increase app usability.

Privacy and Sharing

Researchers also highlighted app privacy and sharing capabilities as important features to consider when designing mHealth suicide safety planning apps. Given the personal nature of the information saved, most mHealth apps required a username and password to log in [ 54 - 56 , 61 , 63 , 64 ]. For example, ED-SAFE [ 64 ] used the username and password feature to verify user identity and connect information collected in the emergency department setting to the mHealth app. Other apps disabled GPS for location tracking or did not use external servers to store users’ information for privacy and security concerns [ 61 , 63 ]. Several apps (5/7, 71%) included features allowing users to share self-monitoring data or share safety plans with clinicians or trusted individuals [ 54 , 56 , 58 , 61 , 64 ]. For instance, ED-SAFE [ 64 ] allowed users to share safety plans as well as appointment information, self-care education, helplines, referrals, and distractions through password-protected privileges given to authorized family members.

mHealth App Evidence of Effectiveness

The qualitative and quantitative findings were categorized into 3 main research themes: app usability and acceptability, app use and engagement, and suicide-related outcomes.

App Usability and Acceptability Findings

Across 71% (10/14) of the studies [ 54 - 56 , 58 - 64 ] that assessed the initial usability or acceptability of mHealth suicide safety planning apps, stakeholders’ experiences testing the mHealth apps were generally positive. Four studies [ 55 , 60 , 61 , 64 ] included standard rating scales (ie, System Usability Scale [ 68 ]) to assess the perceived usability of their apps, and scores exceeded the minimum usability standards (ie, >70). The remaining studies used qualitative feedback from focus groups, case reports, and open-ended questionnaires. For example, in the study by Buus et al [ 54 ], participants found the MYPLAN safety planning app useful in recognizing patterns of impending crises and for reinforcing personalized strategies to cope with distress. In describing the benefits of the BeyondNow safety planning app, participants in the study by Melvin et al [ 58 ] reported developing a sense of hope and connection from using the app. Researchers have attributed these findings to the accessibility of the app and its customizable features. According to the authors, stakeholders regarded apps as highly intuitive, easy to use, and visually appealing interface in terms of the design [ 59 , 61 , 62 , 64 ].

App Use and App Engagement

Five studies examined app use over time [ 58 - 60 , 63 , 64 ]. Overall app engagement and use were minimal. Across 3 studies, >70% of the participants used the apps at least once during the testing period, which ranged from 1 to 10 weeks [ 58 , 59 , 63 ]. In the study by Melvin et al [ 58 ], 77% (17/22) of the participants reported using the mHealth app “occasionally” or “a lot,” including to make changes to safety plans. Most participants also reported using the mHealth app during a suicidal crisis (15/22, 68%) or when experiencing suicidal ideation (18/22, 82%). Increased frequency of app use during a crisis or among participants with high levels of suicide ideation was reported in studies by Pauwels et al [ 63 ] and Muscara et al [ 59 ]. Larkin et al [ 64 ] reported that 2 (40%) out of 5 participants reported downloading the ED-SAFE patient mHealth app after discharge. Low uptake rates were mostly attributed to the participants’ forgetfulness to download the app. Although most participants acknowledged the benefits of using mHealth suicide safety planning apps during times of crisis [ 58 , 63 ], participant feedback from the study by Muscara et al [ 59 ] suggested that participants did not believe or were unsure whether the use of the BeyondNow safety planning app could help them manage their symptoms or keep individuals safe during a crisis. Only 35% (6/17) of the participants favored using the app in the future. Conversely, participants in the study by Nuij et al [ 60 ] noted that easy access to the Backup mHealth app provided a sense of reassurance and helped to deter suicidal thoughts.

Suicide-Related Outcomes

Suicide-related outcomes were examined across 29% (4/14) of the small-scale pilot studies (with sample sizes ranging between 3 and 22) [ 55 , 58 , 59 , 63 ]. The study by Jeong et al [ 55 ] assessed the Theory of Planned Behavior constructs, including attitudes, subjective norms, perceived behavioral control, and intentions toward engaging in suicide attempts, using a pre-posttest design with a small (N=3) sample of adolescent survivors of suicide attempts. The results showed statistically significant changes in attitudes, perceived behavioral control, and intentions, suggesting that the suicide safety planning app helped to positively shift attitudes toward life and reduce beliefs and intentions to engage in self-harm behavior.

Suicide coping or resilience was evaluated in 2 studies using pre-posttest designs [ 58 , 59 ]. Both studies used the same safety planning app (ie, BeyondNow ) to examine the changes in protective factors. Melvin et al [ 58 ] found a statistically significant increase in suicide-related coping among youth and adult participants (n=22). This finding suggests an increase in knowledge and confidence to use internal coping strategies and external resources to manage suicide ideation. However, the researchers did not observe statistically significant changes in suicide resilience (ie, the perceived ability to manage suicidal thoughts and feelings). In contrast, Muscara et al [ 59 ] found a significant increase in 1 subscale of suicide resilience, emotional stability (ie, the ability to regulate emotions), among youth participants (N=17) in their study.

Suicidal ideation or self-harm behavior were measured in 3 studies [ 58 , 59 , 63 ]. In an open-label, single-group design, Melvin et al [ 58 ] found statistically significant reductions in both the severity and intensity of suicide ideation following exposure to an 8-week trial that evaluated the clinical effectiveness of using the BeyondNow suicide safety planning app as an adjunct to treatment as usual (ie, existing mental health services). In an evaluation of the same mHealth app, but with the addition of a personalized toolbox app (ie, BlueIce ), instead of treatment as usual, Muscara et al [ 59 ] also found a reduction in suicide ideation and self-harm behaviors (ie, attempts to harm oneself with and without suicidal intent). However, these findings were not conclusive or statistically significant owing to the small sample size and lack of a control group. Pauwels et al [ 63 ] found a similar, nonsignificant decrease in suicide ideation scores in a study examining pre-posttest changes following exposure to the BackUp suicide safety planning app. Although these studies provide some evidence of clinical utility, these researchers noted study limitations and the need for further evaluation using randomized controlled trials (RCTs).

Principal Findings

The primary aim of this study was to conduct a comprehensive analysis of the integration and inclusion of the SPI components developed by Stanley and Brown [ 31 ] in the design of mHealth suicide safety planning apps. The secondary aim was to synthesize and assess the research methods of studies that reported on the effectiveness of these apps. Implications of these findings and practical recommendations for future directions in mHealth suicide safety planning research are described in the following sections.

Integrating Components of Suicide Safety Planning Into mHealth Apps

Overall, most apps included the core components of the SPI developed by Stanley and Brown [ 31 ], such as the identification of suicide warning signs, coping strategies, and supportive persons. Therefore, the results from this review provide evidence of some level of successful integration of SPI components into mHealth suicide safety planning apps (RQ1). Lethal means safety was 1 component that was not incorporated in 2 of the apps reviewed. Reducing access to lethal means is a critical part of suicide safety planning [ 31 ] and warrants inclusion in mHealth apps as it brings attention to methods that could be used to attempt or die by suicide if not removed from a user’s environment.

An important aspect of suicide safety planning is access to one’s safety plan. In this review, having access to safety plans at any time [ 54 , 55 , 58 , 60 , 61 ] and being able to continually revise the plan were considered benefits over traditional paper-based safety planning. In some apps, users could create associations between different suicide safety planning components (SPCs; eg, triggers and coping strategies) to better contextualize their experiences and create actional plans for mitigating crises [ 54 , 56 , 61 ]. We recommend that additional linkages between the SPCs be included to further personalize users’ experiences.

Despite the integration of SPI components within mHealth suicide safety planning app designs, we also identified important gaps in the literature that warrant the attention of app designers, researchers, and mental health professionals who may use this type of technology within their clinical practice. For instance, researchers have consistently emphasized the importance of completing the initial safety plan alongside a knowledgeable clinician [ 42 , 54 , 58 , 61 ] to ensure that at-risk users and loved ones understand the components and purpose of a safety plan. However, many of the analyzed apps allowed users to complete the safety plan without the recommended clinical support, and in some cases, they lacked disclaimers. Therefore, additional guidance from a professional when using mHealth suicide safety planning apps would further serve to assist users and ensure that the safety planning process is carried out as intended.

This review also found that most of the apps did not go beyond the traditional SPCs of paper-based protocols to integrate more interactive features that could potentially improve adherence or engagement. For instance, daily or weekly check-ins have been shown to improve adherence in other mHealth contexts, such as for smoking cessation [ 69 ] and the management of schizophrenia [ 70 ]. Visualization graphs of patterns or trends in suicide warning signs, triggers, and coping behaviors logged over time may serve to increase engagement and improve outcomes, as visualizing behavior change over time has been recommended in other mHealth contexts [ 71 ], such as alcohol reduction [ 72 ]. Furthermore, other meaningful ways to actively and continuously engage one’s support contacts (eg, clinicians, parents, and family members) and to reinforce the use of healthful coping strategies would be an advantageous direction for future exploration in mHealth app design. Beyond general support contacts, prior research has found that parental support is a significant protective factor against youth suicide [ 73 , 74 ]. For youth, in particular, it may be advantageous to include parents, family members, or other trusted adults in the mHealth suicide safety planning process to increase uptake, enhance help-seeking and coping behaviors, and reinforce ways to keep one’s environment safe. However, future research would need to carefully design and evaluate such interventions to ensure they are effective before making these interventions widely available through the dissemination of mHealth apps for suicide safety planning.

Another variation across the apps was that some apps provided default values for suicide SPCs (eg, suggested coping strategies), whereas others did not. Therefore, an area of future research could be to study whether providing default values is beneficial or detrimental to the safety planning process. Finally, rather than training focused on the technical aspects of using the mHealth app, there is a need to include psychoeducation for suicide safety planning [ 75 ], especially related to coping strategies and lethal means restriction, which should be modeled as a collaborative process between at-risk users and their support systems [ 76 ].

Usability and Design Considerations for mHealth Suicide Safety Planning Apps

Overall, our review highlights three important recommendations to consider when designing safety planning mHealth apps (RQ2): the need to (1) encourage end user collaboration in the design and implementation of the intervention, (2) incorporate personalization or customization capabilities, and (3) develop appropriate privacy safeguards to prevent liability and address other safety concerns that may arise when integrating mental health care and technology. A key strength of most studies in our review was the interdisciplinary collaboration between app developers, computer scientists, and clinical researchers that facilitated the design, development, and evaluation of the various mHealth suicide safety planning apps. In addition, multiple stakeholders were included in the design process, including individuals at risk of suicide, clinicians, usability experts, parents, and extended family members. Only in 1 instance, end users engaged who were not considered part of the target population of at-risk users (eg, students). We strongly recommend that future research continue to include researchers from across multiple disciplines (eg, psychology, public health, social work, medicine, computer science, and human-computer interaction), intended end users, and mental health professionals across each stage of the research process. For instance, researchers from different disciplines may be able to raise important threats to validity during the research design process that could lead to more robust study designs.

A key weakness highlighted within several studies was limited uptake or sustained use of the mHealth suicide safety planning apps over time. Such findings shed suspicion on the feasibility of this type of intervention being effective outside of research, regardless of the high usability and acceptability ratings. Some studies attributed lack of use to the reduction of suicidal behaviors over time, but others suggested that the suicide safety planning process, as designed to be carried out within the apps, was only suited for in-crisis situations and not appropriate for sustained use over time. Although this may be the case, it is also possible that the lack of interactive or engaging features within the apps made them less appealing to users. Being able to customize and personalize app features may help to enhance the user’s experience and increase app engagement. Many of the apps included social distractions (ie, music and pictures) or other features, such as diary cards, which might help increase overall app engagement during noncrisis periods. However, as suicidality is episodic, future research should be conducted to understand how different modalities or features (eg, mood tracking, journaling, mindfulness, and art) could be combined with suicide safety planning in a complementary way for long-term use and engagement. Future work should also consider leveraging advanced technologies and assessments, such as artificial intelligence and ecological momentary assessments [ 77 , 78 ], that could be used to anticipate heightened suicide risk and prompt users to engage in the mHealth app suicide safety planning process when they need it most.

Threats to Validity and Inconclusive Clinical Outcomes Associated With the Use of mHealth Suicide Safety Planning Apps

This review provides some preliminary evidence suggesting that suicide safety planning via mHealth apps could be an easy-to-use mechanism to provide individualized care to those who may otherwise go unserved due to common treatment barriers (RQ3), such as poor accessibility to service providers, lack of knowledge about suicide, and stigmatizing beliefs about help seeking [ 20 - 24 ]. At the same time, several threats to validity were uncovered by our assessment of risk bias, which can inform directions for future research. First, the robustness of the qualitative studies could be improved by stating the positionality of the researchers as well as a clear justification for the design of the mHealth apps. In some cases, articles were published by interdisciplinary teams, whereas in other cases, authors appeared to be from a single discipline (eg, computer science). Details about the composition and expertise of the research team are important, as well-implemented mHealth apps require interdisciplinary skill sets that span clinical, design-based, and technical expertise. Furthermore, the quantitative studies analyzed in our review were constrained by small sample sizes and no published RCTs. Among the pre-posttest studies conducted thus far, the clinical outcomes were inconclusive.

As such, RCTs with control groups, random assignment, and repeated measure outcomes assessed over time are needed in the future to evaluate the efficacy of using suicide safety planning mHealth apps compared with traditional paper-based safety plans [ 54 , 57 ], specifically related to reducing suicidal urges and behaviors and increasing use of coping strategies, as well as increased engagement in crisis and mental health services after the crisis. When doing so, researchers should recruit larger samples to ensure that the results are conclusive and can be generalized to the populations of interest. Furthermore, additional use metrics collected by the apps to track behavioral data associated with using different app features, such as user engagement with the 6 components of the SPI developed by Stanley and Brown [ 31 ], should be considered to better understand the potential mediating factors and behaviors that may influence clinical outcomes. Although the usability of the apps would be an important consideration to control for in future studies, it is necessary to move beyond such measures to determine the efficacy of mHealth apps in reducing suicide-related outcomes. In summary, the inclusion of more advanced study design methodologies and recommendations from lessons learned in future mHealth apps could serve to mitigate suicide risk and promote overall safety.

Limitations and Future Research

This systematic review included 14 peer-reviewed articles that designed, developed, and evaluated mHealth apps for suicide safety planning. There are several limitations of this study that should be addressed in future research. First, although our search process was comprehensive, it is possible that our keywords missed relevant articles and mHealth apps that should have been included in the review. Second, as many of the apps described in the articles were not publicly available for download, we requested access from the corresponding authors to conduct our review. In 2 cases, we were unable to gain access to the apps; therefore, our analysis was based on the description of those apps based on the published paper. As such, it may be possible that some features were not described in the original papers; thus, they were not included in our review. Future research should also consider conducting a systematic feature analysis of mHealth suicide safety planning apps that are publicly available for download but not studied within the peer-reviewed literature. Finally, a limited number of published RCTs at the time of the review restricted our ability to report on app use and suicide-related outcomes. As such, the main call-to-action from this review is the need to move beyond usability studies of newly developed mHealth suicide safety planning apps to robust clinical research designs to examine their efficacy in reducing suicidality among at-risk user populations.

Conclusions

Overall, most articles included in this review did little to evaluate the efficacy of mHealth suicide safety planning apps beyond usability assessments, signaling that these apps and corresponding research are still in their infancy in terms of validating clinical outcomes. Although most of the mHealth safety planning apps included in our review are not yet downloadable and broadly available for public use, the prevalence and popularity of mHealth suicide prevention and mental health support apps on the open market that have been deployed without rigorous peer-reviewed research is a concern. As such, there is a critical need for future research to ensure that mHealth apps for suicide safety planning integrate the lessons learned from empirical user-based and clinical research, are upheld to high ethical mental health care standards, and show clinical efficacy for reducing suicidality before the apps are released to end users. This is especially true given the delicate and important goal of preventing suicide among at-risk populations. It is promising to see that future randomized clinical trials have been registered to build upon this important preliminary work on mHealth suicide safety planning apps.

Acknowledgments

The authors would like to thank the following authors who provided access to their apps to conduct this feature analysis: Niels Buus and JL Stovgaard Larsen (MYPLAN); Glenn Melvin (BeyondNow); James Duggan (SafePlan); and Lea Meier, Caroline Gurtner, and François von Kaenel (SERO).

Conflicts of Interest

None declared.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

Example search strategy.

Detailed summary of the selected articles and key findings (N=14).

Critical appraisal results.

• WISQARS™ — web-based injury statistics query and reporting system. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/injury/wisqars/index.html [accessed 2023-01-11]
• Curtin SC, Heron M. Death rates due to suicide and homicide among persons aged 10-24: United States, 2000-2017. NCHS Data Brief. Oct 2019.(352):1-8. [ Medline ]
• Congressional Black Caucus. Ring the alarm: the crisis of black youth suicide in America. The National Action Alliance for Suicide Prevention. Dec 2019. URL: https://theactionalliance.org/resource/ring-alarm-crisis-black-youth-suicide-america [accessed 2023-06-05]
• Ramchand R, Gordon JA, Pearson JL. Trends in suicide rates by race and ethnicity in the United States. JAMA Netw Open. May 03, 2021;4(5):e2111563. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaylor EM, Krause KH, Welder LE, Cooper AC, Ashley C, Mack KA, et al. Suicidal thoughts and behaviors among high school students - youth risk behavior survey, United States, 2021. MMWR Suppl. Apr 28, 2023;72(1):45-54. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Plemmons G, Hall M, Doupnik S, Gay J, Brown C, Browning W, et al. Hospitalization for suicide ideation or attempt: 2008-2015. Pediatrics. Jun 2018;141(6):e20172426. [ CrossRef ] [ Medline ]
• Johns MM, Lowry R, Haderxhanaj LT, Rasberry CN, Robin L, Scales L, et al. Trends in violence victimization and suicide risk by sexual identity among high school students - youth risk behavior survey, United States, 2015-2019. MMWR Suppl. Aug 21, 2020;69(1):19-27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Peña JB, Wyman PA, Brown CH, Matthieu MM, Olivares TE, Hartel D, et al. Immigration generation status and its association with suicide attempts, substance use, and depressive symptoms among latino adolescents in the USA. Prev Sci. Dec 2008;9(4):299-310. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Johns MM, Lowry R, Andrzejewski J, Barrios LC, Demissie Z, McManus T, et al. Transgender identity and experiences of violence victimization, substance use, suicide risk, and sexual risk behaviors among high school students - 19 states and large urban school districts, 2017. MMWR Morb Mortal Wkly Rep. Jan 25, 2019;68(3):67-71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stone DM, Jones CM, Mack KA. Changes in suicide rates - United States, 2018-2019. MMWR Morb Mortal Wkly Rep. Feb 26, 2021;70(8):261-268. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jin SS, Dolan TM, Cloutier AA, Bojdani E, DeLisi L. Systematic review of depression and suicidality in child and adolescent (CAP) refugees. Psychiatry Res. Aug 2021;302:114025. [ CrossRef ] [ Medline ]
• Fortune S, Stewart A, Yadav V, Hawton K. Suicide in adolescents: using life charts to understand the suicidal process. J Affect Disord. Jun 2007;100(1-3):199-210. [ CrossRef ] [ Medline ]
• Moskos M, Olson L, Halbern S, Keller T, Gray D. Utah youth suicide study: psychological autopsy. Suicide Life Threat Behav. Oct 2005;35(5):536-546. [ CrossRef ] [ Medline ]
• Pagura J, Fotti S, Katz LY, Sareen J, Swampy Cree Suicide Prevention Team. Help seeking and perceived need for mental health care among individuals in Canada with suicidal behaviors. Psychiatr Serv. Jul 2009;60(7):943-949. [ CrossRef ] [ Medline ]
• Sankey M, Lawrence R. Brief report: classification of adolescent suicide and risk-taking deaths. J Adolesc. Dec 2005;28(6):781-785. [ CrossRef ] [ Medline ]
• Pisani AR, Wyman PA, Petrova M, Schmeelk-Cone K, Goldston DB, Xia Y, et al. Emotion regulation difficulties, youth-adult relationships, and suicide attempts among high school students in underserved communities. J Youth Adolesc. Jun 2013;42(6):807-820. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rajappa K, Gallagher M, Miranda R. Emotion dysregulation and vulnerability to suicidal ideation and attempts. Cogn Ther Res. Nov 9, 2011;36:833-839. [ CrossRef ]
• Hung P, Busch SH, Shih YW, McGregor AJ, Wang S. Changes in community mental health services availability and suicide mortality in the US: a retrospective study. BMC Psychiatry. Apr 25, 2020;20(1):188. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hester RD. Lack of access to mental health services contributing to the high suicide rates among veterans. Int J Ment Health Syst. Aug 18, 2017;11:47. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Curtis C. Youth perceptions of suicide and help-seeking: ‘they'd think I was weak or “mental”’. J Youth Stud. Jul 28, 2010;13(6):699-715. [ CrossRef ]
• Aguirre Velasco A, Cruz IS, Billings J, Jimenez M, Rowe S. What are the barriers, facilitators and interventions targeting help-seeking behaviours for common mental health problems in adolescents? A systematic review. BMC Psychiatry. Jun 11, 2020;20(1):293. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, Horwitz AG, Eisenberg D, Kramer A, King CA. Self-reported barriers to professional help seeking among college students at elevated risk for suicide. J Am Coll Health. 2013;61(7):398-406. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hom MA, Stanley IH, Joiner TE. Evaluating factors and interventions that influence help-seeking and mental health service utilization among suicidal individuals: a review of the literature. Clin Psychol Rev. Aug 2015;40:28-39. [ CrossRef ] [ Medline ]
• Wilson CJ, Deane FP. Adolescent opinions about reducing help-seeking barriers and increasing appropriate help engagement. J Educ Psychol Consult. 2001;12(4):345-364. [ CrossRef ]
• Ougrin D, Tranah T, Stahl D, Moran P, Asarnow JR. Therapeutic interventions for suicide attempts and self-harm in adolescents: systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry. Feb 2015;54(2):97-107.e2. [ CrossRef ] [ Medline ]
• Pfeiffer PN, King C, Ilgen M, Ganoczy D, Clive R, Garlick J, et al. Development and pilot study of a suicide prevention intervention delivered by peer support specialists. Psychol Serv. Aug 2019;16(3):360-371. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evidence. Zero Suicide. URL: https://zerosuicide.edc.org/evidence [accessed 2023-10-25]
• Ferguson M, Rhodes K, Loughhead M, McIntyre H, Procter N. The effectiveness of the safety planning intervention for adults experiencing suicide-related distress: a systematic review. Arch Suicide Res. 2022;26(3):1022-1045. [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, de Beurs D, Juniar D, Erlangsen A, Portzky G, et al. Safety planning-type interventions for suicide prevention: meta-analysis. Br J Psychiatry. Aug 2021;219(2):419-426. [ CrossRef ] [ Medline ]
• Conti EC, Jahn DR, Simons KV, Edinboro LP, Jacobs ML, Vinson L, et al. Safety planning to manage suicide risk with older adults: case examples and recommendations. Clin Gerontol. 2020;43(1):104-109. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract. May 2012;19(2):256-264. [ CrossRef ]
• Chesin MS, Stanley B, Haigh EA, Chaudhury SR, Pontoski K, Knox KL, et al. Staff views of an emergency department intervention using safety planning and structured follow-up with suicidal veterans. Arch Suicide Res. Jan 02, 2017;21(1):127-137. [ CrossRef ] [ Medline ]
• Stanley IH, Hom MA, Joiner TE. A systematic review of suicidal thoughts and behaviors among police officers, firefighters, EMTs, and paramedics. Clin Psychol Rev. Mar 2016;44:25-44. [ CrossRef ] [ Medline ]
• Stanley B, Brown GK, Brenner LA, Galfalvy HC, Currier GW, Knox KL, et al. Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry. Sep 01, 2018;75(9):894-900. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Labouliere CD, Stanley B, Lake AM, Gould MS. Safety planning on crisis lines: feasibility, acceptability, and perceived helpfulness of a brief intervention to mitigate future suicide risk. Suicide Life Threat Behav. Feb 2020;50(1):29-41. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, King CA, Biermann BJ. Motivational interviewing-enhanced safety planning for adolescents at high suicide risk: a pilot randomized controlled trial. J Clin Child Adolesc Psychol. 2019;48(2):250-262. [ CrossRef ] [ Medline ]
• Stewart KL, Darling EV, Yen S, Stanley B, Brown GK, Weinstock LM. Dissemination of the safety planning intervention (SPI) to university counseling center clinicians to reduce suicide risk among college students. Arch Suicide Res. 2020;24(sup1):75-85. [ CrossRef ] [ Medline ]
• Rudd MD, Joiner TE, Rajab MH. Treating Suicidal Behavior: An Effective, Time-Limited Approach. New York City, NY. Guilford Publications; 2001.
• Asarnow JR, Berk M, Hughes JL, Anderson NL. The SAFETY program: a treatment-development trial of a cognitive-behavioral family treatment for adolescent suicide attempters. J Clin Child Adolesc Psychol. 2015;44(1):194-203. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bryan CJ, Mintz J, Clemans TA, Leeson B, Burch TS, Williams SR, et al. Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army Soldiers: a randomized clinical trial. J Affect Disord. Apr 01, 2017;212:64-72. [ CrossRef ] [ Medline ]
• Little V, Neufeld J, Cole AR. Integrating safety plans for suicidal patients into patient portals: challenges and opportunities. Psychiatr Serv. Jun 01, 2018;69(6):618-619. [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, Ruwaard J, de Beurs D, Mokkenstorm J, van Duijn E, et al. Smartphone-based safety planning and self-monitoring for suicidal patients: rationale and study protocol of the CASPAR (Continuous Assessment for Suicide Prevention And Research) study. Internet Interv. May 05, 2018;13:16-23. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kennard BD, Biernesser C, Wolfe KL, Foxwell AA, Craddock Lee SJ, Rial KV, et al. Developing a brief suicide prevention intervention and mobile phone application: a qualitative report. J Technol Hum Serv. Oct 01, 2015;33(4):345-357. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moscardini EH, Hill RM, Dodd CG, Do C, Kaplow JB, Tucker RP. Suicide safety planning: clinician training, comfort, and safety plan utilization. Int J Environ Res Public Health. Sep 04, 2020;17(18):6444. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, Arango A, Healy N, King CA, Walton M. Augmenting safety planning with text messaging support for adolescents at elevated suicide risk: development and acceptability study. JMIR Ment Health. May 25, 2020;7(5):e17345. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anderson M, Jiang J. Teens, social media and technology 2018. Pew Research Center. May 31, 2018. URL: https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/ [accessed 2024-02-28]
• Comer JS. Introduction to the special series: applying new technologies to extend the scope and accessibility of mental health care. Cogn Behav Pract. Aug 2015;22(3):253-257. [ CrossRef ]
• Lal S. E-mental health: promising advancements in policy, research, and practice. Healthc Manage Forum. Mar 2019;32(2):56-62. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Turvey CL, Roberts LJ. Recent developments in the use of online resources and mobile technologies to support mental health care. Int Rev Psychiatry. 2015;27(6):547-557. [ CrossRef ] [ Medline ]
• Shingleton RM, Richards LK, Thompson-Brenner H. Using technology within the treatment of eating disorders: a clinical practice review. Psychotherapy (Chic). Dec 2013;50(4):576-582. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ryan R, Horey D, Oliver S, McKenzie J, Prictor M, Santesso N, et al. Cochrane Consumers and Communication Group Standard protocol text and additional guidance for review authors. La Trobe University. Mar 2019. URL: https:/​/opal.​latrobe.edu.au/​articles/​journal_contribution/​Standard_Protocol_Text_and_Additional_Guidance_for_Review_Authors/​6692969 [accessed 2023-05-19]
• Andreasson K, Krogh J, Bech P, Frandsen H, Buus N, Stanley B, et al. MYPLAN -mobile phone application to manage crisis of persons at risk of suicide: study protocol for a randomized controlled trial. Trials. Apr 11, 2017;18(1):171. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buus N, Erlangsen A, River J, Andreasson K, Frandsen H, Larsen JL, et al. Stakeholder perspectives on using and developing the MYPLAN suicide prevention mobile phone application: a focus group study. Arch Suicide Res. 2020;24(1):48-63. [ CrossRef ] [ Medline ]
• Jeong YW, Chang HJ, Kim JA. Development and feasibility of a safety plan mobile application for adolescent suicide attempt survivors. Comput Inform Nurs. Aug 2020;38(8):382-392. [ CrossRef ] [ Medline ]
• Meier L, Gurtner C, Nuessli S, Miletic M, Bürkle T, Durrer M. SERO - a new mobile app for suicide prevention. Stud Health Technol Inform. May 16, 2022;292:3-8. [ CrossRef ] [ Medline ]
• Melia R, Francis K, Duggan J, Bogue J, O'Sullivan M, Young K, et al. Using a safety planning mobile app to address suicidality in young people attending community mental health services in Ireland: protocol for a pilot randomized controlled trial. JMIR Res Protoc. Feb 21, 2023;12:e44205. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Melvin GA, Gresham D, Beaton S, Coles J, Tonge BJ, Gordon MS, et al. Evaluating the feasibility and effectiveness of an Australian safety planning smartphone application: a pilot study within a tertiary mental health service. Suicide Life Threat Behav. Jun 2019;49(3):846-858. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muscara F, Ng O, Crossley L, Lu S, Kalisch L, Melvin G, et al. The feasibility of using smartphone apps to manage self-harm and suicidal acts in adolescents admitted to an inpatient mental health ward. Digit Health. Nov 26, 2020;6:2055207620975315. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, de Beurs D, de Winter RF, Gilissen R, O'Connor RC, et al. The feasibility of using smartphone apps as treatment components for depressed suicidal outpatients. Front Psychiatry. Sep 27, 2022;13:971046. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Grady C, Melia R, Bogue J, O'Sullivan M, Young K, Duggan J. A mobile health approach for improving outcomes in suicide prevention (SafePlan). J Med Internet Res. Jul 30, 2020;22(7):e17481. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Skovgaard Larsen JL, Frandsen H, Erlangsen A. MYPLAN - a mobile phone application for supporting people at risk of suicide. Crisis. May 2016;37(3):236-240. [ CrossRef ] [ Medline ]
• Pauwels K, Aerts S, Muijzers E, De Jaegere E, van Heeringen K, Portzky G. BackUp: development and evaluation of a smart-phone application for coping with suicidal crises. PLoS One. Jun 21, 2017;12(6):e0178144. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Larkin C, Djamasbi S, Boudreaux ED, Varzgani F, Garner R, Siddique M, et al. ReachCare mobile apps for patients experiencing suicidality in the emergency department: development and usability testing using mixed methods. JMIR Form Res. Jan 27, 2023;7:e41422. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. Adelaide, South Australia. Joanna Briggs Institute; 2020.
• Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. Sep 2015;13(3):179-187. [ CrossRef ] [ Medline ]
• De Jaegere E, Portzky G, van den Berg M. Ethical guidelines for technology-based suicide prevention programmes. European Regions Enforcing Actions against Suicide. 2014. URL: https:/​/www.​researchgate.net/​publication/​260297369_Ethical_Guidelines_for_Technology-Based_Suicide_Prevention_Programmes [accessed 2024-02-28]
• Brooke J. SUS: a 'quick and dirty' usability scale. In: Usability Evaluation In Industry. Boca Raton, FL. CRC Press; 1996.
• McClure JB, Anderson ML, Bradley K, An LC, Catz SL. Evaluating an adaptive and interactive mHealth smoking cessation and medication adherence program: a randomized pilot feasibility study. JMIR Mhealth Uhealth. Aug 03, 2016;4(3):e94. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stubbe DE. Enhancing adherence: using mobile health technology to improve self-management for individuals with schizophrenia. Focus (Am Psychiatr Publ). Oct 2020;18(4):424-427. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res. Nov 16, 2018;20(11):e292. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bell L, Garnett C, Qian T, Perski O, Williamson E, Potts HW. Engagement with a behavior change app for alcohol reduction: data visualization for longitudinal observational study. J Med Internet Res. Dec 11, 2020;22(12):e23369. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hatchel T, Merrin GJ, Espelage AD. Peer victimization and suicidality among LGBTQ youth: the roles of school belonging, self-compassion, and parental support. J LGBT Youth. 2019;16(2):134-156. [ CrossRef ]
• Flouri E, Buchanan A. The protective role of parental involvement in adolescent suicide. Crisis. 2002;23(1):17-22. [ CrossRef ] [ Medline ]
• Abbott-Smith S, Ring N, Dougall N, Davey J. Suicide prevention: what does the evidence show for the effectiveness of safety planning for children and young people? - a systematic scoping review. J Psychiatr Ment Health Nurs. Oct 2023;30(5):899-910. [ CrossRef ] [ Medline ]
• Asarnow JR, Zullo L, Ernestus SM, Venables CW, Goldston DB, Tunno AM, et al. "Lock and protect": development of a digital decision aid to support lethal means counseling in parents of suicidal youth. Front Psychiatry. Oct 6, 2021;12:736236. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Auerbach RP, Lan R, Galfalvy H, Alqueza KL, Cohn JF, Crowley RN, et al. Intensive longitudinal assessment of adolescents to predict suicidal thoughts and behaviors. J Am Acad Child Adolesc Psychiatry. Sep 2023;62(9):1010-1020. [ CrossRef ] [ Medline ]
• Bhatt P, Liu J, Gong Y, Wang J, Guo Y. Emerging artificial intelligence-empowered mHealth: scoping review. JMIR Mhealth Uhealth. Jun 09, 2022;10(6):e35053. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by J Torous; submitted 14.09.23; peer-reviewed by HL Tam, B Leckning; comments to author 06.10.23; revised version received 19.12.23; accepted 31.12.23; published 28.03.24.

©Kim Gryglewicz, Victoria L Orr, Marissa J McNeil, Lindsay A Taliaferro, Serenea Hines, Taylor L Duffy, Pamela J Wisniewski. Originally published in JMIR Mental Health (https://mental.jmir.org), 28.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

Transforming the understanding and treatment of mental illnesses.

Información en español

Celebrating 75 Years! Learn More >>

• About the Director
• Advisory Boards and Groups
• Strategic Plan
• Offices and Divisions
• Careers at NIMH
• Staff Directories
• Getting to NIMH

Decoding the Mind: Basic Science Revolutionizes Treatment of Mental Illnesses

By Linda Brady, Margaret Grabb, Susan Koester, Yael Mandelblat-Cerf, David Panchision, Jonathan Pevsner, Ashlee Van’t-Veer, and Aleksandra Vicentic on behalf of the NIMH Division of Neuroscience and Basic Behavioral Science

March 21, 2024 • 75th Anniversary

Follow the NIMH Director on

For 75 years, NIMH has transformed the understanding and treatment of mental illnesses through basic and clinical research—bringing hope to millions of people. This Director’s Message, guest written by NIMH’s Division of Neuroscience and Basic Behavioral Science , is part of an anniversary series celebrating this momentous milestone.

The Division of Neuroscience and Basic Behavioral Science (DNBBS) at the National Institute of Mental Health (NIMH) supports research on basic neuroscience, genetics, and basic behavioral science. These are foundational pillars in the quest to decode the human mind and unravel the complexities of mental illnesses.

At NIMH, we are committed to supporting and conducting genomics research as a priority research area . As the institute celebrates its 75th Anniversary , we are spotlighting DNBBS-supported efforts connecting genes to cells to circuits to behavior that have led to a wealth of discoveries and knowledge that can improve the diagnosis, treatment, and prevention of mental illnesses.

Making gene discoveries

Medical conditions often run in families. For instance, if someone in your immediate family has high blood pressure, you are more likely to have it too. It is the same with mental disorders—often they run in families. NIMH is supporting research into human genetics to better understand why this occurs. This research has already led to the discovery of hundreds of gene variants that make us more or less likely to develop a mental disorder.

There are two types of genetic variation: common and rare. Common variation refers to DNA changes often seen in the general population, whereas rare variation is DNA changes found in only a small proportion of the population. Individually, most common gene variants have only a minor impact on the risk for a mental disorder. Instead, most disorders result from many common gene variants that, together, contribute to the risk for and severity of that disorder.

NIMH is committed to uncovering the role of genes in mental disorders with the aim of improving the lives of people who experience them. One of the many ways NIMH contributes to the discovery of common gene variants is by supporting the Psychiatric Genomics Consortium (PGC)   . The consortium of almost 1,000 scientists across the globe, including ones in the NIMH Intramural Research Program and others conducting NIMH-supported research, is one of the largest and most innovative biological investigations in psychiatry.

Global collaborations such as the PGC are critical to amassing the immense sample sizes needed to identify common gene variants. Data from the consortium’s almost one million participants have already led to transformative insights about genetic contributors to mental illnesses and the genetic relationships of these illnesses to each other. To date, studies conducted as part of the consortium have uncovered common variation in over a dozen mental illnesses.

In contrast to common gene variants, rare gene variants are very uncommon in the general population. When they do occur, they often have a major impact on the occurrence of an illness, particularly when they disrupt gene function or regulation. Rare variants involving mutations in a single gene have been linked to several mental disorders, often through NIMH-supported research. For instance, a recent NIMH-funded study found that rare variation in 10 genes substantially increased the risk for schizophrenia. However, it is important to note that genetics is not destiny; even rare variants only raise the risk for mental disorders, but many other factors, including your environment and experiences, play important roles as well.

Because of the strong interest among researchers and the public in understanding how genes translate to changes in the brain and behavior, NIMH has developed a list of human genes associated with mental illnesses. These genes were identified through rare variation studies and are meant to serve as a resource for the research community. The list currently focuses on rare variants, but NIMH plans to continue expanding it as evidence accumulates for additional gene variants (rare or common).

Moreover, mental illnesses are a significant public health burden worldwide . For this reason, NIMH investments in genomics research extend across the globe. NIMH has established the Ancestral Populations Network (APN) to make genomics studies more diverse and shed light on how genetic variation contributes to mental disorders across populations. APN currently includes seven projects with more than 100 researchers across 25 sites worldwide.

Connecting biology to behavior

While hundreds of individual genes have been linked to mental illnesses, the function of most of these genes in the brain remains poorly understood. But high-tech advances and the increased availability of computational tools are enabling researchers to begin unraveling the intricate roles played by genes.

In addition to identifying genetic variation that raises the risk for mental illnesses, NIMH supports research that will help us understand how genes contribute to human behavior. This information is critical to discovering approaches to diagnose, treat, and ultimately prevent or cure mental illnesses.

An NIMH-funded project called the PsychENCODE consortium   focuses on understanding how genes impact brain function. PsychENCODE is furthering knowledge of how gene risk maps onto brain function and dysfunction by cataloging genomic elements in the human brain and studying the actions of different cell types. The PsychENCODE dataset currently includes multidimensional genetic data from the postmortem brains of thousands of people with and without mental disorders.

Findings from the first phase of PsychENCODE were published as a series of 11 papers   examining functional genomics in the developing and adult brains and in mental disorders. A second batch of PsychENCODE papers will be published later this year. These findings help clarify the complex relationships between gene variants and the biological processes they influence.

PsychENCODE and other NIMH-supported projects are committed to sharing biospecimens quickly and openly to help speed research and discovery.

Facilitating these efforts is the NIMH Repository and Genomics Resource (NRGR)   , where samples are stored and shared. NRGR includes hundreds of thousands of samples, such as DNA, RNA, and cell lines, from people with and without mental disorders, along with demographic and diagnostic information.

Another NIMH initiative to connect risk genes to brain function is Scalable and Systematic Neurobiology of Psychiatric and Neurodevelopmental Disorder Risk Genes (SSPsyGene) . This initiative uses cutting-edge techniques to characterize the biological functions of 250 mental health risk genes—within the cells where they are expressed—to better understand how those genes contribute to mental illnesses. By systematically characterizing the biological functions of risk genes in cells, SSPsyGene will empower researchers to learn about biological pathways that may serve as new targets for treatment.

Genes also affect behavior by providing the blueprint for neurons, the basic units of the nervous system. Neurons communicate with each other via circuits in the brain, which enables us to process, integrate, and convey information. NIMH supports many initiatives to study the foundational role of neural networks and brain circuits in shaping diverse mental health-related behaviors like mood, learning, memory, and motivation.

For instance, studies supported through a basic-to-translational science initiative at NIMH focus on modifying neural activity to improve cognitive, emotional, and social processing  . Similarly, another new funding opportunity encourages studies in humans and animals examining how emotional and social cues are represented across brain circuits  to help address a core deficit in many mental disorders. These studies will increase understanding of the biological mechanisms that support behavior throughout life and offer interventions to improve these functions in healthy and clinical populations.

Developing treatments and therapeutics

The gene discovery and biology-to-behavior programs described here will lay the foundation for delivering novel therapeutics. To be prepared to rapidly implement findings from this research, NIMH supports several initiatives to identify behavioral and biological markers for use in clinical studies and increase our ability to translate research into practice.

Through its therapeutics discovery research programs , NIMH advances early stage discovery and development studies in humans and early efficacy trials for mental disorders. Taking these efforts a step further, NIMH supports the National Cooperative Drug Discovery/Development Groups for the Treatment of Mental Disorders , which encourage public–private partnerships to accelerate the discovery and development of novel therapeutics and new biomarkers for use in human trials. Moreover, NIMH is one of several institutes and centers in the NIH Blueprint Neurotherapeutics Network  , launched to enable neuroscientists in academia and biotechnology companies to develop new drugs for nervous system disorders.

For the treatments of tomorrow, NIMH is building a new research program called Pre-Clinical Research on Gene Therapies for Rare Genetic Neurodevelopmental Disorders  , which encourages early stage research to optimize gene therapies to treat disorders with prominent cognitive, social, or affective impairment. In parallel, NIMH’s Planning Grants for Natural History Studies of Rare Genetic Neurodevelopmental Disorders  encourage the analysis of pre-existing data from people with rare disorders to learn about disease progression and enable future clinical trials with these populations.

NIMH's Division of Neuroscience and Basic Behavioral Science supports many different research projects that help us learn about genes and gene functions, how the brain develops and works, and impacts on behavior. By investing in basic neuroscience, genetics, and behavioral research, we're trying to find new targets for treatment and develop better therapies for mental disorders. We're hopeful these efforts will lead to new ways to treat and prevent mental illnesses in the near future and, ultimately, improve the lives of people in this country and across the globe.