Introduction

Among youth (10–14 years old), suicide is the second leading cause of death and for those 15–24 years old, it was the third most common form of death in 20201. About 93% of Americans believe suicide is preventable2. One study3 found minimal evidence that suicide can be predicted or preventable at clinical settings. Paris noted that population-based suicide prevention strategies such as restricting access to fatal means, depression screening, and educating gatekeepers provide strong evidence for suicide prevention3. Schwartz-Lifshitz et al. concluded that public education, gatekeeper training, and primary care physician education were effective prevention strategies for suicide prevention4. Wyman et al. concluded that gatekeeper training for secondary school staff increased staffs’ knowledge of risk factors for suicide and suicide identification behaviors5.

Preventing youth suicide and gatekeeper training

Psychiatric factors commonly influence suicidal behavior6, while positive mental health reduces suicidal ideation and suicide attempts7. However, most who die by suicide did not seek mental health support before attempting suicide3. Thus, gatekeepers (family members, friends, educators, and social workers) could be positive forces countering suicidal ideation for those at risk. For youth, early identification is crucial in preventing suicide; parents and teachers are frontline gatekeepers who can identify and intervene in such circumstances8. Kern et al.9 noted “evidence suggests that school-based mental health services (SBMHS) have the highest likelihood of reaching youth in need” (p. 205). Are educators prepared adequately to help students with behavioral and mental health needs?

There is evidence10,11,12 that school personnel gain knowledge about students’ behavioral and mental health issues from interventions and training. Jorm et al.13 found South Australian teachers increased their knowledge and changed their beliefs about treatment of students’ mental health issues after training. Similar knowledge gains occurred among U.S. mental health professionals after training in suicide prevention14. However, Anderson et al.15 found limited evidence that mental health training programs improved teachers’ helping behavior or students’ mental health. Burnette et al.16 found limited evidence on the impacts of suicide prevention training that led to changes in gatekeepers’ beliefs or attitudes about suicide prevention. Torok et al. concluded there was no relationship between gatekeeper training and recognizing and/or responding to youth at risk of suicide; however, such training increased gatekeepers’ awareness and knowledge of youth at risk of suicide8.

Both Torok et al.8 and Anderson et al.15 found training activities on mental health and suicide prevention are tailored toward teachers more than for other school personnel (e.g., principals, counselors). We identified school-based mental health professionals and other personnel (e.g., coaches, librarians, nurses) are gatekeepers for youth with behavioral and mental health needs. For example, school-based mental health professionals and school administrators were significantly more concerned about students’ mental health than were teachers17,18,19 supporting the notion that school personnel are under-utilized in promoting positive behavioral and mental health in schools. A need exists to train school personnel to support students’ behavioral and mental health development and to determine if such training is effective.

Emotional backpack project (EBP)

Emotional Backpack Project was developed by the Center for School Behavioral Health (CSBH) at Mental Health America of Greater Houston (MHAGH) in response to Texas legislation20 requiring school districts to provide school personnel with training in mental health intervention and suicide prevention. EBP increases educators’ awareness of youth mental health issues and available resources for effective response to youth behavioral matters. The CSBH initiated the EBP in selected Texas school districts to help educators address youth trauma from events such as Hurricane Harvey. EBP objectives are to increase in awareness of suicide warning signs and increase in awareness of resources for help related to suicide risk.

Burnette et al.16 identified four factors that influence gatekeepers’ decisions to intervene when people are at risk of suicide: knowledge about suicide, beliefs and attitudes about suicide prevention, reluctance to intervene (especially stigma of mental illness), and self-efficacy to intervene. The authors16 emphasized the stigma of mental illness (i.e., negative stereotypes against people with mental illness) as potential causes to reduce gatekeepers’ decisions to intervene. Additionally, secondary traumatic stress resulting from interventions may reduce the likelihood of acting in future interventions. Lawson et al.21 noted that “educators need a trauma-informed literacy that enables self-care, facilitates and safeguards interactions with trauma-impacted students and colleagues, and paves the way for expanded school improvement models” (p. 421). As such, we deemed it important to consider self-care practices, including mindfulness and knowledge about self-care21 in gatekeeper training.

We identified a holistic approach for gatekeeper training that includes knowledge about suicide prevention, mental issues, and self-care. Specifically, EBP curricula address students’ behavioral and mental health needs through five intervention modules: Children’s Mental Health (CMH), Trauma Informed Classrooms (TIC), Youth Suicide Prevention (YSP), Trauma 102 (T102), and Self-Care and Mindfulness (SCM). EBP training is recognized by the Texas Education Agency as training that fulfills the requirements for mental health and suicide prevention training. Briefly, EBP interventions are:

  • Children’s Mental Health (CMH): defines mental health, symptoms of mental health disorders, and presents strategies for empathetic conversations.

  • Trauma Informed Classrooms (TIC): examines the effects of trauma on brain development, creating trauma informed spaces, and addresses grief in childhood.

  • Youth Suicide Prevention (YSP): presents myths and facts of suicide and intervention strategies for youth with suicidal ideation.

  • Advanced Trauma (T102): strategies to create trauma informed classrooms and campuses, including impacts of cultural awareness, equity, and racial/historical trauma in the classroom.

  • Self-Care and Mindfulness (SCM): addresses employee burnout, establishing achievable SMART goals, and developing self-care plans.

The purpose of this program evaluation was to assess the effectiveness of EBP training (i.e., five EBP interventions). We measured effectiveness as changes in training participants’ perceptions and/or knowledge about youth behavioral and mental health issues and suicide prevention. Kirkpatrick and Kirkpatrick’s22 four-level approach (i.e., reaction, learning, behavior, and results) is a popular model for evaluating learner outcomes in training programs23. We applied the learning level (i.e., learner outcomes) to measure changes in learners’ perceptions and/or knowledge. Kirkpatrick and Kirkpatrick’s22 model does not account for intervening variables such as trainer’s expertise or learner motivation or previous knowledge. Therefore, results are confined solely to training program participants.

Methods

A longitudinal trend survey24 was used because it permits data collection at different points. The target population included school personnel (e.g., teachers, principals, counselors), parents, and community members (i.e., child-serving agencies in school districts) affiliated with public or private schools (Table 1).

Table 1 Participants at Intervention Training (EBP Training), 2019–2020.
Full size table

The CSBH notified selected school districts (general announcements in school newsletters) about its professional development programs 1–2 months in advance (typically in mid-spring) to coincide with school districts’ end-of-year continuing education programs. An online consent script informed each participant that anonymous responses would be used in group format for technical reports and publications; consent required acknowledgement (I Agree) before entry to the training surveys. A combination of instruments were used for data collection. Pre-intervention data were collected during training session registration for the CMH, SCM, T102 interventions. Immediately after those sessions, participants answered post-intervention questions that were the same as the pre questions. In training settings where changes in perceptions were sought, retrospective only post-intervention questions were administered immediately after training to reduce response-shift bias. The retrospective post-then-pretest design takes less time to administer and potentially eliminates participants’ over/underestimation of knowledge, skills, or experiences that may be interpreted differently during a pretest/post-test design, in which the pretest may alert participants to potential training topics. Not all participants completed all assessments (i.e., pretest/post-test or retrospective “before and after” questions); therefore, only paired data are reported herein.

Multiple purposive (non-probability) samples were drawn from the EBP interventions (i.e., target population) for matching data only on the variables of interest. Due to space limitations, primary demographics are in Table 2 (missing and ‘decline to answer’ data were excluded on a case-by-case basis). Each intervention included more than 400 participants. A post-power analysis using G*Power 3.1.9.7. Faul et al.25 indicated a sample of 400 was large enough to detect within-subject interactions (i.e., post hoc matched pairs; α = 0.05) as small as d = 0.2 with 98% power. Participants, across all EBP modules, were characterized as female (~ 82%), white (~ 58%), teachers (~ 76%), who were 36–45 years old (~ 37%) (Table 2). They were 41 years (SD = 10.90) old.

Table 2 Participants’ Characteristics from EBP Intervention Trainings (2019–2020).
Full size table

The professional development activities and data reported herein are not considered research involving human subjects as defined by DHHS and/or FDA regulations. This study was exempt by an Institutional Review Board at Texas A&M University; it is a program evaluation.

Training intervention: emotional backpack project (EBP) training

The CSBH provided more than 150 training sessions, using EBP interventions in selected Texas school districts from 2019 to 2020. The EBP is a year-long, train-the-trainer program. MHAGH’s train-the-trainer workshops lasted nine hours with about 1.5 h/topic. After EBP training, trainees returned to their respective schools to teach other school staff, using EBP intervention materials, about youth behavioral and mental health needs.

In person training was delivered in schools before the novel Coronavirus 2019 lockdown and virtual training was delivered after March 2020. Three MHAGH training specialists led the in-person and virtual workshops. Training methods included experiential activities such as role play simulation, problem solving scenarios, and small group discussion about youth mental health issues.

Instrument and data collection

Perceptions and knowledge of EBP interventions were collected with instruments based on Kirkpatrick and Kirkpatrick’s22 second level of evaluation (i.e., learning); all instruments used Likert-type, 5-point scales (1 = strongly disagree, 5 = strongly agree; 1 = low, 5 = high; 1 = very uncomfortable, 5 = very comfortable; 1 = poor, 5 = excellent; 1 = never, 5 = always). Concerning perceptions, the CMH intervention included five pre/post items rated on a 5-point scale (strongly disagree to strongly agree). Cronbach’s alphas were 0.92 (pre) and 0.94 (post); both were highly reliable. Five retrospective post-then-pre items were rated with 5-point scales (poor to excellent; very uncomfortable to very comfortable). Cronbach’s alphas were 0.80 (before) and 0.79 (after), also deemed reliable. The TIC intervention used retrospective post-then-pre items with 5-point scales (low to high). Cronbach’s alphas were 0.89 (before) and 0.92 (after); deemed highly reliable. The YSP intervention included eight items (retrospective post-then-pre) with a 5-point scale (poor to excellent). Cronbach’s alphas were 0.93 (before and after); deemed highly reliable. T102 intervention included eight items for knowledge and comfort levels (retrospective post-then-pre) on a 5-point scale (poor to excellent). Cronbach’s alphas were 0.95 (before) and 0.97 (after); deemed highly reliable. The SCM intervention had three pre/post and ten retrospective post-then-pre items for knowledge and comfort on 5-point scales (never to always; poor to excellent). Cronbach’s alphas were 0.74 (pretest) and 0.97 (post-test) and 0.84 (before) and 0.95 (after) in retrospective post-then-pre items; all were highly reliable.

Knowledge was tested with questions from the EBP curricula. Twelve pre/post questions in T102 and 11 pre/post SCM knowledge questions were posed as single (4-item multiple-choice) or multiple response (check all correct) items. Testing effect threats were minimized by randomizing question and response order26. Participants’ responses were dichotomously coded (0 = incorrect, 1 = correct). Kuder-Richardson 20 (KR20) reliability coefficients were used because the KR20 appropriately determines internal consistency of measurements with dichotomous data24. KR20 for T102 was 0.71 (pre) and 0.64 (post); SCM yielded reliability 0.47 (pre) and 0.58 (post). Reliability was attributed to heterogeneity of items and item discrimination27, and limited number of questions. Results should not be generalized beyond the sample when KR20 scores are less than 0.6028. Knowledge results represent those in MHAGH’s EBP training.

Participants demographic characteristics were recorded during training session registration; not all participants provided demographic information in each session. All valid paired data were assigned unique numbers and personal information was removed before analyses. Descriptive and inferential statistics described the data. Alpha was 0.05 a-priori to determine if significant differences existed in pre/post perceptions or knowledge. Non-parametric tests (Friedman, McNemar) were used because analyses were conducted on individual Likert-type items that were not assumed to be on continuous scales and cannot be summed to represent EBP constructs or concepts.

Ethics declaration

All experimental protocols were reviewed by the Texas A&M University Institutional Review Board (#IRB2019-1309). All methods were carried out in accordance with relevant guidelines and regulations. The Texas A&M University Institutional Review Board determined this activity was a program evaluation and not research involving human subjects as defined by DHHS and FDA regulations.

Consent to participate

Informed consent was obtained from all subjects and/or their legal guardian(s) by acknowledging agreement with an online consent script before accessing online surveys. The Texas A&M University Institutional Review Board waived the process to document consent because this study did not involve more than minimal risk.

Results

Changes in perceptions of the EBP intervention

CMH intervention

Friedman tests revealed significant changes in respondents’ (n = 1065) perceptions of mental illnesses involving changes in health (Table 3). Perceptions changed from agree (Mdpre = 4.0) to strongly agree (Mdpost = 5.0) after the CMH intervention. Respondents’ comfort levels significantly increased from good (Mdbefore = 3.0) to very good (Mdafter = 4.0) for understanding whether a student may be developing a mental health challenge, and from comfortable (Mdbefore = 4.0) to very comfortable (Mdafter = 5.0) for approaching the student, school support staff, or administrators to discuss students’ mental health issues. Effect size magnitude was interpreted using WKendall, based on Cohen’s benchmarks29. Negligible (≤ 0.1), moderate (0.3), and large (≥ 0.5) effects were found.

Table 3 Significance and Effect Sizes for Changes in CMH Perceptions, 2019–2020 (n = 1063).
Full size table

TIC intervention

Friedman tests revealed significant increases in respondents’ (n = 1215) median rankings (Table 4) for self-perceived understanding of the influence of trauma on learning, increasing from average (Mdbefore = 3.0) to high (Mdafter = 5.0), and knowledge of trauma and/or grief informed classrooms, which increased from average (Mdbefore = 3.0) to above average (Mdafter = 4.0). Large (≥ 0.5) effect sizes were noted.

Table 4 Significance and Effect Sizes for Changes in TIC Perceptions, 2019–2020 (n = 1215).
Full size table

YSP intervention

Friedman tests revealed significant changes (Table 5) in respondents’ (n = 1160) perceived knowledge of the actions to take when a student is at risk of suicide and the signs and symptoms of suicide risk in students. Both items changed from good (Mdbefore = 3.0) to very good (Mdafter = 4.0) after the YSP intervention. Confidence in approaching a student, helping a student, asking a student about suicide, and approaching school counselors, administrators, or parents/guardians about students at risk of suicide changed from good (Mdbefore = 3.0) to very good (Mdafter = 4.0). Large (≥ 0.5) effect sizes were noted.

Table 5 Significance and effect sizes for changes in YSP perceptions, 2019–2020 (n = 1160).
Full size table

T102 intervention

Friedman tests revealed (Table 6) significant increases in respondents’ (n = 428) median rankings for self-perceived knowledge of the cultural considerations of trauma, historical trauma, racial trauma, implicit bias, and equity; all increased from good (Mdbefore = 3.0) to very good (Mdafter = 4.0) after the T102 intervention training in 2020. Comfort levels for responding to students with trauma, historical trauma, and/or racial trauma increased significantly from good to very good. Moderate (0.3) to large (≥ 0.5) effect sizes were noted.

Table 6 Significance and effect sizes for changes in T102 perceptions, 2020 (n = 428).
Full size table

SCM intervention

Friedman tests revealed (Table 7) significant differences (χ2(1) = 127.52, p = 0.001) in respondents’ (n = 401) median rankings for engaging in self-care, although it was ranked as often (Md = 4.0) in pre/post-intervention. Moderate (0.3) effects were found in respondents’ perceived engagement in mindfulness practice and practicing mindfulness with students. Both items increased from sometimes (Mdpre = 3.0) to often (Mdpost = 4.0). Respondents’ perceived knowledge and understanding increased significantly (p < 0.001) from good to very good for incorporating selfcare into daily routines, use of SMART goals in self-care, research that supports mindfulness, how to be attentive and aware and knowledge of practices to help oneself be present and attentive increased after the SCM intervention. Self-perceived knowledge of the impact stress has on health and how self-care can improve stress were rated as very good (Md = 4.0), before and after training. Respondents’ knowledge and understanding of the role of the vagus nerve, as described in EBP training materials, increased significantly (χ2(1) = 200.84, p = 0.001) from fair to good after the SCM intervention.

Table 7 Significance and Effect Sizes for Changes in SCM Perceptions, 2020 (n = 401).
Full size table

Changes in knowledge

T102 intervention

An exact McNemar’s test revealed (Table 8) significant increases in proportions (n = 428) of correct responses for trauma-informed schools are consistent, predictable, positive (p = 0.001) and safe (p = 0.01). Knowledge gains ranged from ~ 4 to ~ 29%. Significantly more correct responses were chosen for “trauma informed educators foster cultural awareness and promote equity” (p = 0.001) after the T102 training. Knowledge gains ranged from ~ 5 to ~ 7%.

Table 8 Frequencies and Chi-Square Results for Correct T102 Knowledge Items, 2020 (n = 428).
Full size table

SCM intervention

An exact McNemar’s test showed (Table 9) significant (p = 0.001) increases (8.5%) in correct responses for the post-intervention (n = 401) item “scholarly research finds that mindfulness practice increases attention.” Significantly (p = 0.001) more correct responses were selected for post-intervention items, “Diaphragmatic breathing activates [the vagus nerve]” (12% increase), and “What does the S in SMART goals mean? [Specific—a goal must be focused]” (~ 14% increase).

Table 9 Frequencies and Chi-Square Results for Correct SCM Knowledge Items, 2020 (n = 401).
Full size table

Discussion

We evaluated the EBP training program in terms of changing school personnel perceptions (beliefs and attitudes) and knowledge about youth behavioral and mental health issues and suicide prevention. After training, respondents reported significant improvements in mental health literacy and knowledge of students’ behavioral and mental health issues. MHAGH’s trainings were “successful in meeting educators’ needs as well as enhancing educators’ confidence in effectively addressing youth mental health needs in schools” (12, p. 221). This study supports the notion that professional development for school personnel helping students cope with behavioral and mental health issues is time well spent.

We affirmed previous work13 about educators’ increased knowledge and changed beliefs about students’ mental health issues and increased their confidence in addressing such issues. Changes in beliefs about students’ mental health may lead to broader understanding of student behavior and academic achievement. We found significant increases in educators’ confidence to help students at risk of suicide. EBP intervention training could help educators learn how to detect earlier those students with suicidal ideation, which may lead to earlier treatment from mental healthcare professionals. However, increased awareness of students’ mental health issues and/or confidence to help students at risk of suicide does not specifically translate into active prevention of youth suicide8,15,16. Future post-training indicators of participants’ active practice of the lessons learned from EBP intervention training are needed to understand better if this training produces observable effects in preventing youth suicide.

Consistent with previous studies10,11,12,13, participants’ perceived knowledge of students’ behavioral and mental health issues increased after training. Self-care practices, including mindfulness and knowledge about self-care, are important components to reduce secondary traumatic stress21. Post-intervention results showed participants felt very good (confident) about approaching school counselors, administrators, or students to discuss youths’ at-risk behaviors. These findings may reduce educators’ secondary traumatic stress by encouraging discussion about at-risk students and fostering self-care practices needed after traumatic experiences. These trends also lend themselves to earlier interventions and better outcomes.

Educators’ responses to students can be harmful or healing and may depend on the teacher’s mental and emotional state. We encourage educators to have short- and long-term self-care plans to remain clear minded when responding to disruptive students. Lawson et al.21 noted that educators who interacted with traumatized students were themselves vulnerable to secondary traumatic stress. Dye et al.30 found that mindfulness training significantly helped counselors’ relaxation strategies. We advocate for schools integrating mindfulness practices into daily routines to support staff’s mental health and wellness.

After training, school-based mental health professionals’ confidence or comfort levels for approaching students to discuss mental health needs increased. School-based mental health professionals and administrators are underutilized resources in promoting positive mental health in schools17,18,19. Our interventions were attended by non-teachers, sometimes as much as 33% were school personnel from outside the classroom; increasing those participants’ confidence and comfort is needed when approaching students to discuss mental health needs. If more school personnel were confident in approaching students, those students might find support systems sooner, considering that youth (< 18 years) normally do not seek support for mental health15.

Participants were comfortable in approaching parents or guardians of students they were concerned about following the CMH intervention training and very comfortable approaching students or school support staff and administrators for the same reasons. We see a common, but potentially harmful gap in addressing students’ behavioral and mental health issues through a whole-community approach. Do teachers not discuss concerns about at-risk students with parents or guardians for fear of crossing a caregiver boundary? Hatton et al.31 reported secondary school teachers feared intervening with potentially suicidal students, while Ohrt et al.32 noted teachers’ training programs on mental health were missing communication skills building to interact with students with mental health issues. Our findings extend the lacking communication skills’ premise to include parents and/or guardians.

We believe less comfort in approaching parents or guardians was because of lacking communication skills or confidence as an advocate for student behavioral and mental health. It could be the result of unfamiliarity in discussing such matters with diverse families. Future training can build educators’ confidence and comfort in approaching parents or guardians by practicing improved communication skills. Although time-consuming, adequate training includes practicing communications, messaging, and listening skills. Role-play simulation, debriefing, and lessons learned reinforces communication strategies and increases teachers’ confidence in mental health training.

Limitations

Although this report addressed outcomes of a program evaluation, a limitation existed in not having a control group to compare the impact of participation vs. non-participation in the EBP training. Future attempts should use experimental or quasi-experimental designs to measure the impact of professional development activities more effectively. Limitations exist in using retrospective post-then-pretest evaluation questions. Retrospective assessments might have introduced threats that interfered with measuring knowledge and/or confidence33. An improved program evaluation design will reduce or control most threats to internal validity. We used Kirkpatrick and Kirkpatrick’s22 program evaluation model, which was limited solely to program participants and cannot be generalized to other populations. Additionally, greater emphasis on tracking the number of school districts that participated in the EBP training, as well as number and type of school personnel attendees, will expand understanding of the true effectiveness of this professional development program. Finally, future investigations of EBP trainings should include measurements of post-training indicators such as teacher intervention, suicide data, and other school data (e.g., youth who self-report increased stressors following traumatic events) to expand understanding of the practical effectiveness of this mental health intervention and suicide prevention program.

Conclusion

After EBP training, educators and school personnel reported having improved their knowledge and confidence in addressing students’ behavioral and mental health issues. Increased knowledge of mental illnesses can affect action because informed educators and school personnel can identify and refer students to mental health services. We recommend mental health training be conducted more frequently than one time annually and should include all school personnel. Training to build school personnel’s confidence in helping students with mental health issues and communication skills with primary caregivers is needed. It may help to study the implications of including cultural relevance in mental health training curricula to increase participants’ confidence for such roles. Confidence-boosting interventions can be designed for specific audiences (i.e., students, parents, educators, and school personnel) for use when communicating with at-risk students. Online intervention materials about students’ behavioral and mental health needs could help educators retain knowledge after in person or online training. Role-play simulation will improve educators’ confidence and communication skills for interacting with others to address students’ behavioral and mental health issues.

Students’ mental health issues were exacerbated during the novel Coronavirus 2019 (COVID-19) pandemic. Hill et al.34 found rates of suicide ideation and attempts were higher in 2020 compared to 2019, for youth aged 11 to 21 in a major Texas city. The onset of COVID-19 led to high unemployment, diminished income, and increased food and housing assistance needs35. These challenges compound trauma and toxic stress, leaving youth feeling unsafe, overwhelmed, and helpless. Educators and school personnel who are comfortable addressing such issues are crucial frontline responders for student mental health development. We advocate for increased school personnel training to build capacity for helping students cope with behavioral and mental issues in Texas.