Notes
Article history
The research reported in this issue of the journal was funded by the PHR programme as project number 17/151/05. The contractual start date was in January 2020. The final report began editorial review in November 2021 and was accepted for publication in May 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PHR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.
Permissions
Copyright statement
Copyright © 2024 Ponsford et al. This work was produced by Ponsford et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.
2024 Ponsford et al.
Chapter 1 Background
Introduction
Whole-school interventions aim to modify the school environment to promote health. 1 A subset aims to promote student commitment to school to prevent outcomes such as substance use (i.e. tobacco, alcohol and other drugs) and violence, which are important, intercorrelated outcomes2–5 often associated with disengagement from school. 6,7 Such interventions are informed by theories of change that postulate that interventions build student commitment to school, and therefore prevent substance use and violence by improving relationships within schools and between schools and local communities,8 for example via improving pedagogy, revising school policies, encouraging student volunteering or increasing parental involvement in school. This review synthesises evidence on such interventions. It categorises such interventions into subtypes, examines theories of change, explores factors affecting implementation and assesses effectiveness and cost-effectiveness in preventing violence and substance use, as well as improving educational attainment.
Description of the problem
Substance use and violence among young people remain important public health problems, hence our focus on them in this review. According to English surveys,9 rates of regular drinking have decreased in recent years, with only a small proportion (7%) of young people aged 11, 13 or 15 years reporting that they drank alcohol three or more times during the preceding month. About one-quarter of those aged 15 years reported that they had been drunk at least twice ever. 9 Alcohol has been suggested to be the most harmful substance in the UK. 10 Treating alcohol-related diseases costs the NHS in England an estimated £3.5B annually. 11 The annual societal costs of alcohol use in England are estimated at £21B. 12 Alcohol-related harms are strongly stratified by socioeconomic status. 13 Early initiation of alcohol use and excessive drinking are linked to later heavy drinking and alcohol-related harms14,15 and poor health. 16 Alcohol use among young people is associated with truancy, exclusion and poor attainment, unsafe sexual behaviour, unintended pregnancies, youth offending, accidents/injuries and violence. 17
Preventing young people from initiating smoking is another key public health objective, with 80,000 deaths due to smoking annually. 18 Rates of regular smoking have also decreased, with 3% of young people aged 11, 13 or 15 years reporting that they smoked. 9 Smoking has been estimated to cost the NHS £5.2B per year, and wider societal costs amount to £96B. 19,20 Of smokers, 40% start in secondary school21 and early initiation is associated with heavier and more enduring smoking and greater mortality. 22,23 Smoking among young people is a key driver of health inequalities. 21
Among UK 15- to 16-year-olds, 21% have used cannabis9 and 9% have used other illicit drugs. 22 Early initiation and frequent use of ‘soft’ drugs may be a pathway to later, more problematic, drug use. 24 Drugs such as cannabis and ecstasy are associated with increased risk of mental health problems, particularly among frequent users. 24–27 Young people’s drug use is also associated with accidental injury, self-harm, suicide28–30 and other ‘problem’ behaviours. 31–34
The prevalence, harms and costs of violence among young people indicate that addressing this is a public health priority. 35,36 The most recent evidence shows that, in England, 17% (24% of boys vs. 9% of girls) of young people aged 11, 13 or 15 years reported involvement in a fight two or more times in the preceding 12 months. 9 A UK study found that 10% of young people aged 11–12 years reported carrying a weapon and 8% reported attacking someone with intent to hurt them seriously. 37 By age 15–16 years, 24% of students report that they have carried a weapon and 19% reported attacking someone with the intent to hurt them seriously. 37 There are also associations between aggression and antisocial behaviours in youth and violent crime in adulthood. 38,39 In addition to leading to further health inequalities, the economic costs to society of youth aggression, bullying and violence are high. For example, the cost of crime attributable to conduct problems in childhood is estimated at about £60B a year in England and Wales. 40
Description of the intervention
There is increasing interest in whole-school interventions promoting student commitment to school as a means of addressing complex public health problems and improving educational attainment. Interest in such interventions reflects awareness that health education lessons struggle to find a place in school timetables and have patchy results that tend to dissipate with time. 41–44 It also reflects interest in socioecological determinants of health including the school environment. 45 If effective, such interventions might represent a pragmatic and efficient means of addressing multiple intercorrelated risk behaviours.
Theory of change
According to a previous review we conducted on the effects on student health of the school environment and interventions to address this,1 the theory of human functioning and school development46 provides the most comprehensive theory of how schools can influence student commitment to school and, consequently, students’ health behaviours. In the present review, we therefore use this theory to define our initial theory of change and inclusion criteria for the interventions examined. We also draw on this theory as a starting point for synthesising intervention theories, in the course of which we refine the theory of change. The starting theory of change is also used to help inform our categorisation of interventions, which, in turn, we use to inform our syntheses of evidence about effectiveness. In the discussion section, we reflect on what the evidence synthesised in this review suggests about the usefulness of this theory of change.
What then does the theory of human functioning and school organisation propose as to how school environments might be modified to increase student commitment to school and thereby prevent substance use and violence, and improve educational attainment? The theory proposes that, to promote students’ health, schools should help students build their capacity for practical reasoning and affiliation. Practical reasoning concerns the ability to think and reason. This allows them to make choices, including about their health. Capacity for affiliation involves a concern for other humans, and to experience mutually satisfying interactions and attachments. Such affiliation provides a sense of belonging and feeling of being socially supported, which can protect heath.
The theory proposes that students are more likely to develop these assets if they feel committed to two school ‘orders’. The school ‘instructional order’ is concerned with learning and involves the relay of knowledge and skills. The school ‘regulatory order’ is concerned with conduct and involves the relaying of values and beliefs. Committed students are theorised to feel connected to these orders, enabling the realisation of their capacities for practical reasoning and affiliation.
Students who do not commit to the instructional or regulatory orders are theorised to fall into three categories, depending on the orders to which they are not committed. Alienated students are disconnected from both, either not understanding or rejecting the instructional and regulatory orders. The theory proposes that working-class students are more likely to fall into this category because of the lesser cultural alignment between working-class values and the middle-class values of school. Detached students can meet the demands of the instructional order but do not understand or share the values of the school’s regulatory order. Finally, estranged students cannot meet the instructional order’s demands but share the values of the regulatory order. The theory suggests that schools can promote health by increasing the extent to which students commit to these orders, and thereby realise their capacities for practical reasoning and affiliation.
This is theorised to occur via a number of specific processes. First, schools can modify ‘classification’: the boundaries that exist between the school and the outside world, and boundaries within the school that occur between teachers and students, between students and between subjects. Weakening boundaries between the school and the surrounding community is theorised to increase student commitment via increasing alignment between the school culture and that of the community and families in which students live. This could occur via students volunteering in local communities or via parents being involved in school activities. Weakening boundaries between teachers and students, or among students, is theorised to occur through improved staff/student relationships and co-operation, which will promote greater insights of both students and staff into each other’s realities. Such co-operation and relationship-building could occur via staff involving students in classroom decisions or whole-school policy-making, or via school policies aiming to improve relationships, for example via changes to school discipline systems. 47 This is theorised to promote commitment and facilitate realisation of students’ capacity for practical reasoning and affiliation. Weakening boundaries between subjects is theorised to occur through cross-subject teaching and learning, which will increase student engagement in learning and facilitate the development of the capacity for practical reasoning.
Second, schools can increase student commitment by modifying ‘framing’: reducing teaching that is didactic and teacher-led, and increasing student input into learning. This is theorised to increase student commitment to the school instructional and regulatory orders by communicating the school’s commitment to students and their values and needs.
Thus, the theory proposes that schools engaging in these processes will engender student commitment to school and increase the extent to which students realise their potential for practical reasoning and social affiliations. The theory proposes that, because alienation and detachment are more likely to occur among working-class students, weakening classification and reframing are likely to be more important for schools serving working-class students.
Intervention components
A systematic review requires clear inclusion criteria for interventions that are applicable to all potentially relevant study reports. Rather than requiring interventions to be informed by theories of change with similar constructs to the theory of human functioning and school organisation (which could not be applied because of inconsistencies in how studies report these), we defined inclusion criteria for interventions in terms of intervention activities that aligned with this theory, as this will be more consistently and clearly described in study reports. Therefore, informed by the theory of human functioning and school organisation,46 this review focuses on whole-school interventions aiming to reduce violence or substance use via:
-
modifying teaching to increase student engagement in academic learning
-
enhancing student–staff relationships
-
revision of school policies that involves students and/or that goes beyond health or behaviour management policies
-
encouraging all students to volunteer in the community
-
increasing parental involvement in school.
In line with the theory of human functioning and school organisation, these actions are theorised to promote young people’s health by modifying the whole-school environment to engender student commitment to learning and to the school community, thus increasing their capacity for practical reasoning and affiliation and to make healthier choices (Figure 1).
Rationale for the current study
As mentioned, we previously conducted a systematic review of the effects of schools and school-environment interventions on a broad range of student health outcomes. 1 The review synthesised existing theory, identifying the theory of human functioning and school organisation as the most comprehensive theory of how schools might engage in whole-school actions to promote health. 1,46
This review synthesised evidence from trials of interventions modifying the school environment and studies of how school environments influenced health. These syntheses provided evidence that school environments that developed student commitment to school, and whole-school interventions that aimed to increase student commitment to school, were associated with reduced rates of alcohol use, smoking, drug use and violence among students. 48,49
The review was influential on research50 and policy,51 but is now a decade old and was exploratory in scope. Its inclusion criteria for interventions were not informed by a theory of change and it examined student outcomes across a breadth of health domains. It found only a few outcome studies and included no economic evaluations or evaluations assessing educational outcomes. It was thus limited in its ability to test whether or not the theory of human functioning and school organisation is a sound basis on which to inform intervention theories of change, and to determine the best ways for schools to promote student commitment and thereby reduce substance use and violence. The review also excluded interventions that included health education components so that the review could assess whether or not school environment action alone could affect health. With hindsight, it is clear that curriculum and school environment components are potentially synergistic (e.g. a social and emotional skills curriculum preparing students to engage in whole-school change52) and the decision to exclude such interventions led to the exclusion of many otherwise relevant studies.
Finally, the review was limited in not synthesising process evaluations, and therefore not being able to examine factors affecting implementation of interventions. Previous reviews have examined what factors influence the initial delivery and sustained implementation of school-based health interventions,53,54 reporting that key enablers are supportive senior management, alignment of the intervention with school ethos and priorities, positive pre-existing student and teacher attitudes, and parental support of interventions. Given the greater complexity of whole-school interventions as opposed to the largely curriculum-based interventions examined in previous reviews, a review of what factors affect the implementation of whole-school interventions is warranted.
A Cochrane review conducted at about the same time55 synthesised evidence on the effectiveness of ‘health-promoting schools’ interventions (defined as those comprising school environment, curricular and parent/community components). This reported significant effects of such interventions on bullying victimisation and tobacco use, as well as emerging evidence for effects on alcohol and drug use, bullying perpetration and other violence. Although presenting promising evidence, the review was limited by the inclusion of interventions with a variety of school environment components ranging from posters to changes in student participation in decisions. These lacked a common theory of change and this limited the review’s ability to test specific theories of change or recommend which specific whole-school interventions should be implemented. This review also did not synthesise process or economic evaluations.
Since these reviews were published, there has been an upsurge in evaluations of whole-school interventions aiming to prevent student substance use and violence by building student commitment to school. In the light of this and the limitations of earlier reviews, we undertook this new review to focus on such interventions. Unlike the previous reviews, this review focuses on interventions involving activities that align with a specific theory of change informed by the theory of human functioning and school organisation. 46 This was intended to ensure that the review could draw more specific conclusions about which approaches to whole-school change are likely to be effective in preventing student substance use and violence and achieving educational benefits. The review does not exclude whole-school interventions that also include health curricula for the reasons discussed previously.
Review aim and questions
The aim was to search systematically for, appraise the quality of and synthesise evidence to address the following research questions (RQs).
-
What whole-school interventions that promote student commitment to school to prevent student substance use and/or violence have been evaluated, what intervention subtypes are apparent and how closely do these align with the theory of human functioning and school organisation?
-
What factors relating to setting, population and intervention influence the implementation of such interventions?
-
Overall and by intervention subtype, what are the effects of such interventions on student substance use, violence and educational attainment?
-
What is the cost-effectiveness of such interventions, overall and by intervention subtype?
-
Are the effects of such interventions on student substance use and violence mediated by student commitment to school, or moderated by setting or population?
Review objectives
-
To conduct electronic and other searches.
-
To screen references and reports for inclusion in the review.
-
To extract data from and assess the quality of included studies.
-
To synthesise intervention descriptions to describe subtypes and alignment with theory.
-
To synthesise process evaluations to explore factors influencing implementation:
-
To consult with policy/practice and community stakeholders on the results of these analyses.
-
-
To synthesise outcome evaluations to examine effects, mediators and moderators:
-
To consult with policy/practice and community stakeholders on the results of these analyses.
-
-
To draw on the above work to draft and submit to the National Institute for Health and Care Research a report addressing our RQs.
Chapter 2 Review methods
Research design overview
We carried out a multimethod systematic review examining intervention types, theories of change, influences on implementation, outcomes and cost-effectiveness of whole-school interventions promoting student commitment to school to prevent substance use and/or violence, and improve educational attainment. The review followed existing criteria for review conduct and reporting of systematic reviews (e.g. Centre for Reviews and Dissemination56 and Preferred Reporting Items for Systematic Reviews and Meta-analyses). 57 The protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews) on 14 October 2019.
Inclusion criteria for the review
Types of participant
Studies were included for which children and young people aged 5–18 years attending mainstream school were the intervention targets.
Types of intervention
The review focused on whole-school interventions aiming to reduce student violence or substance use via actions aligning with the theory of human functioning and school organisation:
-
modifying teaching to increase student engagement in academic learning
-
enhancing student–staff relationships
-
revision of school policies that involves students and/or that goes beyond health or behaviour management policies
-
encouraging all students to volunteer in the community
-
increasing parental involvement in school.
We excluded studies of interventions that:
-
involved health or social and emotional skills curricula only
-
targeted selected students or parents rather than being universal interventions
-
addressed behaviour management in the classroom or school-wide without addressing student engagement or commitment to school
-
involved students as peer educators or peer social marketers without being involved in school policy- or decision-making
-
involved revising policies or procedures relating purely to health or behaviour management without student input.
Types of control
The review focused on treatment-as-usual, no-treatment or other active-treatment control groups.
Types of outcome
Substance use and violence are important intercorrelated outcomes often associated with lack of school commitment. 2–7 Studies focused on violence and/or substance use. We also synthesised evidence on educational attainment, but included studies were not required to focus on this. Our definition of violence includes interpersonal physical, emotional or social abuse. Substance use included use of tobacco, alcohol or other drugs. Outcome measures were quantitative and could be self- or teacher-reported via questionnaires or diaries, or drawn from clinical or administrative data. Outcome measures could draw on dichotomous, categorical or continuous variables. Behavioural outcomes could focus on the following: behaviours over a specific period, frequency, the number of episodes of a behaviour or an index constructed from multiple measures. Violence measures could combine indicators of behaviour and the upset or injury this caused. Measures could examine particular or general forms of behaviours or convictions. Violence measures could combine interpersonal violence with other forms of antisocial behaviour if the former constituted a majority of items. Educational outcomes could be assessed via research-administered tests or routine data on academic progress or performance in tests or exams. Economic analyses could examine the above outcomes and/or health-related quality of life.
Types of study
To address RQ1, we drew on descriptions of interventions and theories of change from included studies. To address RQ2, we included process evaluations that drew on quantitative and/or qualitative data to examine how intervention planning, delivery or receipt was affected by factors relating to interventions, populations or settings. To address RQs 3 and 5, we included cluster randomised controlled trials (RCTs) and/or quasi-experimental evaluations in which schools were allocated non-randomly to intervention and control groups. Studies addressing RQ4 included economic evaluations that related costs to outcomes or benefits.
Search methods for the identification of studies
Database search strategy
Search terms
A draft search strategy was compiled in the OvidSP MEDLINE database by an information specialist (JF). This included strings of terms, synonyms and controlled vocabulary terms (when available) to reflect:
-
population (children and young people aged 5–18 years attending school)
-
intervention (whole-school interventions aiming to prevent violence or substance abuse)
-
evaluation methods.
These concepts were combined using the Boolean operator AND.
Search terms were determined via a text analysis of 77 known articles featuring terminology relevant to the review,58–132 using NVivo 12 Plus software (QSR International, Warrington, UK). Titles, abstracts and keywords were imported and word frequency and cluster analyses were used to create an OvidSP MEDLINE search string that included common subject headings and incorporated relevant words in proximity. These were augmented by additional terms and synonyms so that the search terms adequately described the search topics. These were then combined together and tested systematically. 133 This search strategy was refined with the project team until the results retrieved reflected the scope of the project. When run in an education-focused database, rather than a health one, it became apparent that the sensitivity-to-specificity ratio of the search was too high, resulting in many irrelevant results. Therefore, the search strategy for concept 2 was narrowed. This search was peer reviewed by a librarian not involved with the project using the Peer Review of Electronic Search Strategies guidance. 134 The agreed OvidSP MEDLINE search was adapted for each database to incorporate database-specific syntax, controlled vocabularies and search-interface limitations. Details of the search strings used for each database can be found in Report Supplementary Material 1, and in the London School of Hygiene & Tropical Medicine’s data repository. 135
Databases
The following databases were searched in full between 16 and 27 January 2020:
-
ProQuest Applied Social Sciences Index and Abstracts (ASSIA) (1987 to 23 January 2020)
-
ProQuest Australian Education Index (complete database as of 23 January 2020)
-
EBSCOhost British Education Index (complete database as of 16 January 2020)
-
EBSCOhost Cumulative Index to Nursing and Allied Health Literature (CINAHL) Plus (complete database as of 16 January 2020)
-
Wiley Online Library Cochrane Library (issue 1 of 12, January 2020) was used to search for results from the following databases:
-
Cochrane Database of Systematic Reviews
-
Cochrane Central Register of Controlled Trials
-
-
Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) database of health promotion research (BiblioMap) (complete database as of 23 January 2020)
-
EPPI-Centre Database of Promoting Health Effectiveness Reviews (DoPHER) (complete database as of 23 January 2020)
-
OvidSP EconLit (1886 to 9 January 2020)
-
EBSCOhost Education Abstracts (HW Wilson) (complete database as of 17 January 2020)
-
ProQuest Education Database (complete database as of 24 January 2020)
-
EBSCOhost Educational Administration Abstracts (complete database as of 17 January 2020)
-
EBSCOhost Education Resources Information Center (ERIC) (complete database as of 17 January 2020)
-
OvidSP EMBASE® (1947 to 14 January 2020)
-
OvidSP Global Health (1910 to 2020, week 1)
-
OvidSP MEDLINE ALL (1946 to 14 January 2020)
-
OvidSP PsycInfo (1806 to 2020, week 1)
-
Elsevier Scopus® (complete database as of 16 January 2020)
-
OvidSP Social Policy & Practice (October 2019)
-
Clarivate™ Web of Science™ and Social Sciences Citation Index (1970 to present; data last updated on 9 January 2020)
-
EBSCOhost Teacher Reference Center (complete database as of 24 January 2020).
These databases were selected to retrieve literature from the fields of health and education. We amended the list of databases originally intended to be searched (see Appendix 1, Table 8, for deviations from, and clarifications of, protocol) on the advice, informed by initial pilot searches, of the information scientist (JF).
The searches were updated between 11 and 19 May 2021. The search strings used for the updated searches can be found in Report Supplementary Material 1. The updated searches used the following databases:
-
EBSCOhost CINAHL Plus (complete database as of 11 May 2021)
-
Wiley Online Library Cochrane Library (issue 4 of 12, April 2021) was used to search for results from the following databases:
-
Cochrane Database of Systematic Reviews
-
Cochrane Central Register of Controlled Trials
-
-
EPPI-Centre database of health promotion research (BiblioMap) (complete database as of 13 May 2021)
-
EPPI-Centre DoPHER (complete database as of 13 May 2021)
-
OvidSP EconLit (1886 to 29 April 2021)
-
EBSCO ERIC (complete database as of 19 May 2021)
-
OvidSP Embase (1947 to 10 May 2021)
-
OvidSP Global Health (1910 to 2021, week 18)
-
OvidSP MEDLINE ALL (1946 to 10 May 2021)
-
OvidSP PsycInfo (1806 to May 2021, week 1)
-
Elsevier Scopus (complete database as of 13 May 2021)
-
OvidSP Social Policy & Practice (January 2021)
-
Clarivate Web of Science and Social Sciences Citation Index (1970 to present; data last updated 13 May 2021).
Owing to COVID-19 restrictions, visitor access to libraries was not allowed. Therefore, it was not possible to update searches on the following databases:
-
ProQuest ASSIA
-
ProQuest Australian Education Index
-
EBSCOhost British Education Index
-
EBSCOhost Education Abstracts (HW Wilson)
-
ProQuest Education Database
-
EBSCOhost Educational Administration Abstracts
-
EBSCOhost ERIC
-
EBSCOhost Teacher Reference Center.
Search strategy for other literature sources
The following clinical trials registers were searched for relevant ongoing and unpublished trials on 27 January 2020:
-
ClinicalTrials.gov (complete database as of 27 January 2020)
-
EPPI-Centre Trials Register of Promoting Health Interventions (TRoPHI) (complete database as of 27 January 2020)
-
World Health Organization International Clinical Trials Registry Platform (ICTRP) (complete database as of 27 January 2020).
The searches for the following clinical trials registers were updated on 11 May 2021:
-
ClinicalTrials.gov (complete database as of 11 May 2021)
-
EPPI-Centre TRoPHI (complete database as of 11 May 2021).
Because the World Health Organization ICTRP was not returning results, this source could not be updated.
We attempted to search the US Institute of Education Sciences, What Works Clearinghouse on 27 January 2020, but the website was unavailable.
Search terms were derived from the OvidSP MEDLINE search compiled for database searching. Details of the search strings used for these can be found in Report Supplementary Material 1 and in the London School of Hygiene & Tropical Medicine’s data repository. 135 All trial details were examined for their relevance and associated papers were included if they met our inclusion criteria.
We also searched the following websites to identify relevant studies between 17 and 20 January 2020, with updated searches occurring between 20 and 25 May 2021:
-
Cambridge Journals Online (www.eifl.net/e-resources/cambridge-journals-online)
-
Centers for Disease Control and Prevention: Smoking and Tobacco Use (www.cdc.gov/tobacco/index.htm)
-
Child and Adolescent Health Research Unit (www.cahru.org/)
-
Childhoods Today (www.childwatch.uio.no/publications/journals-bulletins/childhoodstoday.html)
-
Children in Scotland (https://childreninscotland.org.uk/)
-
Children in Wales (www.childreninwales.org.uk/)
-
European Union Community Research and Development Information Service (https://cordis.europa.eu/)
-
Database of Education Research (EPPI-Centre) – not available
-
Drug and Alcohol Findings Effectiveness Bank (https://findings.org.uk/e-bank.php)
-
Google (Google Inc., Mountain View, CA, USA)
-
Google Scholar
-
Welsh Government (https://gov.wales/)
-
Scottish Government (www.gov.scot/)
-
Joseph Rowntree Foundation (www.jrf.org.uk/)
-
National Criminal Justice Reference Service (www.ncjrs.gov/)
-
National Society for the Prevention of Cruelty to Children (www.nspcc.org.uk/)
-
National Youth Agency (https://nya.org.uk/)
-
Northern Ireland Executive (www.northernireland.gov.uk/)
-
OpenGrey (www.opengrey.eu/)
-
Personal Social Services Research Unit (www.pssru.ac.uk/)
-
Project Cork (www.centerforebp.case.edu/resources/tools/project-cork-clinical-screening-tools; accessed 8 November 2021)
-
University College London-Faculty of Education and Society (UCL-IOE) Digital Education Resource Archive (https://dera.ioe.ac.uk/)
-
UK Clinical Research Network Portfolio (www.nihr.ac.uk/researchers/collaborations-services-and-support-for-your-research/run-your-study/crn-portfolio.htm)
-
University of Illinois at Urbana-Champaign (https://illinois.edu/)
-
US Center for Substance Abuse Prevention (www.samhsa.gov)
-
Social Issues Research Centre (www.sirc.org/)
-
The Campbell Library (www.campbellcollaboration.org)
-
The Children’s Society (www.childrenssociety.org.uk/)
-
Open Library (https://openlibrary.org/)
-
Schools and Students’ Health Education Unit Archive (https://sheu.org.uk/)
-
World Health Organization ICTRP (www.who.int/clinical-trials-registry-platform)
-
Young Minds: Child and Adolescent Mental Health (https://youngminds.org.uk/).
The following search terms were used: (school) AND (teaching OR teacher OR engagement OR engaging OR commitment OR relationships OR policies OR policy OR volunteer OR parent environment OR whole school OR health promoting school) AND (evaluation OR effectiveness OR outcomes OR process OR intervention OR programme OR program). Results were screened in batches of 50. When no relevant results were returned from the last batch of 50, researchers did not progress to the next batch of 50.
Subject experts in the field known to the research team were contacted to identify additional reports. See standalone project information for the experts contacted and a template of the e-mail sent to them (www.journalslibrary.nihr.ac.uk/phr/AHSN4485). We also searched reference lists from all reports to identify further studies. The protocol specified that we would hand-search journals that published included studies found only via reference checking, and not indexed on databases we had searched, but none met this criterion.
Information management and study selection
All citations identified by our 2020 searches were uploaded to EndNote X9 (Clarivate) for duplicate removal using existing methods. 136,137 De-duplicated results were then uploaded to EPPI-Reviewer (version 4.0) software (EPPI-Centre, University of London, London, UK). The updated search results were uploaded to the same EndNote library as those identified in 2020. Duplicates found within the results of the 2021 search were removed. Again, de-duplicated results were uploaded to EPPI-Reviewer software.
Two reviewers (CB and RP) piloted the screening of successive batches of the same randomly generated 50 titles/abstracts, discussing disagreements over inclusion and calling on a third reviewer (GJMT) when necessary. Once a batch-level agreement rate of > 90% was reached, the remaining references were screened on title and abstract for inclusion by a single reviewer (RP or CB). Full reports were obtained for references judged, based on title and abstract, as meeting our inclusion criteria or for which there was insufficient information to judge. Screening of full study reports was then carried out by two reviewers (CB and RP), applying a comparable dual piloting process before moving to independent screening. We maintained a record of the selection process for all screened material.
Data extraction
Two reviewers independently extracted data from included theory (CB and RP), process (CB and RP), outcome (CB and GJMT) and economic (CB and AM) evaluation reports, using existing tools. 138–141 When reviewers disagreed on data extraction, they met to resolve this, referring to a third reviewer when necessary.
For intervention descriptions, date were extracted on domains included in a standard framework. 141 For intervention theories of change, we extracted data on constructs, mechanisms, contextual contingencies affecting these and scientific theories informing theories of change. For all studies, we extracted information on the following: study details (study location, timing and duration; individual and organisational participant characteristics); study design and methods (design, sampling and sample size, allocation, blinding, control of confounding, accounting for data clustering, data collection, attrition, analysis); process evaluation findings and interpretation; outcome measures (timing, reliability of measures, intraclass correlation coefficients, effect sizes); relevant mediation and moderation analyses; and economic data (inputs and outputs relating to costs, consequences/ benefits, disaggregated by time period when appropriate). Pairs of reviewers independently entered data into EPPI-Reviewer 4 for each study. We would have involved a translator when necessary, but this issue did not arise. Data extraction tools for the theory of change and for the process, outcome and economic evaluations are provided as part of the standalone project information (www.journalslibrary.nihr.ac.uk/phr/AHSN4485).
When reports were incomplete and there was a risk of missing data affecting our analysis, we contacted authors to request additional information (see standalone project information for template: www.journalslibrary.nihr.ac.uk/phr/AHSN4485). If authors were not traceable or if the sought information was unavailable from the authors within 2 months, we recorded that the study information was missing in EPPI-Reviewer, and this was included in our risk-of-bias assessment.
Assessment of quality and risk of bias
Drawing on Sterne et al. ’s142 guidance, we reduced the effect of reporting bias by focusing synthesis on studies rather than publications. Following the Cho et al. 143 statement on redundant publications, we attempted to detect duplicate studies and, if multiple articles reported on the same data, these data were extracted only once. We minimised location bias by searching across multiple databases. We minimised language bias by not excluding articles based on language.
All included studies were subjected to quality assessment by two independent reviewers using existing tools. Chris Bonell and Ruth Ponsford assessed descriptions of theory and process evaluations, outcome evaluations were assessed by Chris Bonell and GJ Melendez-Torres and economic evaluations were assessed by Chris Bonell and Alec Miners. Each pair then met to compare their assessments, resolving any disagreements through discussion, when necessary calling on the judgement of a third reviewer.
Assessment of theories of change
We assessed the quality of descriptions of intervention theories of change using a modified version of the criteria developed in our previous systematic reviews,144–146 informed also by our prior work on realist evaluation methods. 147 The assessment focused on the extent to which the theory of change described a path from intervention to outcomes; the clarity with which constructs were defined; the clarity with which causal inter-relationships between constructs were defined; the extent to which underlying mechanisms were explained; and the extent to which the theory of change considered how mechanisms and outcomes could vary with context (see standalone project information for tool used: www.journalslibrary.nihr.ac.uk/phr/AHSN4485).
Assessment of process evaluations
We assessed the quality of process evaluations using the EPPI-Centre tool148 addressing the rigour of sampling, data collection, data analysis, the extent to which the study findings were grounded in the data, whether or not the study privileged student perspectives, and the breadth of findings and depth of findings (see standalone project information for tool used: www.journalslibrary.nihr.ac.uk/phr/AHSN4485). These assessments were used to assign studies to two categories of ‘weight of evidence’. First, reviewers assigned a weight (low, medium or high) to rate the reliability or trustworthiness of the findings (the extent to which the methods employed were rigorous/could minimise bias and error in the findings). Second, reviewers assigned an additional weight (low, medium or high) to rate the usefulness of the findings for addressing the RQs. Study reliability was judged as being high when steps were taken to ensure rigour in four or more assessment criteria, as medium when addressing only three and as low when addressing two or fewer. To achieve a rating of ‘high’ usefulness, studies needed to be judged to have privileged student perspectives and to present findings that achieve both breadth and depth. Studies that were rated as having ‘medium’ usefulness only partially met this criterion, and studies rated as having ‘low’ usefulness were judged to have sufficient, but limited, relevant findings.
Assessment of outcome evaluations
For RCTs, we assessed risk of bias within each included study using the tool outlined in the Cochrane Handbook for Systematic Reviews of Interventions. 138 For each study, reviewers judged the likelihood of bias in seven domains: sequence generation, allocation concealment, blinding (of participants, personnel or outcome assessors), incomplete outcome data, selective outcome reporting, other sources of bias (e.g. recruitment bias in cluster randomised studies) and intensity/type of comparator (see standalone project information for tool used: www.journalslibrary.nihr.ac.uk/phr/AHSN4485). Each study was subsequently identified as having a ‘high risk’, ‘low risk’ or ‘unclear risk’ of bias within each domain. For non-random evaluations, we assessed quality using the Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I) tool149 (see standalone project information: www.journalslibrary.nihr.ac.uk/phr/AHSN4485).
Assessment of economic evaluations
We assessed the quality of economic evaluations using an adapted version of the Drummond et al. 150 checklist. This required reviewers to answer questions regarding each study, ranging from the type of economic evaluation [e.g. cost–utility analysis (CUA)] to the time horizon and rationale for the choice of modelling approach. However, we altered the wording in one question to ensure that information particularly relevant to this review was extracted (see standalone project information for tool used: www.journalslibrary.nihr.ac.uk/phr/AHSN4485); 31 questions were listed in total.
Data analysis
Typology of intervention approaches
To categorise intervention subtypes (RQ1) we drew on intervention descriptions and theories of change using intervention component analysis. 151 Intervention component analysis is a systematic approach enabling identification of critical features of interventions. As a starting point, we took the intervention description outlined in Chapter 1, which was informed by the theory of human functioning and school organisation,46 to define intervention elements promoting student commitment to school. These included activities involving changes to teaching to increase student engagement in academic learning, enhancing student–staff relationships, revision of school policies that involves students and/or that goes beyond health or behaviour management policies, encouraging students to volunteer in the community and promotion of parental involvement in school. This set of definitions informed line-by-line coding of intervention descriptions and corresponding theories of change by two reviewers (CB and RP), who also used inductive coding to capture the full content of interventions, and refine and subdivide the a priori list of key intervention elements to better reflect the descriptions of interventions in included studies. When more than one report addressed the same intervention, reviewers drew on the descriptions of interventions and theories of change from all relevant reports to inform their analysis.
The two reviewers then met to discuss and refine their coding, accounting for inconsistency in the description of concepts across reports to agree a final, detailed list of key intervention elements and sub-elements that aligned with the descriptions of interventions evaluated in the included studies. Elements were also categorised according to the school subsystems (individual student, classroom or school) they targeted. Any disagreements were resolved through discussion, drawing on the judgement of a third researcher when necessary. The reviewers then worked together to describe interventions in terms of whether or not they included these key elements, creating detailed tables to map the presence and absence of different components contained in each intervention (see Chapter 4). These provided a visual resource to compare and contrast content across interventions. Although grouping interventions into discrete, non-overlapping categories was not straightforward, given the wide variety of intervention strategies used and level of overlap of these across interventions, drawing on this analysis, the two reviewers were able to identify key aspects of difference and similarity between interventions to group them into discrete categories and subcategories forming a final typology of intervention subtypes, described in Chapter 4.
Synthesis of theories of change
To synthesise theories of change (RQ1) we used a form of best-fit framework synthesis. 152 This approach is appropriate when seeking to understand the applicability of an existing conceptual model to a body of evidence and enables the building on of a priori models through the elaboration and incorporation of additional concepts from other sources included in a review. The method begins by defining a priori themes based on an existing conceptual framework and then coding data from included studies against these themes. When concepts from the included studies cannot be coded with the a priori codes, these are coded using inductive thematic analysis. This inductive coding is then used to augment, modify or elaborate the existing model to produce a refined conceptual model that better fits the evidence present.
For this synthesis, one reviewer (CB) reduced the theory of human functioning and school organisation46 to its key elements to form a set of a priori themes for use in the coding of theory reports. Two reviewers (CB and RP) undertook pilot analysis on two reports deemed to include high-quality descriptions of theory of change. The reviewers independently coded the reports using the a priori framework, generating new codes when concepts in theories of change were not captured by the a priori framework. Each reviewer also created memos to explain new codes. These new codes could reflect a rejection, augmentation, refinement or elaboration of concepts within the theory of human functioning and school organisation, based on reviewers’ interpretations and constant comparison of themes across the included theories of change. 153
The two reviewers then met to compare and contrast their application of a priori themes and the emergence of new codes, developing a refined set of themes before going on to code the remaining reports for each intervention, exploring the extent to which theories of change differed between intervention subtypes. Further analysis drew on the agreed set of themes, with reviewers continuing to augment and develop new codes as these arose during the analytic process, and again writing memos to explain these codes. At the end of this process, the two reviewers met again to compare and modify their code sets and application of these, thereby agreeing a final framework composed of a priori and new themes; inconsistencies and disagreements were through discussion, calling on the judgement of a third reviewer when necessary.
Then, drawing on concepts from meta-ethnography used in our previous reviews,145,146 the reviewers developed a synthesis of themes identified through coding for each intervention subtype. This involved identification of patterns of ‘reciprocal translation’ within subtypes whereby similar concepts were expressed across theories of change for different interventions, as well as cases of ‘refutational synthesis’ whereby concepts expressed in some descriptions of intervention theories of change conflicted with other descriptions. This enabled us to build an overall ‘line-of-argument synthesis’ describing two distinct theories of change across intervention subtypes, one of which elaborated the theory of human functioning and school organisation and one of which did not align with this prior theory (see Chapter 5).
When more than one report addressed the same intervention, reviewers used theory of change descriptions from all relevant reports to inform their analysis. The synthesis was also not restricted to reports judged as being of high quality. Instead, conclusions drawing on poorer-quality descriptions were given less interpretive weight.
Synthesis of process evaluations
We synthesised process evaluation findings and interpretations on factors influencing the implementation of interventions (RQ2), again using meta-ethnographic synthesis methods. As with earlier reviews,154 these were applied to textual reports, not only of qualitative research, but also of quantitative research (as it is not possible to synthesise quantitative findings from process evaluations using statistical pooling because of the heterogeneity of research aims, methods and measures). In the case of findings from quantitative elements, we coded author interpretations, first checking as part of quality assessment whether or not these aligned with the quantitative data presented. Meta-ethnography examined themes across sources, identifying cases of ‘reciprocal translation’, whereby similar concepts were expressed in different ways in different sources, and cases of ‘refutational synthesis’, whereby concepts from different sources contradicted one another. We then developed a ‘line-of-argument’ synthesis drawing together concepts from different sources to develop an overall account of factors influencing the implementation of interventions evaluated in included studies.
Second-order constructs (authors’ interpretations of qualitative data) were distinguished from first-order constructs (directly quoted data). The synthesis was not restricted to studies judged to be of high quality. Instead, conclusions drawing on poorer-quality reports were given less interpretive weight.
In terms of procedure, the following steps were taken for the synthesis of process evaluations. First, two reviewers (CB and RP) prepared detailed tables to describe the quality of each report, its empirical focus and study site/population. Second, the two reviewers undertook pilot analysis of two high-quality reports. They read and re-read the results from these reports, applying line-by-line codes to capture the content of the data. They then drafted memos explaining these codes. Coding began with in vivo codes that closely reflected the words used in the findings sections. The reviewers then grouped and organised codes, applying axial codes reflecting higher-order themes. The two reviewers then met to compare and contrast their coding of these first two high-quality studies, developing an overall set of codes from their discussion. Two reviewers went on to code the remaining studies drawing on the agreed set of codes, but developing new in vivo and axial codes as these arose from the analytic process, and again writing memos to explain these codes. At the end of this process, the two reviewers met to compare their code sets and memos. They identified commonalities, differences of emphasis and contradictions in the code sets with the aim of developing a single set of overarching themes drawing on the strengths of the two sets of codes, resolving any contradictions or inconsistencies and drawing on a third reviewer if necessary. Analysis produced tables demonstrating how first-, second- and third-order constructs related to one another, enhancing transparency about these emergent themes.
Analysis of process evaluations was also informed by May’s155 general theory of implementation. This was not part of our protocol, but we found that it provided a useful heuristic for interpreting and organising the emerging themes.
Synthesis of outcome evaluations
To address RQ3, we first produced a narrative account of the effectiveness of these types of interventions overall and by intervention subtype. This narrative synthesis was ordered by outcome then, within this, by age group, intervention subtype, follow-up time and study design. Outcomes were categorised into violence, smoking tobacco, drinking alcohol, using other drugs and academic attainment. We were not able to categorise by key stage as many interventions spanned multiple age groups and included multiyear longitudinal follow-up. This is explained further in Chapter 7 and is summarised in the deviations and clarifications to the protocol in Appendix 1, Table 8. Categorisation by intervention subtype was informed by our prior categorisation of intervention descriptions and theories of change (RQ1). For a description study characteristics and results, see Table 7. We then produced forest plots for each of our review outcomes, with separate plots for different outcomes and age groups, intervention subtypes and follow-up times (see Figures 11–47). Plots included point estimates and standard errors (SEs) for each study, such as risk ratios for dichotomous outcomes or standardised mean differences (SMDs) for continuous outcomes.
We then examined the extent of heterogeneity among the studies (as determined by both Cochran’s Q test and inspection of the I2-value). If an indication of substantial heterogeneity was determined (e.g. study-level I2-value of > 50%) that could not be explained through meta-regressions, we investigated this further using subgroup and sensitivity analyses. We then undertook meta-analysis to generate pooled estimates of intervention effects. We estimated separate models for substance use and violence. We examined substance use outcomes together in one analysis, as well as separated into smoking tobacco, drinking alcohol, other drug use and any ‘omnibus’ measures of substance use. We regarded follow-up times of up to 1 year and those of > 1 year post baseline as different outcomes, this being a deviation from our original protocol, so that follow-up times better aligned with those provided in the studies reviewed. We ran these models for interventions overall and, when sufficient studies were found, we ran separate models for different intervention subtypes and comparators. This categorisation was informed by our analysis of intervention descriptions and theories of change (RQ1).
When studies were found to be statistically heterogeneous, we used a random-effects model; otherwise, we used a fixed-effects model. When using the random-effects model, we conducted a sensitivity check by using the fixed-effects model to reveal differences in results. We considered using a robust variance estimation meta-analysis model to synthesise effect sizes. This was because outcome evaluations included multiple measures of conceptually related outcomes and robust variance estimation meta-analysis improves on previous strategies for dealing with multiple relevant effect sizes per study, such as meta-analysing within studies or choosing one effect size by including all relevant effect sizes, but adjusting for interdependencies within studies. 156 Unlike multivariate meta-analysis, it does not require the variance–covariance matrix of included effect sizes to be known. Where meta-analyses were performed, included pooled effect sizes were presented in forest plots, with the individual study point estimates weighted by a function of their precision.
Prior to synthesis, we checked for correct analysis by cluster, and report values of intracluster correlation coefficients, cluster size, data for all participants or effect estimates and SEs. When proper account of data clustering was not taken, we corrected for this by inflating the SE by the square root of the design effect. 156 When intracluster correlation coefficients were not reported, we contacted authors to request this information or imputed one, based on values reported in other studies. When imputation was necessary, we undertook sensitivity analyses to assess the impact of a range of possible values. In other instances of missing data (such as missing population information), it was not possible to include a study in a particular analysis if, for example, it was impossible to classify the population using our equity tool.
We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach as described in the Cochrane Handbook for Systematic Reviews of Interventions138 to present the quality of evidence and summary-of-findings tables. The downgrading of the quality of a body of evidence for a specific outcome was based on five factors: limitations of the study, indirectness of evidence, inconsistency of results, precision of results and publication bias. The GRADE approach specifies four levels of quality (high, moderate, low and very low). If sufficient studies were found, we drew funnel plots to assess the presence of possible publication bias (trial effect vs. SE). Although funnel plot asymmetry may indicate publication bias, this can be misleading with a small number of studies. In Chapter 7, we discuss possible explanations for any asymmetry in the review in the light of the number of included studies. We assessed the impact of risk of bias in the included studies via restricting analyses to studies deemed to be at low risk of selection bias, performance bias and attrition bias.
Finally, we undertook further work to examine mediation and moderation of effects (RQ5). Mediation analyses involved a narrative synthesis reporting whether or not, within studies, measures of student commitment to school appear to mediate intervention effects on student violence, substance use or educational attainment outcomes. Moderation analyses examined what factors relating to setting and population moderated intervention effects within and between studies. To examine within-study moderation, we narratively synthesised evidence from relevant subgroup analyses conducted within primary studies to explore what subgroup characteristics explain heterogeneity of effects within studies, assessing whether or not interactions are significant. To examine between-study moderation, we aimed to use meta-regression to examine what factors related to setting and population influenced intervention effectiveness157,158 (as long as there are not too many confounders or insufficient data, or if meta-regression is unable to account for interdependencies in complex interventions), or qualitative comparative analysis, adapted for use in research synthesis,159,160 to assess necessary and sufficient conditions related to setting and population for intervention effectiveness. Any meta-regression and qualitative comparative analyses would be exploratory, hypothesis-building analyses because these drew on observational rather than experimental comparisons.
Synthesis of economic evaluations
Measures of costs and indirect resource use and cost-effectiveness were summarised using tables. When information was available, the tables are presented by time horizon so that both the short-and longer-term economic effects could be identified. If measures of resource use had been judged to be sufficiently homogeneous across studies, these would have been synthesised using statistical meta-analysis,140 although the paucity of evidence found precluded this. Measures of costs, indirect resource use and cost-effectiveness were adjusted for currency and inflation to the current UK context. These data were used to inform a narrative synthesis of economic analyses and applicability to the UK context. We did not perform de novo economic modelling because the identified interventions and their outcomes were too heterogeneous.
Policy and practice consultation
We consulted a group of policy and practice stakeholders (representatives from Public Health England, Department of Health and Social Care, Department for Education, Association for Young People’s Health, Healthy Schools London, Education Endowment Foundation and the National Association of Head Teachers), and consulted a separate group of young people under the auspices of the Advice Leading to Public Health Advancement (ALPHA) Centre for Development, Evaluation, Complexity and Implementation in Public Health Improvement (DECIPHer) group of young researchers.
Both groups were consulted only once, rather than the planned two consultations. This was because of disruption to the project arising from the COVID-19 pandemic and the challenges this raised for those working in public health and education. Each group reviewed the results of our syntheses of intervention descriptions, and process, outcome and economic evaluations to inform any refinements to the analysis and drafting of the report. The groups considered and advised us on whether or not the evidence of effectiveness and cost-effectiveness overall and by subgroups suggested that it would be worth investing in the development of a new intervention to be evaluated in the UK. We name stakeholders by their organisation only and do not attribute specific comments to individuals or organisations.
Chapter 3 Results: included studies
Results of the search
In total, 126,180 references were identified from the electronic literature searches run in January 2020. Of these 63,438 (50%) were identified as duplicates and removed. The updated May 2021 search identified 105,777 results. Of these, 96,068 (91%) were duplicates or already retrieved by the earlier search. This left 9709 new references, giving a total of 72,451 references that were screened on title and abstract. The numbers of results pre and post de-duplication for the 2020 and 2021 searches are listed in Table 1.
Database name | Number of results retrieved in 2020 | Number of results once duplicates removed | Number of results retrieved in 2021 | Number of new results retrieved in 2021 once duplicates removed |
---|---|---|---|---|
ProQuest ASSIA | 7627 | 3444 | N/A | N/A |
ProQuest Australian Educational Index | 4738 | 4414 | N/A | N/A |
EBSCO British Education Index | 440 | 199 | N/A | N/A |
EBSCO CINAHL Plus | 6011 | 1728 | 7075 | 745 |
ClinicalTrials.gov | 991 | 935 | 1162 | 165 |
EPPI-Centre database of health promotion research (BiblioMap) | 0 | 0 | 0 | 0 |
EPPI-Centre DoPHER | 0 | 0 | 0 | 0 |
Wiley Online Library Cochrane Library (includes results from Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials) | 3316 | 268 | 3736 | 989 |
OvidSP EconLit | 223 | 208 | 268 | 33 |
EBSCO Education Abstracts (HW Wilson) | 4567 | 2056 | N/A | N/A |
ProQuest Education Database | 9115 | 2209 | N/A | N/A |
EBSCO Educational Administration Abstracts | 1429 | 511 | N/A | N/A |
EBSCO ERIC | 14,891 | 10,140 | 15,414 | 1301 |
OvidSP Embase | 11,214 | 4746 | 12,630 | 1536 |
OvidSP Global Health | 3512 | 1030 | 3988 | 455 |
World Health Organization ICTRP | 601 | 384 | N/A | N/A |
OvidSP MEDLINE | 8646 | 8007 | 9589 | 970 |
OvidSP PsycInfo | 17,477 | 13,548 | 19,126 | 1387 |
Elsevier Scopus | 17,484 | 4832 | 19,365 | 1096 |
OvidSP Social Policy & Practice | 722 | 441 | 743 | 64 |
Clarivate Web of Science and Social Sciences Citation Index | 11,189 | 3232 | 12,681 | 970 |
EBSCO Teacher Reference Center | 1987 | 410 | N/A | N/A |
EPPI-Centre TRoPHIct | 0 | 0 | 0 | 0 |
Total | 126,180 | 62,742 | 105,777 | 9709 |
Following de-duplication, 62,742 references from the original search and 9709 references from the updated search were screened for inclusion on title and abstract. Pilot screening of four batches of 50 (carried out by CB and RP) revealed consistent (> 90%) agreement between reviewers over which references should be excluded on the basis of title and abstract. Disagreements were discussed between reviewers and the meaning of inclusion and exclusion criteria clarified. As per the protocol, given consistent batch-level agreement of > 90%, the remaining references were screened by a single reviewer (either CB or RP), with each checking with the other if they were unsure about the exclusion of a report.
After screening on title and abstract, 346 references were retained for full-text screening from the initial database searches and 54 were retained from the updated searches, making a total of 400 references from database searches to be screened on full text.
Thirteen additional references were identified from the web searches as includable for full-text screening (Table 2).
Website name | Number of papers identified for full-text screening |
---|---|
Cambridge Journals Online (www.eifl.net/e-resources/cambridge-journals-online) | 0 |
Centers for Disease Control and Prevention: Smoking and Tobacco Use (www.cdc.gov/tobacco/index.htm) | 0 |
Child and Adolescent Research Unit (www.cahru.org/) | 0 |
Childhoods Today (www.childwatch.uio.no/publications/journals-bulletins/childhoodstoday.html) | 0 |
Children in Scotland (https://childreninscotland.org.uk/) | 0 |
Children in Wales (www.childreninwales.org.uk/) | 0 |
European Union Community Research and Development Information Service (https://cordis.europa.eu/) | 0 |
Database of Educational Research (EPPI-Centre) | Not available |
Drug and Alcohol Findings Effectiveness Bank (https://findings.org.uk/e-bank.php) | 0 |
0 | |
Google Scholar | 0 |
Welsh Government (https://gov.wales/) | 0 |
Scottish Government (www.gov.scot/) | 0 |
Joseph Rowntree Foundation (www.jrf.org.uk/) | 2 |
National Criminal Justice Reference Service (www.ncjrs.gov/) | 10 |
National Society for the Prevention of Cruelty to Children (www.nspcc.org.uk/) | 0 |
National Youth Agency (https://nya.org.uk/) | 0 |
Northern Ireland Executive (www.northernireland.gov.uk/) | 0 |
OpenGrey (www.opengrey.eu/) | 0 |
Personal Social Services Research Unit (www.pssru.ac.uk/) | 0 |
Project Cork (www.centerforebp.case.edu/resources/tools/project-cork-clinical-screening-tools) | 0 |
UCL-IOE Digital Education Resource Archive (https://dera.ioe.ac.uk/) | 0 |
UK Clinical Research Network Portfolio (www.nihr.ac.uk/researchers/collaborations-services-and-support-for-your-research/run-your-study/crn-portfolio.htm) | 0 |
University of Illinois at Urbana-Champaign (https://illinois.edu/) | 0 |
US Center for Substance Abuse Prevention (www.samhsa.gov) | 0 |
Social Issues Research Centre (www.sirc.org/) | 0 |
The Campbell Library (www.campbellcollaboration.org) | 1 |
The Children’s Society (www.childrenssociety.org.uk/) | 0 |
Open Library (https://openlibrary.org/) | 0 |
Schools and Students’ Health Education Unit Archive (https://sheu.org.uk/) | 0 |
World Health Organization ICTRP (www.who.int/clinical-trials-registry-platform) | 0 |
Young Minds: Child and Adolescent Mental Health (https://youngminds.org.uk/) | 0 |
Total | 13 |
A further six records already known to reviewers from earlier scoping searches were also added for full-text screening.
Full texts for six references from our initial searches and one reference from our updated searches were unobtainable online or through interlibrary loans, leaving 412 records available for full-text screening, 359 from the original searches and 53 from the updated searches. Double-screening of a set of 50 full-text papers (carried out by CB and RP) revealed > 90% agreement on which items to include from the review. Following this, the two reviewers (CB and RP) moved to independent screening of the remaining references, resolving any uncertainties with each other as the arose. Sixty-two reports remained after full-text screening. 52,61,63,64,66–69,71,73,80,85,87–89,98,103,108–110,113,115,117,123,124,161–197 An additional two reports were added from reference-checking,198,199 and one further article was added from the consultation with subject experts. 200
The 65 reports52,61,63,64,66–69,71,73,80,85,87–89,98,103,108–110,113,115,117,123,124,161–200 deemed eligible for inclusion in the review were then coded according to which review question they answered. Sixty-three reports were identified for inclusion in the review of theories of change (RQ1),52,61,63,64,66–69,71,73,80,85,87,88,98,103,108–110,113,115,117,123,124,161–195,197–200 16 for inclusion in the review of process evaluations (RQ2)63,66,69,71,88,98,110,113,167,189–195 and 48 for inclusion in the review of outcome evaluations (RQ3 and RQ5). 52,61,64,67,68,73,80,85,87,103,108,109,115,117,123,124,161–188,197–200 Three reports were included for inclusion in the review of economic evaluations (RQ4). 89,167,196
Figure 2 summarises the flow of references through the review and the number of studies included in each synthesis. The left side of the diagram describes the original search, and the right side depicts the updated search. Each identified included studies addressing each of our RQs, which is indicated at the bottom.
Included studies and reports
Overview
The 65 reports included in the review covered 22 distinct interventions examined in 27 separate empirical studies. Six interventions were each the subject of one study report. 161,170,174,189,193,197 The remaining 59 reports covered 16 interventions, examined in 21 separate studies. Only three interventions were evaluated in more than one study. One intervention (Learning Together) was the subject of two separate studies, one pilot and one full trial in the UK, covered by nine included reports. 71,88,89,165,166,168,194,196 A further intervention (Positive Action) was examined in 13 publications from five separate empirical studies (four in the USA and one in the UK) 52,61,63,64,85,108–110,113,123,124,173,179 One paper reported on a study carried out in Australia evaluating two relevant interventions (the Friendly Schools intervention and the combined Friendly Schools and Cool Kids Taking Control interventions),197 one of which (Friendly Schools) was also the subject of a separate study and report. 169
The reports and the interventions and studies they correspond to are reported in Table 3, organised according to the RQ they answer. These are summarised as follows: included reports that describe theory of change, which helped answer the second part of RQ1 (how closely do interventions align with the theory of human functioning and school organisation?); included reports that evaluate processes, which helped answer RQ2 (what factors relating to setting, population and intervention influence the implementation of interventions); included reports that evaluate outcomes, which helped answer RQ3 (overall and by intervention subtype, what are the effects of interventions on student substance use, violence and education?) and/or RQ5 (are these effects mediated by student commitment to school or moderated by setting or population); and included reports that evaluate economic outcomes, which helped answer RQ4 (what is the cost-effectiveness of interventions, overall and by subtype).
Intervention name | All included reports | Included reports that | |||
---|---|---|---|---|---|
Describe theory of change | Evaluate processes | Evaluate outcomes | Evaluate economic outcomes | ||
AAYP school/community intervention | Not applicable | Not applicable | |||
CDP | Not applicable | Not applicable | |||
Cooperative Learning | Not applicable | Not applicable | |||
CFS | Cross 2018;191 Australia/Perth |
Cross 2016;191 RCT: Australia/Perth | Not applicable | ||
DARE Plus programme | Bosma 2005;190 USA/Minnesota |
Not applicable | |||
DASI | Not applicable | Not applicable | |||
Friendly Schools | Cross 2011;169 Australia/Perth Rapee 2020;197 Australia/New South Wales and Western Australia |
Not applicable | Not applicable | ||
Friendly Schools and Cool Kids Taking Control combined | Rapee 2020;197 Australia/New South Wales and Western Australia | Rapee 2020;197 Australia/New South Wales and Western Australia | Not applicable | Rapee 2020;197 RCT: Australia/New South Wales and Western Australia | Not applicable |
FSFF | Cross 2012;171 Australia/Perth Cross 2018;192 Australia/Perth |
Cross 2018;192 Australia/Perth |
Cross 2012;171 RCT: Australia/Pertha | Not applicable | |
FSTP | Cross 2018;170 Australia/Perth | Cross 2018;170 Australia/Perth | Not applicable | Cross 2018;170 RCT: Australia/Perth | Not applicable |
Gatehouse Project | Bond 2001;66 Australia/Victoria Bond 2004;67 Australia/Victoria Bond 2004;68 Australia/Victoria Patton 2006;115 Australia/Victoria |
Bond 2001;66 Australia/Victoria |
Not applicable | ||
Going Places programme | Simons-Morton 2005;182 USA/Maryland Simons-Morton 2005;183 USA/Maryland |
Not applicable | Not applicable | ||
GST | Knight 2018;98 Uganda/Luwero District |
Greco 2018;89 Uganda/Luwero District |
|||
HSE | Bonell 2010;69 UK/SE England |
Bonell 2010;73 QE: UK/SE England | Not applicable | ||
Learning Together | |||||
PPP | Mitchell 1991;193 USA/Portland | Mitchell 1991;193 USA/Portland | Mitchell 1991;193 USA/Portland | Not applicable | Not applicable |
Positive Action |
|
|
Not applicable | ||
Project PATHE | Gottfredson 1986;174 USA/Charleston | Gottfredson 1986;174 USA/Charleston | Not applicable | Gottfredson 1986;174 QE: USA/ Charleston | |
Responsive Classroom | Anyon 2016;189 USA | Anyon 2016;189 USA | Anyon 2016;189 USA | Not applicable | Not applicable |
Restorative Practices Intervention | Acosta 2019;161 USA/Maine | Acosta 2019;161 USA/Maine | Not applicable | Acosta 2019;161 RCT: USA/Maine | Not applicable |
SEHER programme | Not applicable | Not applicable | |||
Whole-of-school intervention | Not applicable | Not applicable | |||
Total (n) | 65 | 63 | 16 | 48 | 3 |
All 65 reports included in the review described an intervention that helped to address the first part of RQ1 (what whole-school interventions that promote student commitment to school to prevent student substance use and violence have been evaluated and what subtypes are apparent?). Sixty-three reports covering all 22 interventions evaluated in the review were included in the synthesis of theories of change (RQ1). 52,61,63,64,66–69,71,73,80,85,87,88,98,103,108–110,113,115,117,123,124,161–195,197–200 Two reports were not included in the synthesis of theories of change as they did not add any new information to more detailed descriptions provided in other reports. 89,196 Sixteen reports covering 13 studies of 10 interventions evaluated processes and all of these also described a theory of change, addressing both RQs 1 and 2. 63,66,69,71,88,98,110,113,167,189–195 Forty-eight reports covering 23 studies and 20 interventions evaluated outcomes (including moderator and mediator analysis) and all of these were also included in the synthesis of theories of change, addressing RQs 1 and 3. 52,61,64,67,68,73,80,85,87,103,108,109,115,117,123,124,161–188,197–200 Of these, 19 reports were included in the synthesis of moderator analysis61,64,68,80,85,87,103,108,117,163,166,168,171–173,176,177,181,186 and three were included in the synthesis of mediator analysis,124,165,184 addressing RQ5. These are marked with an asterisk in Table 3. No publications reported only on theory, processes or outcomes. Two publications reported on economic outcomes only, addressing RQ489,196 and one publication reported on theory, processes, outcomes and economic outcomes, addressing all RQs. 167
Twelve interventions were examined in outcome evaluations only,87,161–164,169,170,174–178,180–188,197,198,200 two were examined in process evaluations only189,193 and six were examined in both outcome and process evaluations. 52,61,63,64,66–69,73,85,103,108–110,113,115,117,123,124,171,173,179,190–192,199 Two interventions were examined in outcome, process and economic evaluations. 71,80,88,89,98,165–168,172,194–196
Study and intervention characteristics
The following summaries provide details of the rate of report publication, the geographical location of each empirical study, their study designs, the outcome interventions targeted, who delivered the interventions, intervention components and the duration of interventions.
Rate of report publication
Figure 3 provides a breakdown of the rate of report publication according to which RQ(s) they answered. Included reports were published between 1986 and 2021, with only three reports (4%) published before 2000,164,174,193 18 (28%) published between 2000 and 200963,64,66–68,85,87,103,115,117,162,163,173,177,182,183,190,198 and 44 (68%) published from 2010 onwards. 52,61,69,71,73,80,88,89,98,108–110,113,123,124,161,165–172,175,176,178–181,184–189,191,192,194–197,199,200 Twenty-three of the reports included in the review (35%) were published in 2018 or later. 89,98,113,161,165–168,170,176,180,181,184–188,191,192,194–197
Geographical location of studies
Of the 27 studies included in the review, 13 (48%) were conducted in the USA,52,61,63,64,85,87,103,108–110,117,123,124,161–164,173,174,177,179,182,183,185–190,193,198 7 (26%) were conducted in Australia66–68,115,169–171,175,176,191,192,197,199 and 4 (14%) were conducted in the UK. 69,71,73,88,113,165–168,194–196 One study was conducted across Belgium, Cyprus, England, Greece and the Netherlands (4%);178,200 one in India (4%);180,181,184 and one in Uganda (4%)80,89,98,172 (Figure 4).
Study designs
Of the 23 studies of 20 interventions included in the review of outcome evaluations, 18 were RCTs52,61,64,67,68,80,87,89,103,108,109,115,117,123,124,161,165–172,175–188,196,197,199,200 and 5 used quasi-experimental designs. 69,85,162–164,173,174,198 Of the 16 reports covering 13 studies of 10 interventions included in the review of process evaluations, 9 drew on both quantitative and qualitative data,66,69,71,113,167,189,191,193,194 5 reported on quantitative data only63,98,110,190,192 and 2 drew on qualitative data only. 88,195
Intervention characteristics
Outcomes targeted by interventions
Of the 22 interventions included in the overall review, 8 targeted violence outcomes only,69,73,161,169–171,178,191,192,197,199,200 and 1 focused on substance use only. 175,176 Seven targeted both violence and substance use. 66–68,103,115,117,165–168,180–188,190,194–196,200 Two interventions targeted violence and academic outcomes,80,89,98,172,189 and two targeted substance use and attainment. 174,193 Two interventions targeted violence, substance use and academic outcomes. 52,61,63,64,85,108–110,113,123,124,162–164,173,179,198
Phase of schooling targeted by interventions
Of the 22 interventions covered by the reports included in the review, 5 (23%) targeted students broadly corresponding to English primary school age (between 5 and 11 years). 162–164,169,171,178,192,197,198,200 The Friendly Schools,169,197 Friendly Schools, Cool Kids Taking Control197 and Friendly Schools Friendly Families (FSFF)171,192 interventions were delivered in Australian primary schools, which typically accommodate students aged 4–12 years, differing slightly to the English primary school phase. Twelve interventions (54%) targeted students corresponding to the English secondary school phase (i.e. aged 11–18 years)66–69,71,73,88,103,115,117,161,165–168,175,176,180–188,190,191,193–196,199 and five (23%) targeted students across both phases (Figure 5). 52,61,63,64,80,85,87,89,98,108–110,113,123,124,170,172,173,177,179,189
Providers and delivery of interventions
In all the interventions included in the review, implementation was primarily led by schools themselves, with additional support from external agencies or the research team in the form of training only in the case of 6 interventions,169,171,185–188,191–193,197,199 external facilitation only in the case of 1 intervention178,200 and both training and external facilitation in the case of 15 interventions. 52,61,63,64,66–69,71,73,80,85,87–89,98,103,108–110,113,115,117,123,124,161–168,170,172–177,179–184,189,190,194–196,198
Consistent with an emphasis on whole-school change, 17 interventions evaluated in included studies involved the institution of school ‘project teams’, ‘action groups’ or ‘committees’ to engage members of the school community in making decisions about, and overseeing implementation of, intervention activities. These involved school staff only in five interventions;162–164,169,170,175,176,197,198 a mix of school staff and students in four interventions;71,80,88,89,98,165–168,172,191,193–196,199 school staff, students and parents in a further four interventions;52,61,63,64,69,73,85,108–110,113,123,124,173,178–181,184,200 and school staff, students, parents and members of the wider community in two interventions. 87,174,177 In one further intervention, the project team included only staff and a parent representative,171,192 and in another only staff and community representatives. 66–68,115
Of the 13 interventions involving a curriculum element (see Inclusion of curriculum components), 11 had lessons delivered by trained school staff,52,61,63,64,66–68,71,85,88,108–110,113,115,123,124,162–171,173,175,176,179,182,183,191,192,194–199 whereas in 1 intervention, the Aban Aya Youth Project (AAYP) school/community intervention, lessons were delivered by university health educators. 87,177 A further intervention, the Drug Abuse Resistance Education (DARE) Plus programme, involved curriculum delivery by police officers and peer-delivered curriculum elements. 103,117,190 DARE Plus also included neighbourhood action teams aimed at improving community environments, which was facilitated by trained neighbourhood volunteers. 103,117,190
Six interventions involved some element of peer-to-peer delivery. 80,89,98,103,117,172,174,180,181,184,190,191,193,199 This included students acting as peer mentors in Cyber Friendly Schools (CFS) and Portland Peers Project (PPP);191,193,199 students designing and delivering events and awareness-raising campaigns concerning violence and substance use for their peers in the DARE Plus programme, the Strengthening Evidence base on scHool-based intErventions for pRomoting adolescent health (SEHER) programme, Good School Toolkit (GST) and Project Positive Action Through Holistic Education (PATHE);80,89,98,103,117,172,174,180,181,184,190 and, in the case of DARE Plus, as previously mentioned, students delivering classroom curriculum to their peers. 103,117,190
Inclusion of curriculum components
In addition to including one or more of the whole-school components specified in our inclusion criteria (see Chapter 2, Inclusion criteria for the review), over half (n = 13) of the interventions included in the review also contained health curriculum components as part of multicomponent strategies to address violence and/or substance use. 52,61,63,64,66–68,71,85,87,88,103,108–110,113,115,117,123,124,162–171,173,175–177,179,182,183,190–192,194–199 As well as providing health information, 12 interventions also included curricula with a focus on social and emotional learning. 52,61,63,64,66–68,71,85,87,88,103,108–110,113,115,117,123,124,162–171,173,177,179,182,183,190–192,194–199 In five interventions, curricula also addressed bystander action in relation to bullying and aggression. 169–171,191,192,197,199
Use of student data
Eight interventions involved the use of recently collected student data to highlight local priorities and tailor the implementation of various intervention activities to student needs. 66–69,71,73,88,115,165–169,171,175,176,178,192,194–197,200
Duration of interventions
The shortest intervention duration was 8 months. 178,200 Two interventions (9%) were 1 year in duration, 69,73,189 and one intervention (5%) had a duration of 18 months. 80,89,98,172 Ten interventions (45%) were implemented over 2 years;103,117,169,170,175,176,180,181,184–191,193,197,199 six interventions (27%) were 3 years in duration,66–68,71,88,115,162–168,171,174,182,183,192,194–196,198 with one of these (Learning Together) truncated to 1 year in the pilot study included in the synthesis of process evaluations. 71,88 Only one intervention (5%) had a duration of 4 years. 87,177 A further intervention (Positive Action) had an open-ended duration, but was implemented for at least 2 years in each of the studies included in the review,52,61,63,64,85,108–110,123,124,173,179 apart from in the pilot implemented in south-east England, where the intervention ran for 9 months (Figure 6). 113 A descriptive overview of each of the 22 interventions is provided in Appendix 2, Table 9.
Chapter 4 Results: typology of intervention approaches
Development of typology
Drawing on authors’ descriptions of interventions and theories of change, we developed a list of key intervention elements and described interventions in terms of whether or not they included these (Table 4).
Intervention | Intervention elements | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Using more engaging learning methods in classrooms | Improving student–teacher relationships across the school | Student involvement in school-level policy, rules or other decisions | Students volunteer in community | Involving parents in school-level decisions or activities | ||||||||
Improving pedagogy | Co-operative learning | Student-centred/ culturally inclusive curriculum | Peer tutoring | Collaborating on groups/ committees | Restorative practice/ non-punitive discipline | Other approaches | Group policy review | Other | Groups | Other | ||
AAYP School/ community intervention | X | X | X | X | ||||||||
CDP | X | X | X | X | X | |||||||
Cooperative Learning | X | |||||||||||
CFS | X | X | X | |||||||||
DASI | X | X | X | |||||||||
DARE Plus | X | |||||||||||
Friendly Schools | X | |||||||||||
Friendly Schools and Cool Kids Taking Control | X | |||||||||||
FSFF | X | X | X | X | ||||||||
FSTP | X | |||||||||||
Gatehouse Project | X | |||||||||||
Going Places programme | X | |||||||||||
GST | X | X | X | X | ||||||||
HSE | X | X | X | |||||||||
Learning Together | X | X | X | |||||||||
PPP | X | X | X | X | ||||||||
Positive Action | X | X | X | X | X | X | ||||||
Project PATHE | X | X | X | X | X | X | X | |||||
Responsive Classroom | X | X | X | |||||||||
Restorative Practices Intervention | X | |||||||||||
SEHER | X | X | X | X | X | |||||||
Whole-of-school intervention | X | X | X | X |
Student participation in school policy decisions
We then compared the key content of different interventions to identify four discrete intervention subtypes, two of which were further subdivided based on the specific approaches they took (Table 5). These subtypes are described in detail in Table 5. Our analysis led us first to identify a distinct subtype of 14 interventions52,61,63,64,69,71,73,80,85,87–89,98,103,108–110,113,117,123,124,162–168,172–181,184,189–191,193–196,198–200 that contained elements to develop student participation in school policy, rule-making or other school-wide decisions. These were subdivided into those in which students sat on a school decision-making group with staff and sometimes parents and members of the wider community (10 interventions)52,61,63,64,69,71,73,80,85,87–89,98,108–110,113,123,124,165–168,172–174,177–181,184,191,193–196,199,200 and those that used other mechanisms to involve students in school decision-making (four interventions). 103,117,162–164,175,176,189,190,198
Intervention subtype | Further subdivision | Interventions in this category |
---|---|---|
Student participation in school policy decisions | Students sat on a decision-making group with staff | AAYP school/community intervention |
CFS | ||
DASI | ||
GST | ||
HSE | ||
Learning Together | ||
PPP | ||
Positive Action | ||
Project PATHE | ||
SEHER | ||
Other approaches | CDP | |
DARE Plus | ||
Responsive Classroom | ||
Whole-of-school intervention | ||
Student relationships with teachers, but not student participation in school policy decisions | Restorative practice or other non-punitive discipline methods | FSTP |
Restorative Practices Intervention | ||
Other approaches | FSFF | |
Gatehouse Project | ||
Student engagement in learning, but not student participation in decision-making or relationships with teachers | Cooperative Learning | |
Going Places programme | ||
Parent involvement only | Friendly Schools | |
Friendly Schools and Cool Kids | ||
Taking Control |
The interventions included in this subtype also tended to be the most complex in terms of the number and combination of the key intervention elements they contained. As well as elements focused on student participation in decision-making, five of the interventions in this subtype contained elements addressing student–staff relationships via the use of restorative practices or other non-punitive discipline methods,71,88,162–168,174,180,181,184,189,194–196,198 and eight contained a parent-involvement element, with five interventions involving parents in school decision-making groups69,73,87,174,177,178,180,181,184,200 and three engaging parents in school via other activities. 52,61,63,64,85,108–110,113,123,124,162–164,173,175,176,179,198 Ten of the 14 interventions included in this subtype also involved changes to teaching to increase academic engagement,52,61,63,80,85,87,89,98,108–110,113,123,124,162–164,172–177,179–181,184,189,191,193,198,199 and 2 involved encouraging young people to volunteer in the community. 52,61,63,64,85,108–110,113,123,124,173,179,193
Improving staff–student relationships, but not student participation in school policy decisions
We then identified a second set of interventions that did not contain elements involving student participation in decision-making, but did include elements focused on improving staff–student relationships, comprising four interventions. 66–68,115,161,170,171,192 These interventions were subdivided into interventions that aimed to improve student–staff relationships via introducing restorative practices or other non-punitive disciplinary methods (two interventions)161,170 and those that took other approaches to improving staff–student relationships (two interventions). 66–68,115,171,192
Increasing student engagement in learning, but not student participation in decision-making or relationships with teachers
The third distinct category of interventions did not include student participation in decision-making or improving staff–student relationships, but did include elements to increase student engagement in learning. This subtype comprised two interventions. 182,183,185–188 Improving student engagement in learning mainly involved modifications to teaching to promote co-operative and collaborative learning.
Increasing parental involvement in school only
We identified a fourth intervention subtype comprising two interventions169,197 that did not address student participation in decision-making, student–staff relationships, elements to increase student engagement in academic learning or other key elements in our list, but did contain elements aiming to increase parental involvement in school. Specifically, these included involvement of parents in the development and dissemination of school bullying policies.
Chapter 5 Synthesis of theories of change
Included reports
Sixty-three of the reports included in the review contributed to the synthesis of theories of change. 52,61,63,64,66–69,71,73,80,85,87,88,98,103,108–110,113,115,117,123,124,161–195,197–200 Together, these reports provide a description of the theoretical basis for each of the 22 interventions included in the review, with 16 interventions covered by more than one report. 52,61,63,64,66–69,71,73,80,85,87,88,98,103,108–110,113,115,117,123,124,162–169,171–173,175–188,190–192,194,195,197–200 Two reports were not included in the synthesis of theories of change as they were deemed not to contribute anything additional to descriptions provided in other reports. 89,196 A detailed summary of the studies and interventions included in the synthesis of theories of change is provided in Chapter 3. A summary of the theory of change for each intervention and the scientific theories informing them is provided in Appendix 3, Table 10.
Quality assessment
Appendix 4, Table 11, summarises the results of the quality appraisal for each of the reports included in the synthesis of theories of change. The overall agreement rate between independent reviewers (CB and RP) on the quality of descriptions of theory of change was high (85%). When there was disagreement, it was resolved through discussion.
The majority of descriptions of theories of change were judged to be of low to medium quality, with 34 (54%) reports meeting none or one of the criteria assessed52,61,63,64,67–69,73,80,88,98,108–110,115,117,123,124,163,164,172,173,175–180,183,184,189,190,193,197 and 13 (21%) reports meeting two or three of the criteria;66,103,113,162,169–171,174,181,182,191,198,199 16 (25%) descriptions of theories of change were judged to be of high quality, meeting four or five of the criteria. 71,85,87,161,165–168,178,185–188,192,194,195 Seven interventions had at least one high-quality description of theory,71,85,87,161,165–168,185–188,192,194,195,200 nine had at best a medium-quality description66,103,162,169,170,174,181–183,191,198,199 and six interventions had only a low-quality description. 69,73,80,98,172,175,176,189,193,197
Reports varied according to which items across the five criteria they met. Three reports described pathways from intervention components to outcomes only,115,164,189 whereas a further six reports described only the concepts and constructs that made up the theory of change. 63,67,68,173,180,184 Three reports clearly defined pathways to intervention outcomes, as well as the concepts and constructs that made up the theory of change,103,169,181 but not inter-relations between concepts, mechanisms or how these varied by place or person; seven reports clearly defined pathways to outcomes, concepts and constructs, and the inter-relations between these, but not mechanisms or how these varied by place or person. 113,170,171,174,182,191,199 One further report defined intervention theory concepts and constructs, the inter-relations between these and the mechanisms of action, but did not describe how mechanisms were shaped by place or person, or pathways from the intervention to outcomes. 198 Fourteen reports described pathways from intervention outcomes, concepts and constructs, the inter-relations between these and the biological, psychological, social processes or mechanisms expected to be triggered by the intervention, but not how mechanisms might vary by place or person. 71,85,161,165,166,168,185,187,188,192,194,195,200 One report described just the concepts and constructs that made up the theory of change, the inter-relations between these and mechanisms of action,162 and another described the concepts and constructs that made up the theory of change alongside the mechanisms expected to be triggered. 66 Only two reports described how triggering of mechanisms might vary by place or person, with one of these also clearly meeting all other criteria166 and the other describing pathways to intervention outcomes and the concepts and constructs that made up the theory of change and how these inter-related, but not mechanisms themselves. 87 Twenty-five reports52,61,64,67,69,73,80,88,98,108–110,117,123,124,163,172,175–179,183,190,193,197 did not meet any of the criteria assessed. As detailed in our methods, the descriptions of theory in all these reports nevertheless contributed to the synthesis of theories of change, but with more interpretive weight given to studies with theories of change deemed to be of high quality.
Synthesis of theories of change
We initially aimed to synthesise theories of change for each distinct subcategory of intervention to determine if and how the theoretical basis of these differed and how closely these aligned with the theory of human functioning and school organisation. 46 We started with the 14 interventions52,61,63,64,69,71,73,80,85,87,88,98,103,108–110,113,117,123,124,162–168,172–181,184,189–191,193,195,198–200 that involved components that sought to develop student participation in policy decisions via groups or other means, drawing on the thematic coding of intervention theories of change, as described in Chapter 2. There was sufficient reciprocal translation between all but one of the intervention theories in this group to enable a line of argument synthesis to build a single overarching theory of change that resonated with the underlying theories of all interventions in this subtype. The exception was an intervention (Positive Action)52,61,63,64,85,108–110,113,123,124,173,179 that, despite having similar intervention activities, was underpinned by a distinctive theory of change that had more in common with the Going Places intervention182,183 in the ‘increasing student engagement in learning’ category of interventions discussed below. Therefore, we decided to synthesise the theories of change for these two interventions separately.
We then undertook synthesis of theories underlying the four interventions that focused on modifying disciplinary practices or other aspects of the school environment to enhance student relationships with teachers,66–68,115,161,170,171,192 and then the two interventions that included parent involvement only,169,197 out of our specified list of intervention components, intending to identify the distinctive theories of change underlying each intervention type. However, the theories of change underlying these intervention types were not distinct from those underlying interventions aiming to develop student participation in policy decisions, and so below we present a single synthesis for theories of change underlying all three [see Interventions developing student participation (except Positive Action), enhancing staff–student relationships or involving parents].
We then went on to synthesise theories of change for the two interventions aiming to increase student engagement in learning,182,183,185–188 finding that the underlying theory of change for one of these, the Cooperative Learning intervention,185–188 resonated closely with those in the other three intervention categories. The themes from the theory of change for this intervention were therefore synthesised with those aiming to develop student participation in policy decisions, those aiming to enhance student relationships with teachers and those including parent involvement only. As indicated previously, the Going Places intervention182,183 in the ‘increasing student engagement in learning’ category had a distinct theory of change that resonated with that of Positive Action,52,61,63,64,85,108–110,113,123,124,173,179 and so these two theories of change were synthesised together. A second separate synthesis is presented for these two interventions. Each synthesis is organised around a final set of themes (see project web document 1 at www.journalslibrary.nihr.ac.uk/phr/AHSN4485) that were refined through our analysis from our initial set of a priori themes suggested by the theory of human functioning and school organisation46 (see project web document 2 at www.journalslibrary.nihr.ac.uk/phr/AHSN4485).
Interventions developing student participation (except Positive Action), enhancing staff–student relationships or involving parents
Alignment with the theory of human functioning and school organisation
Despite only one intervention being explicitly informed by the theory of human functioning and school organisation,46,71,88,165–168,194,195 a number of themes that were identified in our analysis as reciprocally translating across theories of change in this group aligned closely with the a priori themes suggested by the theory of human functioning and school organisation. The themes apparent across theories of change also enabled some refinement and disaggregation of the a priori themes developed on the basis of the theory of human functioning and school organisation. There were also examples of refutational synthesis apparent whereby some theories of change involved concepts that diverged from those in other theories of change in this group and/or from those suggested by the theory of human functioning and school organisation. 46 These are described to add differentiation and nuance to the analysis. We present a logic model summarising this refined theory of change in Figure 7.
Reframing school organisational management
Elaborating our a priori concepts, the reframing of school organisational management to involve students, parents and other members of the community in school-level decision-making emerged as an overarching theme recurring across descriptions of intervention theories of change. A number of interventions involved surveying students66–69,71,73,88,115,165–169,171,175,176,178,192,194,195,197,200 or consultation with students to identify priorities for action on school improvement,180,181,184 thus making school-level decision-making more inclusive of student views and ideas. This was described by Shinde et al. 181 in their medium-quality account of the theory of change for SEHER:
[T]he speak-out box was a letterbox providing a platform for students to raise concerns, complaints, and suggestions anonymously … issues were addressed through … discussion during various activities of the intervention.
Other strategies for making school decision-making more student-centred involved students participating directly in groups with school staff to make decisions about changes to school learning or discipline systems. 69,71,73,80,87,88,98,165–168,172,174,177,178,180,181,184,191,193–195,199,200 Although such approaches were theorised to make school provision and policies more student-centred and to support the implementation of the interventions concerned, they were further considered to strengthen relationships among and between staff and students, which reciprocally translated with our a priori theme of eroding boundaries among and between these groups. In their high-quality description of the theory of change for Learning Together, Bonell et al. 165 explained:
[The intervention aims to] transform the whole-school climate and improve staff–student relationships, student commitment to learning and sense of belonging and participation in the school community.
Shinde et al. 181 similarly described:
The intervention’s conceptual framework … emphasises the importance of a positive school climate – i.e., supportive relationships between school community members, a sense of belonging to the school, a participative school environment, and student commitment to academic values.
Similarly, interventions that recruited parents or community members onto the groups leading implementation of interventions69,73,87,171,174,177,178,180,181,184,192,200 or that involved them in decision-making via other means169,197 represented efforts to involve the wider school community in school organisational management, enabling local tailoring and support for intervention activities, and the weakening of classification by eroding boundaries between the school and parents or other community members. This was exemplified in Flay et al.’s87 high-quality description of the theory of change for the AAYP school/community intervention:
The community program forged linkages among parents, schools, and local businesses. Each … school formed a local school task force consisting of school personnel, students, parents, community advocates, and project staff to implement the program components.
Reframing the instructional order
Interventions in this category commonly included elements aiming to modify schools’ teaching in some way to increase student engagement in learning. Theories of change differed as to whether this was limited to ensuring the inclusivity of teaching methods and intervention content, 87,177 or improving pedagogic practice or study skills,80,98,172,175,176,180,181,184,189 or whether teaching and learning were more radically reframed to weaken framing by providing students with a more active, collaborative role in constructing their own learning. 162–164,174,198
Below we consider these different forms of reframing of the instructional order and how they were theorised to erode boundaries within and beyond the school, the concept of boundary erosion being a key concept identified as an a priori theme in our coding framework, which reciprocally translated with concepts across many theories of change in this synthesis.
First, theories of change that referred to the importance of integrating culturally specific learning into school curricula described this as a means of recentring provision on the needs of students by including materials and methods that reflected the cultures and experiences of students or enabled them to appreciate those of others. 87,175–177 As explained by Flay et al. 87 in their account of the AAYP school/community intervention:87
Studies suggest that programs for African American youth should incorporate components that … enhance … cultural pride …. Hence, the interventions included the Nguzo Saba principles … which promote African American cultural values such as unity, self-determination, and responsibility.
Aligning with our a priori themes of eroding boundaries between school and the outside world, such approaches were explicitly theorised to erode boundaries between school culture and the diverse cultures of student families and communities, and to erode boundaries between students from different cultures. 87,162–164,177,198
Interventions including changes to pedagogy tended to focus on creating calmer, more respectful and equitable learning environments with better-quality teaching to improve engagement in learning, as well as greater accountability among teachers, as described by Devries et al. :80
The steps contain more than 60 different activities … focused around topics such as … creating a better learning environment, respect and understanding power relationships, improving teaching techniques, creating accountability.
In doing so, such approaches sought also to erode boundaries and improve relationships among students, and between students and teachers. As Kyriakides et al. 200 explained:
Effective teachers use different teaching strategies to keep different students involved in the classroom interactions which promote student learning and establish better relations among students and teachers.
Interventions improving pedagogic practices across a school also frequently organised teachers into teams to share good practice and social support. As well as being a mechanism to ensure effective whole-school change, this also reciprocally translated onto our a priori theme of eroding boundaries between previously individualistic teachers. Kyriakides et al. 178 described it thus:
[T]eachers interact on issues associated with learning and teaching… to create a business-like school and classroom environment … collaboration among teachers can only be beneficial, and could boost [school learning environment] quality … school management team… encourage… teaching staff to learn from each other by exchanging ideas and experiences on facing and reducing bullying.
Teaching study skills aimed to develop meta-learning skills over and above subject topic learning. 174,180,181,184 The cross-subject skills taught in the SEHER intervention, for example, included the following:
… time management, learning style, note taking, reading comprehension, memorisation techniques, and concentration techniques.
Shinde et al. 181
This aligned with our a priori theme of eroding boundaries between different academic subjects.
In one intervention,174 providing careers information and teaching job-seeking skills were theorised to introduce students to the world of work and to render instruction more meaningful to students, concepts that reciprocally translated onto our a priori theme of eroding boundaries between schools and the external world. As was explained by Gottfredson174 in her medium-quality description of the theory of change for Project PATHE:
The Career Exploration Programs, cosponsored by the program and a local technical college, provided high school students opportunities to participate in activities designed to introduce them to technical careers such as engineering, computer science, and industrial technology.
Turning now to the more radical reframings of the instructional order, authors referred to ‘constructivist’ approaches to learning as a means to reorient learning from more didactic approaches so that new knowledge was evaluated in the light of students’ own previous knowledge, skills and ideas. 162–164,174,185–188,198 Battistich et al. ,162 in their medium-quality description of the theory of change for the Child Development Project (CDP), explained it as follows:
Learning is inherently an active process in which students interpret new information in light of previous understandings and experiences, work through discrepancies, and construct new understandings … CDP’s instructional practices are consistent with this ‘constructivist’ approach to learning.
In such interventions, various forms of co-operative learning were used to reorient learning methods so that these moved away from individualistic tasks, assessments and rewards towards learning undertaken collaboratively by groups of students. 162–164,174,185,187,188,198 In the Cooperative Learning intervention,185–188 this involved a commitment to the principles of positive interdependence, whereby individual goals are aligned with the goals of the group such that individual success could be achieved only via group success, and competition and isolation among students are reduced. Via this approach, students were expected to:
… interact in ways that promote the goal attainment of others in the group, such as providing instrumental and emotional support, and sharing information and resources.
Van Ryzin and Roseth187
In the description of Cooperative Learning, the positive feelings that arise from such activities were theorised to result in a ‘benign spiral’188 that further increases such positive social interactions.
Such activities thus aimed to erode boundaries between students, as well as boundaries between academic education and students’ broader social and emotional development. As Battistich et al. 162 described:
[S]tudents reflect on and discuss their group interaction at the conclusion of the task. Thus, in addition to being an effective approach to learning, the co-operative activities also help students to build interpersonal bonds and develop social and ethical understanding and skills.
Van Ryzin and Roseth188 similarly noted:
We propose that this focus on social contact and the development of social skills enables co-operative learning to promote positive peer relations, which supports the development of empathy and, in turn, reduces bullying.
Constructivist co-operative approaches to learning also encouraged students from diverse backgrounds to draw on their own learning styles and experiences, as well as those of others, as a way to make sense and deepen their understanding of new information. As Battistich et al. 162 described:
The approach offers students of diverse cultural and ethnic backgrounds … to engage the curriculum in ways most appropriate to their own learning needs and interests; … meaningful interaction among students from different backgrounds and with diverse perspectives broadens the experiential base from which new and deeper understandings emerge.
By bringing diverse groups of students together, such approaches were theorised to foster insight and understanding of others’ perspectives to facilitate learning and ‘the breakdown of biases and prejudices among students who belong to different social groups’,186 thus encouraging the development of positive relationships and promoting social integration across peers.
As a result, these approaches were explicitly theorised to erode boundaries between school and the diverse cultures of student families and communities by making students’ backgrounds and experiences central to learning, and to erode boundaries between students from different backgrounds and cultures by enabling consideration and understanding of the perspectives of others.
Use of such approaches to learning also aimed to move from encouraging participation in learning via extrinsic rewards towards use of intrinsic rewards, whereby students become motivated by a more profound commitment to learning. 162–164,198 As Battistich et al. 162 described of the CDP:
A major emphasis of the CDP program is to tap into students’ intrinsic motivation to learn and… uphold the values of the school community. Teachers help students… understand that learning is both an inherently satisfying goal and a means to acquire the skills they need for successful, productive lives
Other theories of change emphasised the importance of ensuring that all students experienced academic success as an alternative way to engender students’ intrinsic commitment to learning and to school rules. 162–164,174,178,198 As described by Kyriakides et al. 178 in relation to the Dynamic Approach to School Improvement (DASI) intervention:
[P]erceived academic competence helps to prevent the development of norm-breaking behavior.
In the Gatehouse project, this was described as building a sense of ‘positive regard through valued participation in aspects of school’. 66
As further examples of more radical reframings of the instructional order, some interventions also included peer tutoring or mentoring, which was theorised to erode both boundaries between students (e.g. in different year groups) and boundaries in the role and status of teachers and students. 175,176,191,193,199 Mitchell193 in their lower-quality description of the theory of change for the PPP described it thus:
The Portland Peers Project utilized peer tutoring … [which] promoted student empowerment through involvement in meaningful activities, acquisition of information, and recognition of student responsibility … the project addressed … academic failure, lack of bonding to school, alienation, impaired confidence and self-esteem.
Some interventions also involved engaging parents in homework, to develop parents’ role in educating their children87,103,117,169,177,190,191,197,199 (eroding boundaries between the role of teachers and parents) or to draw on diverse family cultures in reframing the instructional order to erode school/home boundaries. 162–164,198 As described by Battistich et al. :162
The activities are designed to promote extended conversations and communication between students and their parents, connect the home to students’ experiences in school, and help students gain knowledge and understanding of their family’s beliefs, experiences, culture and heritage.
The approaches to reframing the instructional order outlined here were theorised to engender school commitment, as well as student knowledge and social support, reciprocally translating with concepts in the theory of human functioning and school organisation. This is considered in more detail in Engendering positive development and student commitment to reduce risk-taking behaviours.
Reframing the regulatory order
As with the instructional order, there was reciprocal translation of themes from included studies with our a priori themes whereby many theories engaged in some way with the concept of a school regulatory order and reframing this to engender greater student commitment. 69,71,73,88,162–168,194,195,198 As was described by Battistich et al. :162
Although often not explicitly recognized, schooling conveys important moral messages about how we should live our lives and how we should live together as people … The CDP program makes this ‘hidden curriculum’ … overt and supportive.
Theories of change again differed, as to whether reframing of the regulatory order was limited to staff modifying how they managed discipline or involved a more radical reframing. The former involved reorienting school discipline systems so that these better addressed students’ need for a safe and orderly environment, for example via enhanced classroom management, increased staff visibility or monitoring at break time or more consistent enforcement of rules and use of non-violent punishments. 80,98,171,172,180,181,184,192 More radically reframing gave students a more active role, via strategies such as rewriting school rules, learning-based discipline, restorative practice or learning social and emotional skills. 66–68,71,87,88,103,115,117,161–168,170,174,177,190,191,194,195,197–199 As with the split between interventions reframing the instructional order, some interventions included elements of both approaches. 171,178,189,192,200 The ways in which these approaches were theorised to work and how they reciprocally translated with our a priori theme of eroding boundaries within and beyond the school are outlined in School environment interventions developing student engagement in learning/social skills.
Enhanced classroom management, staff visibility at break time, more consistent enforcement of rules and use of non-violent punishments featured in several theories of change. 80,98,171,172,178,180,181,184,189,192,200 These aimed to promote a safer and more orderly school environment, but in some interventions were theorised to also have more direct impacts on violence or substance use by identifying and curtailing such activities, this diverging somewhat from our a priori themes. As Kyriakides et al. 200 described:
Increased monitoring of student behaviour during recesses and before the beginning of lessons can help school stakeholders to identify and intervene when bullying occurs. Teachers should be visible and vigilant in such common areas as hallways, stairwells, the canteen, the gym, and the crib.
In contrast, other approaches aimed to reframe the regulatory order by giving a more participative role to students. These were theorised to achieve impacts via mechanisms that reciprocally translated more closely with our a priori concepts of eroding boundaries within school. Several theories of change suggested that students’ contribution to writing or rewriting school rules or discipline policies with staff69,71,73,88,162–168,174,178,194,195,198,200 worked to erode boundaries between staff and students and among students, and would also increase overall student commitment to the resultant reframed regulatory order they took part in creating. As Kyriakides et al. 178 described:
The active involvement of teachers, students, and parents in defining the school policy on bullying and the strategies and action plans to face bullying may encourage their active participation in implementing these action plans for improvement purposes.
In effect, students became not merely subjects of the regulatory order, but its co-creators. In doing so, interventions aimed not merely to connect students to the regulatory order, but to reshape the regulatory order so that it was underpinned by democratic or ethical values, and aligned with students’ own values and cultures. 162–164,198 This aligned with our a priori concepts, in that this was another way in which interventions aimed to erode boundaries between schools and local communities.
Other theories of change suggested an even more fundamental reframing of the regulatory order that, in effect, eroded the boundary between schools’ regulatory and instructional orders. This constituted an elaboration of our a priori themes. These theories of change proposed a move from punitive discipline towards more learning-based discipline162–164,191,198,199 and the teaching of social and emotional skills66–68,71,87,88,103,115,117,162–171,177,190–192,194,195,197–199 and/or restorative practice71,88,161,165–168,171,192,194,195 whereby discipline became a focus for teaching, and conflict became an opportunity for learning. As described by Battistich et al. 164 in their lower-quality account of the theory of change for the CDP:
developmental discipline emphasizes a proactive ‘teaching’ approach to discipline rather than a coercive approach.
In Cross et al. ’s199 medium-quality description of the CFS theory of change:
[I]n contrast to promoting only punitive-based solutions to cyberbullying such as school suspension and withdrawing access to technology, which may exacerbate the problem, our formative research suggested … raising awareness among students, parents, and educators of the harms associated with the misuse of technology.
Similarly, social and emotional learning curricula aimed to teach students the skills to build strong relationships with peers and how to make healthy decisions, for example about violence and substance use. Cross et al. 171 described it thus in their medium-quality description of the theory of change for FSFF:
[C]lassroom level activities … were designed to complement students’ other social and emotional learning [and] focused on the reciprocal relationship between students who observe bullying, those who are bullied or bully others and their social environment.
Some curricula focused on increasing students’ involvement as active participants in the regulatory order by encouraging them to intervene as bystanders in incidents of bullying. 169,170
Restorative practice interventions also focused on building strong relationships between staff and students, and among students, to prevent conflict and misbehaviour, but also to repair such relationships in the aftermath of conflict. 71,88,161,165–168,171,192,194,195 Prevention occurred through activities such as ‘circle time’, in which students shared their feelings and developed ground rules for behaviour (resonating with other interventions in which students contributed to the formation of school rules and policies). As explained in Acosta et al. ’s161 high-quality description of the Restorative Practices Intervention’s theory of change:
restorative practices (e.g., circles, conferences) help… proactively build relationships with their peer and teachers and to have a more active voice in responding to specific school incidents. When students have positive interactions with their peers, this improves their peer relationships and can lead to more active participation in school.
Conflict was addressed through restorative conferences in which parties to a conflict met so that victims could describe the harm caused, perpetrators could take responsibility for this and work could be done to heal the relationship and identify appropriate reparations. These approaches aimed to erode boundaries between school members, as well as between schools’ regulatory and instructional orders. This aimed to prevent violence and other risk behaviours by ensuring that perpetrators did not become disconnected from the school community as a result of their experience of conflict and punitive discipline. Instead, perpetrators were reintegrated through a process of understanding the harm caused, recognising their responsibilities and making reparations. As was reported by Acosta et al. :161
[R]esponsive practices … ensure that offenders can take public responsibility for their behavior and reintegrate into normal community life.
Violence was prevented both by healing particular relationships and so curtailing further conflict, and by increasing student commitment to school and its norms of behaviour.
Other theories of change also suggested an erosion of the boundaries between the instructional and regulatory orders by aiming to broaden classroom teachers’ relationships with students from being merely educational towards including affective elements to root learning in a caring community. 69,71,73,88,162–168,194,195,198 As described by Battistich et al. :164
Teachers and administrators are encouraged to build warm, nurturing relationships with their students, and encourage students to develop warm, supportive, inclusive relationships with each other.
Some interventions further involved parents in discipline, encouraging parents to more closely monitor students’ behaviour and offering advice on how to do this. 103,117,169,171,190–192,197,199 The intended aims of such work reciprocally translate with our a priori themes of eroding boundaries between teacher and parental roles and eroding boundaries between the school and the family. As illustrated in Cross et al. ’s192 high-quality description of the FSFF programme’s theory of change:
The intervention was designed to… target parenting… identified as being protective of bullying behaviour … The family-level activities therefore worked in partnership with parents, building their awareness, attitudes and self-efficacy to role model, talk with and help their children to develop social competence and to prevent or respond to bullying.
Reframing relationships with the community
The involvement of parents and community members in decision-making groups was described previously as a means of reframing school organisational management and eroding boundaries between schools and the outside world. Several projects went further, aiming to build relationships between schools and local communities via various forms of voluntary work. Some projects encouraged students to volunteer in the local community,103,117,190,193 whereas others involved community members volunteering in schools and schools building alliances with community organisations. 87,103,117,177,190 All such components were theorised to develop school–community relations, a concept that reciprocally translates with our a priori concept of school–community boundary erosion. In a form of refutational synthesis, in one intervention the neighbourhood component was understood also to address community risk factors by building community cohesion and raising awareness of drug and alcohol issues,103,117,190 which diverged from our a priori themes suggested by the theory of human functioning and school organisation. 46
Engendering positive development and student commitment to reduce risk-taking behaviours
A key recurring theme across theories of change was that the abovementioned processes were theorised not merely to minimise risk behaviours, but to contribute towards students’ positive overall development. As emphasised by Battistich et al. :162
CDP’s emphasis is on the promotion of positive development … [to] promote children’s positive social, ethical, emotional, and intellectual development.
A subtheme was that interventions generally sought to achieve this via engendering student commitment to school (in terms of learning and belonging) by ensuring that school provision was reframed to meet students’ needs and align with their culture and values, and was built on positive relationships between members of the school community, as illustrated by this quotation from Bonell et al. :165
We theorised that … transforming the school environment to build student commitment to learning and sense of belonging in school … can be achieved by improving relationships between and among staff and students … and by better integrating students’ academic education and broader personal development.
Different theories used different terminology to refer to this, such as school attachment,67,68,115 bonding,163 school adjustment,169 inclusion115 and engagement and connection,67,68,161,170,184 but with clear reciprocal translation between these terms.
Some theories of change emphasised the importance of reframed provision being highly visible to students. This could consist of visible changes to school environment170,172 or highlighting changes to systems as a symbol of reframing provision based on student needs and informed by student participation. 166,194 Devries et al. 172 in their lower-quality account of the GST’s theory of change described it thus:
Some activities involve creating a better school environment by painting murals on school walls, and hanging codes of conduct in visible places.
In contrast, Bonell et al. 165 emphasised that:
Action groups are school meetings involving diverse students, and senior and junior staff … signal to the wider student body that the school cares about and intends to act on the views of staff and students to build a supportive school climate.
An example of refutational synthesis across theories of change was that there was a distinction between theories of change that viewed student commitment as reflecting a real investment in school activities and real commitment to shared values162–164,198 and theories of change that placed more emphasis on student commitment as outward compliance with rules. 178,200 For example, Solomon et al. 198 explained in their medium-quality description of the CDP’s theory of change how the intervention aimed to transform schools so that students shared their values:
We expect that students who have these needs met by experiencing a caring school community will feel strongly attached to the community, and that this attachment will lead them to feel personally committed to the values and goals the community promotes…
In contrast, the DASI intervention178 emphasised student compliance:
[S]chools could set up a motivation system to improve the school’s social environment by taking action to emphasize maintenance of the behavior code and the promotion of appropriate and positive behaviors outside the classroom.
In the middle were theories of change that viewed student commitment as reflecting students having a stake in the school institution so that they would be unwilling to engage in risk behaviours that might imperil their connection to school. 71,88,165–168,174,194,195 As Gottfredson174 explained:
[S]tudent involvement in constructive learning activities and consistent rewards for successful participation prevent delinquency by fostering attachment to school and giving students something to lose if they misbehave … youths who are ‘bonded’ to the social order have too much to lose by misbehavior.
Theories of change also described in what other ways intervention activities would benefit students’ positive development. These themes reciprocally translated with our a priori themes of promoting practical reasoning and affiliation, but elaborated these concepts. Several theories of change listed the different aspects of practical reasoning, such as student skills in reasoning, problem-solving, conceptual thinking, ethical and moral reasoning, and emotional learning, that would be engendered via intervention activities. 162–164,169,185–188,197,198 Theories of change also suggested impacts beyond cognitive skills to encompass self-efficacy, self-esteem, pro-social values, self-control, team work and perseverance. 87,175–177,185–188 Affiliation-related impacts included empathy and greater orientation to others, pro-social norms, as well as social, conflict resolution and communication skills. 71,87,165–169,175–177,185–188,195
Theories of change thus suggested that interventions would ultimately benefit students in terms of developing their autonomy, competence to make decisions and sense of belonging to place or people. 69,71,88,165–168,194,195 As summarised in Bonell et al. ’s166 high-quality description of Learning Together’s theory of change:
[T]he intervention aimed to enable young people to choose healthier behaviours by promoting their autonomy, motivation, and reasoning ability. These were to be promoted by increasing engagement with school.
Another example is given by Solomon et al. 198 in relation to the CDP:
[A]utonomy/influence refers to the individual’s opportunities to contribute to the group …; competence refers to the effectiveness and acceptance of the individual’s contributions to the group as well as his/her own academic and social efforts; and belonging is feeling that one is personally accepted by others ….
Across theories of change, attributes aligning with practical reasoning and affiliation were theorised to enable students to choose healthier, over riskier, behaviours.
Some theories of change also suggested that increases in student commitment to school might help reduce risk behaviours by mechanisms other than improving students’ affiliation and reasoning ability to choose healthier behaviours. 71,88,162–168,194,195,198 Battistich et al. 162 described it thus:
[T]he experience of being a valued member of a caring school community also should reduce the likelihood that students will seek to satisfy their need for social connection through association with gangs or other counter-cultural peer groups.
This theme reciprocally translated onto concepts previously developed by Jamal et al. 201 in a refinement of the theory of human functioning and school organisation,46 theorising that increasing student commitment to school will increase commitment to pro-school peer groups and decrease commitment to antischool groups and antisocial risk behaviours; our coding framework was elaborated to include this.
Van Ryzin and Roseth187 similarly described how intervention activities were expected to prevent deviant peer clustering to reduce health risk behaviours, but through a more direct mechanism than those already described:
[W]e attempted to interrupt the process of deviant peer clustering and reduce alcohol use by exposing youth to a broad cross section of their peers through collaborative, group-based learning activities in school … In this way, we hoped to interrupt the process of deviant peer clustering.
Such activities were also theorised to enable socially isolated students at risk of being bullied to develop new friendships and break down the process of homophily among bullies to reduce bullying. 186
School environment interventions developing student engagement in learning/social skills
We coded themes for theories underlying interventions to increase student engagement in learning, finding that one of the interventions (Going Places) had a distinctive theory of change,182,183 whereas the other (Cooperative Learning)185–188 resonated more with those described previously and was therefore included in that synthesis. We synthesised themes from the Going Places intervention with those of Positive Action52,61,63,64,85,108–110,113,123,124,173,179 because there was strong reciprocal translation of concepts across descriptions of these interventions.
Alignment with the theory of human functioning and school organisation
The theories of change of this group of two interventions aligned much less obviously with the theory of human functioning and school organisation. There was, however, some degree of reciprocal translation between the theories, as outlined below. A logic model summarising the theory of change for these interventions is provided in Figure 8.
Primary intervention focus on social skills to increase school engagement and reduce risk behaviours
Each intervention was theorised to simultaneously increase students’ school engagement and reduce risk behaviours through activities primarily focused on a discrete social skills curriculum. Interventions did include whole-school elements, as per our inclusion criteria, but these were secondary to the curriculum. This contrasted with the mechanism described in the theory of human functioning and school organisation,46 in which reductions in risk behaviours are to be achieved by increasing commitment to school via changes to the school regulatory and instructional orders. 153
In the case of the Going Places programme,182,183 the main intervention component was a discrete classroom curriculum aiming to provide students with problem-solving, self-management, self-control, school involvement, communication and conflict resolution skills. As Simons-Morton et al. 182 described it:
[T]he primary component is a curriculum designed to influence perceptions, attitudes, and expectations about substance use and antisocial behavior and to improve self-efficacy and social skills and competence.
A secondary component encouraged all teachers to incorporate the use and rewarding of use of such skills in other lessons and across the school to reinforce curriculum learning; parents were also provided with materials to improve discipline and reinforce concepts from the curriculum at home.
The Positive Action intervention52,61,63,64,85,108–110,113,123,124,173,179 aimed to teach positive behaviours such as engaging in learning, having positive relationships with peers and avoiding antisocial behaviours), as well as the social skills needed to engage in these positive behaviours and how engaging in these activities would develop students’ sense of positive self-concept. In turn, this positive self-concept was theorised to be reinforced by students’ engagement in positive behaviours, and avoidance of risk behaviours including conflict and substance use. As reported in Flay et al. ’s85 high-quality description of the intervention’s theory of change:
The program teaches children what actions are positive, that they feel good when they do positive actions, and that they then have more positive thoughts and future actions.
The interventions aimed to develop skills in creative thinking, study, healthy habits, self-management, interpersonal and emotional management, self-honesty, goal-setting and decision-making to prevent engagement in violence and substance use. 85,123
Although the Positive Action intervention was informed by the theory of triadic influence, which includes environmental as well as individualistic mechanisms of reducing risk behaviour,202 and it did include whole-school actions, these were primarily focused on generalising (and rewarding) the application of skills developed in Positive Action lessons to other aspects of school, similar to the way in which the Going Places182,183 intervention aimed to generalise skills learnt through the curriculum.
There was some element of reciprocal translation of Positive Action’s theory of change with the theory of human functioning and school organisation46 in that it did aim to improve relationships among students and between staff and students, and promote students’ involvement in the local community through volunteering. This resonated with our a priori themes of eroding boundaries among students, between students and staff and between school and the outside world. However, these were theorised to occur primarily directly through the teaching of positive behaviours and the development of social skills achieved via the Positive Action curriculum, rather than through changes to the school regulatory or instructional orders, as theorised in the theory of human functioning and school organisation. 46
Although both interventions did aim to modify teaching to increase engagement in learning, the primary mechanism of reducing risk behaviours was a direct one, whereby teaching students new skills would reduce their involvement in bullying or substance use.
Consultation with stakeholders
Policy and practice stakeholders broadly supported the theory of change synthesised as plausible. Building a sense of school community and student belonging were seen as central to this, and terminology that schools would support.
Several participants said some aspects of the theory of change would jar with some school staff. Boundary erosion would jar for some staff, especially in relation to student/staff roles and distinct academic subjects. It was suggested that the theory could be presented not in terms of eroding staff/student boundaries, but in terms of improving relationships, and not in terms of eroding subject boundaries, but in terms of developing overall skills and other assets. In terms of reframing, it was suggested that, as well as making the curriculum relevant to students, schools need to ensure that students understand the importance of academic learning. It was suggested that the theory could be revised in collaboration with teachers to ensure appropriate terminology.
The ALPHA group of young researchers similarly saw the synthesised theory of change as being plausible, although many had not previously considered the role of school commitment in improving health outcomes. They reflected that this was quite a different way of thinking about how school-based health interventions might work, but that it made sense to them. Participants reasoned that improving relationships between students and between students and staff was likely to make for a warmer and more welcoming school environment where students would feel more connected and engaged in learning. However, this was not their usual experience of secondary school. Participants also thought that it was plausible that creating such an environment would have other important health impacts in terms of reducing stress and improving mental health. ALPHA participants were interested in the idea of reframing school provision on student needs and felt that this could be a very powerful way of increasing school commitment, particularly for more disadvantaged students and in schools where there is a lot of ethnic diversity. However, some were more sceptical about the role of school commitment in reducing substance use and violence and determined that other external factors, including peers and home environment, would have a much greater impact on risk behaviours.
Chapter 6 Synthesis of process evaluations
Included reports
Sixteen reports, covering 13 separate studies and 10 interventions, were included in the synthesis of process evaluations. 63,66,69,71,88,98,110,113,167,189–195 Two interventions (Learning Together and Positive Action) were the subject of more than one study and report, with Learning Together evaluated in two studies across five reports71,88,167,194,195 and Positive Action evaluated in three studies, each with one corresponding report. 63,110,113 Of the 10 interventions evaluated, 1 was delivered to children corresponding to English primary school age (5–11 years),192 6 to children corresponding to English secondary school age (11–18 years) 66,69,71,88,167,190,191,193–195 and 3 to children across both age ranges. 63,98,110,113,189 Of the 13 studies, 5 were conducted in the USA,63,110,189,190,193 4 in the UK69,71,88,113,167,194,195 3 in Australia66,191,192 and 1 in Uganda. 98 Nine included reports drew on both quantitative and qualitative data from process evaluations,66,69,71,113,167,189,191,193,194 five drew on quantitative data only63,98,110,190,192 and two reports drew on qualitative data only. 88,195 Appendix 5, Table 12 summarises the characteristics of process evaluations included in this synthesis.
Quality assessment
The quality of study reports is detailed in Appendix 5, Table 12. Initial agreement over the quality of studies was high (> 90%). Appendix 5, Table 12 presents a consolidated overview of quality after two reviewers (CB and RP) reached consensus. Three studies were judged to be of both high reliability and high utility in addressing our RQs. 69,71,88,167,194,195 These provided high-quality, in-depth data on a range of factors influencing intervention implementation from different participant perspectives. One study was judged to be of high reliability and of medium utility in addressing our RQs. 189 In this case, the study provided high-quality data on a range of implementation factors, but did not include the perspectives of young people. One study was judged as having medium reliability but low utility, owing to a lack of depth and breadth and privileging of young people’s perspectives in the findings. 63 A further study was judged as being of low reliability, owing to poor reporting of methods, but of medium utility because it explored multiple aspects and a range of perspectives, although the voices of young people were not included. 66 Four studies were judged as being of low quality and low utility,113,190,191,193 largely because of poor reporting of methods, focus on a narrow range of implementation factors and a lack of depth and inclusion of the voices of young people. Three studies were rated as being of high98,110 or medium reliability,192 but low utility, as they provided limited information to help answer our RQs. Only three studies were judged to privilege the voices of young people. 69,71,88,167,194,195 Studies judged to be of poorer reliability or utility were not excluded from the synthesis but were given less interpretive weight in our analysis.
Synthesis of process evaluation reports
Although our protocol did not specify that we would use May’s155 general theory of implementation as a framework to inform our analysis, it was clear when we began our synthesis of evidence from process evaluations that this theory aligned with emergent themes and provided a useful heuristic through which to interpret our findings on the factors affecting the implementation of interventions evaluated in included process evaluations.
Findings from our synthesis of process evaluations are organised below in relation to the four domains of deliverers’ contribution to implementation identified in May’s155 general theory of implementation, which aligned with the emerging themes in our analysis: sense-making (understanding the intervention), cognitive participation (committing to be involved in delivery), collective action (collaborating with others to identify each individual’s role in implementation) and reflexive monitoring (formally and informally assessing the success of implementation and determining further actions). Subthemes are organised in relation to the constructs identified in May’s155 theory as influencing the enactment of these processes. These comprise intervention capability (the possibilities presented by the intervention), institutional capacity (the roles and social norms and the material and cognitive resources present in a setting) and potential (individual and collective commitments to enact the intervention).
There were no obvious patterns regarding which influences on implementation affected which types of interventions. Some themes presented below drew, either in terms of the weight of evidence supporting them or, more often, in terms of the quotations used to illustrate them, largely on UK studies of interventions focused on increasing student participation in decisions. However, rather than reflecting the implementation of this subtype of intervention being particularly affected by certain types of influence, it is more likely that this simply reflects these evaluations being particularly focused on influences on implementation (they involved several of the investigators of the present review), so that they provide detailed evidence and/or rich illustrative quotations. However, where the weight of evidence supporting a theme or subtheme does derive overwhelmingly from such UK studies, this is highlighted. A structure of the way in which themes were organised is provided in the standalone documentation (www.journalslibrary.nihr.ac.uk/phr/AHSN4485).
Sense-making
The concept of sense-making was a recurrent theme in studies as a process that contributed to the enactment of interventions. Sense-making was reported to accrue over time and pervade all, not just the initial, stages of implementation. 66,69,88,189,194 School staff and students were reported to interpret and critically asses intervention resources to make sense of activities and their role in implementing these across time.
Intervention capability to be made sense of
A subtheme from studies of a diversity of interventions suggested that sense-making could be facilitated by an intervention’s ‘capability’ to be made sense of. This could be in terms of providing good-quality materials and ongoing support in the form of training, external facilitation, coaching or peer support. 66,69,71,88,113,167,189,193,194 In particular, materials and resources that included tangible, contextually relevant examples were reported to enable providers to better understand intervention activities and how they might use these in their setting, as was reported by one head teacher in Bonell et al. ’s71 evaluation of Learning Together of high reliability and usefulness:
The one thing schools need is a model, of how it’s going to work in the school, in a real-life school, so that they can almost touch it, taste it, feel it, and then start implementing it in their own schools.
In two studies, it was reported that staff were sometimes initially confused by intervention materials or external providers. 69,71 For example, in the study of the Healthy School Ethos (HSE) intervention, rated as being of high reliability and usefulness,69 an initial presentation by an external facilitator was reported to have caused staff and students to misunderstand the aims of a whole-school intervention:
‘Healthy School Ethos’ was … confused with the government’s ‘National Healthy Schools Programme’, and staff and students, including action-team members, initially assumed the focus was health education or healthy eating.
Bonell et al. 69
School capacity to make sense of an intervention
Another subtheme apparent in one UK study was66,69,71,167,189,190 that staff making sense of interventions could also be influenced by existing school priorities and capacity. 69,167,189,194 For example, those leading implementation in one school were said to have creatively reinterpreted Learning Together, an antibullying intervention, as an intervention aiming to maintain the emotional health of pressurised students in an academically selective school. 194 Warren et al. ’s194 evaluation of high reliability and usefulness reported:
One academically selective school did not view itself as having significant problems with bullying or aggression, and therefore used the [action group] primarily as a way to revise the homework policy.
Warren et al. 194 further reported that, in another school implementing Learning Together, the lead reinterpreted the staff–student action group as being a site for students to learn the skills needed to avoid or respond to bullying, rather than as a site for them to work with staff to co-ordinate intervention activities. This occurred in the context of low school capacity, in terms of the lead’s imprecise grasp of the intervention and inability to involve other staff in the group.
Cognitive participation
The notion of cognitive participation also recurred as a theme across studies, presented as an important process of buying in to implement an intervention. A range of factors concerning the intervention and the school were identified as influencing the extent to which school agents felt able to enrol themselves into a potential ‘community of practice’ to enact intervention activities. As with sense-making, cognitive participation was a process that built across stages of implementation, rather than being a one-time investment at the outset, as illustrated by this quotation from Anyon et al. ’s189 study of the Responsive Classroom intervention, deemed to be of high reliability and medium usefulness:
At the beginning of the year you’re trying to do so much and it was new this year, so I’ve liked being able to just try it out and not have to feel like I’m being judged.
Intervention capability for local tailoring and adding value
A key subtheme apparent in several studies was that school staff assessed interventions in terms of ease of integration with existing school practices and interactions, which influenced whether or not they bought in to an intervention. 66,69,71,167,189,190 Interventions that could be locally tailored or had the potential to build on existing work were more likely to secure staff cognitive participation. Bosma et al. ’s190 evaluation, deemed to be of low reliability and usefulness, of the DARE Plus intervention described how the assessment phase of the intervention was essential to tailor the intervention to the school and hence develop the action team’s buy-in:
The assessment phase of the organizing process is critical to its long-term success. It is invaluable to take the required time to get to know the community before attempting to launch an action team.
Bonell et al. 71 describe how school staff bought in to use of restorative practice because this was viewed as providing a means of building on existing work and developing a consistent approach to discipline across the school:71
Although staff reported that some ‘restorative-type’ approaches were already being used in their school, this intervention was highly attractive because it could provide a new framework, process and resources for embedding restorative practices more consistently and more widely across the whole school.
Interventions not viewed as being capable of local tailoring or fitting with existing work often failed to engender staff buy-in, as reported by evaluations of the Responsive Classroom189 intervention (judged to be of high reliability and medium usefulness) and Positive Action113 intervention (judged to be of low reliability and usefulness). For example, O’Hare et al. ’s113 evaluation of Positive Action reported:
Some teachers and head teachers reported that they were reluctant to make whole-school changes. ‘In terms of ethos, it was a perfect fit. But a lot of what the programme does, we were already doing in ways that were slightly better.’.
A subtheme apparent in some studies was that this lack of intervention capability for tailoring or adding value was particularly undermining for whole-school elements, such as proposed changes to school policies or discipline systems. 113,189 School leaders could withhold buy-in when they felt that whole-school actions might jeopardise their wider strategies. This could be the case, for example, when interventions required changes to school rewards or discipline policies that school leaders thought might weaken the school’s ability to pass school inspections or attract parents to send their children to the school. As O’Hare et al. ’s113 evaluation of Positive Action reported:
Reluctance to change whole-school policy may be exacerbated by circumstances such as an upcoming … inspection: ‘It was hard to make a whole-school change to sanction and reward policy, so whole-school activity was harder to implement. [The government inspectorate] was coming and it would have been too big a change.’.
Intervention capability for using data to build buy-in
Another subtheme that recurred across the UK evaluations was that the provision of local data as part of the intervention could be useful in building buy-in. 66,69,71,88,167,190,194 The evaluation of the Learning Together intervention suggested that providing such data could make it harder for staff to dismiss the need for intervention. 71,167,194 A staff member on a pastoral team, quoted in Warren et al. ’s194 report on Learning Together, commented:
I remember when [facilitator] came to present to [senior leadership team] and said how terrible our data was … it was like a tumbleweed moment … it was a realistic … realisation for everyone if you know what I mean ….
However, in an example of refutational synthesis, several studies identified that the provision of data could sometimes undermine staff commitment when staff interpreted the data as a criticism of their work to date or when data did not indicate positive trends in the course of implementing an intervention. 69,88,167,195 Fletcher et al. ’s88 evaluation of the Learning Together intervention, deemed to be of high reliability and usefulness, reported thus:
The only negative issue… identified was that some… [senior managers] reported that the needs assessment felt too ‘negative’ at times, especially for school managers who had been in post for several years (and who likely saw this as a reflection on their many years of work and leadership at the school).
Similarly, Warren et al. 194 also reported on Learning Together:
Where year-on-year trends in such factors did not improve, staff members sometimes reported feeling dispirited. Such disappointments may have contributed to these schools not continuing intervention activities in the final year of the study.
Intervention capability in terms of provider credibility
The UK studies also suggested that staff were more likely to buy in to interventions when external providers appeared credible and had direct experience working in schools. 66,71,167 For example, in the case of the Learning Together intervention, school staff commitment to implementation was facilitated when the trainer was perceived as someone who understood and could speak to the contextual realities of what implementation might look like for staff in practice:
[T]he assistant head teacher at… [intervention school] suggested that the training would be more acceptable if it was delivered by: ‘Somebody who has established this in their school and can talk about nitty gritty things, like the systems, the processes, how they operate the structures on a day to day basis’.
Bonell et al. 71
Intervention capability in terms of student participation
A subtheme from several different UK evaluations of interventions aiming to encourage student participation in decisions was that students were more likely to buy in to an intervention when this offered an opportunity for them to take on active roles in decision-making or express their views. 69,71,113,167,194,195 Bonell et al.’s71 evaluation of Learning Together, for example, reported:
Key cross-cutting themes in student and staff accounts were that the focus on students ‘having a say’ and greater ‘respect’ for students’ views were important sources of acceptability… the action group was seen as a new ‘experience’ to be more ‘involved’ in ‘having a say’ at school…
The concept of ‘having a say’ reciprocally translated with the notion of having ‘their voice heard’ in the account of a staff member quoted in Warren et al. ’s195 evaluation, deemed to be of high reliability and usefulness:
I think… students will certainly enjoy… that we’re doing something like this so they can be involved in it and… can actually have their voice heard, that they can feel safe at school, that they can feel engaged with the teachers, that they can feel they’re listened to.
Such commitment from students, however, relied on the practice of genuine participation, which, in turn, relied on effective staff facilitation of such processes. When such processes were poorly facilitated and did not result in achieving visible action, this could result in disappointment and disengagement among students. 195
Staff potential for buy-in based on perceived need
Staff buy-in to interventions was reported to be influenced not only by the characteristics of the intervention but also by staff ‘potential’: whether they or the school were ready for such an intervention. A key subtheme here was that interventions should offer school leaders something they already knew they needed. 63,66,113,167,189 This might, for example, be a way of responding to government policies, pressures from parents or inspection requirements. Or it might address internal imperatives, such as school leaders’ existing strategies for school change. This theme was particularly clear in the UK studies of both HSE69 and Learning Together. 71,88,167,194 Fletcher et al. ’s88 pilot evaluation of Learning Together, for example, reported:
The [intervention] appeared… facilitated by features of the English secondary-school ‘market’ whereby parents have… choice over schools and… judge them… on… reputation and performance in ‘league tables’… [H]ead teachers and their [management teams] consistently reported that it was important to address aggressive behaviours … to recruit and retain ‘the best’ parents and students.
Thus, the intervention addressed an existing identified need to tackle aggression. Another example from the Learning Together evaluation concerned schools’ existing need to use needs data to improve school performance so that an intervention that provided this was attractive. From a study deemed to be of high reliability and usefulness, Bonell et al. 167 quote an intervention external facilitator:
[T]here’s an [English school inspectorate] agenda which drives all schools. And [the head] wanted to make sure as well that she was … they want ‘outstanding’, so she wants to make sure that she’s got all the information possible to make sure that they can get through that.
Similarly, interventions aiming to achieve whole-school change were reported to be more likely to get school leaders’ buy-in when there was already a recognition of the need for change, for example because of poor inspection results. 71,88 Reciprocally translating with this concept, it was apparent that, in schools where leaders perceived no such urgent imperative for change, genuine school buy-in was less likely. For example, Bonell et al.’s69 evaluation of the HSE intervention reported:
Progress was initially good at Hillside but was then observed to slow in term 2. Our facilitator thought this partly reflected … less sense of urgency in Hillside about using the project to improve the school.
Staff potential for buy-in based on existing strategies and values
A related subtheme was that school staff were reportedly more likely to buy in to a whole-school intervention when their existing strategies and values made this seem attractive. 63,71,88,110,167,189 For example, new head teachers were reported as particularly likely to buy in to interventions involving whole-school change because these aligned with their desire to make their mark and change schools. Fletcher et al. ’s88 evaluation of Learning Together reported:
[A] factor that appeared to be important at Goldstone Park was that the school had a new head teacher, which supported broader, structural changes to school environment to ensure that restorative practices could be implemented across the whole school.
Reciprocally translating with this concept of school leaders’ potential was teachers’ potential. 63,71 For example, teachers with a prior commitment to social and character education within their classes were more likely to implement curricula addressing this, according to Beets et al.’s63 study of Positive Action, deemed to be of medium reliability and low usefulness:
[T]eacher beliefs regarding their responsibility to teach [social and character development] concepts were significantly … related to their attitudes towards Positive Action …; attitudes towards the Positive Action program were positively related to the amount of the Positive Action curriculum delivered.
Similarly, school staff and students were also reported as more likely to commit to enacting restorative practices if staff already had some interest in this approach. A head teacher reported this in Bonell et al.’s71 evaluation of Learning Together:
…what attracted us was the idea that, you know, we could get something out of it… I mean, we’ve been trying to develop a kind of, more restorative approach to secure behaviour over a period of time [and] so this fitted in with the way we want to work.
In cases in which the values or priorities did not align, staff buy-in appeared less likely. 63,110,189 For example, when staff or students perceived restorative practice to be a softer option, they were reportedly unlikely to commit to enacting it, either rejecting it or implementing tokenistically. 189 As Anyon et al.’s189 study of Responsive Classroom reported:
In contrast, some middle school staff members’ beliefs about the value of punitive responses to problem behavior were incompatible with the core tenets of the intervention … These staff members believed that zero-tolerance policies … were more effective than RC [Responsive Classroom] approaches, which aim to enhance students’ intrinsic motivation to improve.
A subtheme among the UK studies concerned the possibility of schools committing to implementing only those intervention components that aligned with their existing strategies and values, rejecting or soft-pedalling components that they regarded as deviating from these. 69,167 Bonell et al.’s69 evaluation of the HSE, for example, reported:
[A]spects of each school’s ethos supported… the intervention while [others] were less supportive …. At Hillside… aspects of HSE were supported by the school’s… ethos of… involving students in decisions, while others such as the action on rewards may have been impeded by… limited baseline attention to engaging disaffected students.
Collective action
As well as exploring staff and student understandings of, and buy-in to, interventions, evaluations also examined the extent to which those in schools actually worked together to divide up responsibilities for delivering interventions. A number of factors were identified as influences on such processes.
Intervention capability as workable
A key subtheme was the importance of interventions being locally workable in staff enacting interventions as planned. 69,167,189 For example, curriculum materials that did not fit into the school curriculum or that did not provide staff with clear lesson plans or engaging materials tended to be adapted before they were delivered, or were not delivered at all. 66,167,194 Bonell et al.,167 for example, reported:
Many schools did not deliver the curriculum materials as they were packaged. Staff in several schools commented that that they needed to adapt the curriculum because it did not address the needs of their students, or because the materials were too simplistic.
Reciprocally translating with this general notion of workability was the notion of workability in terms of enactment in some settings more than others. The evaluation by Anyon et al. of the Responsive Classroom intervention reported that classroom elements were more workable in primary grade classrooms than in middle-school grade classrooms. 189 Teachers in the middle-school grade classrooms reported that the interventions could not be made to fit into their much more structured school day:
[T]he implementation of RC [Responsive Classroom] strategies took time away from teaching reading or math skills.
Anyon et al. 189
An important aspect of workability was the extent to which guidance materials spelt out how delivery should proceed. For example, materials underpinning restorative practice interventions needed to specify which staff members were responsible and whether the intervention was intended to complement or replace punitive discipline. 189
Some interventions were not collectively enacted as had been planned because their developers had not anticipated whether or not they would really work in the complex environment of schools. For example, Anyon et al.’s189 evaluation of the Responsive Classroom intervention found that a new approach to discipline failed to work in the reality of schools:
[P]articipants reported that a key RC [Responsive Classroom] strategy, Logical Consequences, in which a response to student misbehavior is tied to the specific incident and creates an opportunity for learning, was too unwieldy to implement in a way that students could anticipate and incorporate.
Planning groups as a key element of intervention capability
An important subtheme here was that interventions that included planning groups consisting of staff, sometimes also with involvement of students, parents or other community members, were reported to be particularly successful in ensuring collective action to enact interventions. This was apparent from reports of the Gatehouse Project intervention (of low reliability and medium usefulness), the Learning Together intervention (of high reliability and usefulness)66,69,71,167,194,195 and other interventions. 63,66,110,113,190,193
Diverse participation in such groups could support implementation by ensuring that the decisions made by the group were pragmatic and by achieving wider buy-in across the school community. Such groups were reported to be particularly facilitative of whole-school approaches. 66,69,194,195 Bond et al. ,66 in relation to the Gatehouse Project, reported that:
Because the make-up of the team was broadened to include administration and curriculum, not only has the work of the adolescent health team facilitated reviews of organizational structure, but it has also contributed to a substantial shift in the perceptions of what is the core business of schools.
Such groups could also help ensure that intervention activities added up to a co-ordinated process of integrated school transformation, rather than merely a disparate set of initiatives, for example as reported in Bond et al. ’s66 evaluation of the Gatehouse Project:
A consistent response from the key informants was that the Gatehouse Project offered a framework for coordinating such work and a set of lenses with which to review current strategies and implement new ones …. Supporting schools to make those links and to see the connections was … important.
Planning groups and participative decisions as a potential source of deviation
An element of refutational synthesis was also apparent in the themes arising on the topic of planning groups. A recurring subtheme from evaluations of UK interventions promoting student participation in decisions was that, although such processes could encourage collective action, this could also lead to schools enacting activities that deviated from intervention aims. 69,88,167,194 In some cases, school leaders were aware of this and sometimes judged that it was important to endorse decisions made by planning groups, even if they did not agree with them:
I’m not sure how effective [the safe spaces is] gonna be, but it’s an idea that’s come from the group, therefore it has to happen.
Head, Woodbridge, Bonell et al. 69
Risks of deviation could be mitigated by deployment of external facilitators or thorough training of those leading implementation within schools. 88 Facilitators might aim to ensure that ‘fidelity of function’ (sticking to the goal of activities) was preserved even when there was latitude about ‘fidelity of form’ (maintain the form such activities took). 203 Fletcher et al. ’s88 evaluation of the Learning Together pilot, for example, reported:
Observations of action group meetings did include some examples of students’ suggestions that may have done more harm than good … but external facilitators’ oversight role meant that they were able to limit any actions that may have done more harm than good in terms of bullying and aggression.
In some cases, schools deviated from the prescribed process, but in a way that still aligned well with intervention functions. Warren et al. ’s evaluation of Learning Together reported:194
A few [schools] deviated from the processes of working set out in the manual. Two schools broke the [action group] into multiple sub-committees.
In other cases, local decisions resulted in actions that did not accord with intervention theories of change. This was particularly likely to occur when the sense that school agents made of the intervention differed from that intended by the intervention developers, with the examples already given of schools using action groups to support pressurised students’ mental health or teaching students social and emotional skills in what was ostensibly a planning meeting.
Synergy between intervention components as a key element of intervention capability
A further notable subtheme was that some interventions had better synergies between intervention components than others, and this was important in enabling them to be implemented more successfully. 66,69,71,88,167,189,194,195 Synergy appeared to occur when the consequences of enacting one intervention activity helped facilitate the conditions required for the implementation of other components. This could ensure that priorities were identified, commitments built, and co-ordination and momentum created. In such cases, some intervention activities created the informational and relational resources needed to enable agents to enact other actions. For example, Bond et al. ’s66 evaluation of the Gatehouse Project reported the following:
It is clear from our work that these elements – the adolescent health team, the school social climate profile, and the critical friend – do not work in isolation …. The critical friend provides expertise, impetus, motivation, and links to external resources.
As described previously, data on student needs being provided as part of an intervention could encourage others to implement intervention activities. Data being provided as part of the intervention could lead to school staff producing or sharing other data. This, for example, was reported as an unexpected consequence of the sharing of student needs data in the UK studies of the Learning Together pilot:71,88
It was clear that all the schools in the pilot were very ‘data rich’ environments and this presentation of new, external student survey data to the action group triggered them to identify, analyse and triangulate multiple sources of data which they would not typically have used.
Fletcher et al. 88
External facilitation or coaching could similarly enable better implementation of other intervention activities. As was reported in Bond et al. ’s66 study of the Gatehouse Project:
School personnel identified the critical friend as an important aspect of the support network within the school, helping to keep up the momentum for the school-based team and developing shared understandings.
One obvious area of intervention synergy was where training components provided staff with the skills they needed to deliver other intervention elements. This could be valuable in ensuring staff accumulated and consolidated their skills, as was reported by a staff member in Anyon et al.’s + 208 study of the Responsive Classroom intervention:
One thing that was really helpful … is that I would use the language and then forget the language and so you’re coming back in and checking in with us in the [booster session] meetings, [which] really remind[s] me, ‘oh, yeah, I gotta do that’.
Some evaluation reports focused on the concept of synergy as facilitative of collective action, whereas other reports focused on lack of synergy as an inhibitor of collective action. For example, some evaluations reported that there was a noticeable lack of effective interaction between curriculum and whole-school components. In some cases, classroom curriculum activities were enacted, but whole-school changes intended to reinforce what students learned in these classes were incompletely delivered. 113 In another case, whole-school elements that aimed to build on existing school achievements were enacted, but curriculum elements were not delivered with fidelity because these were judged as unworkable by staff. 167,194
If intervention materials did not explain how components were intended to interact, this could also hinder collective action. Bonell et al.’s71 evaluation of Learning Together reported that:
[O]other staff reported ‘not knowing enough’ and that it was not always clearly communicated to them how it all ‘married up’.
School capacity to support collective action
The extent to which agents in schools could come together to collectively enact interventions also depended on the school resources available to these agents.
Time resources
Lack of time was reported to hinder collective action in almost all schools and interventions. The chronic lack of space in school timetables, and the lack of non-contact time within which school staff could plan intervention activities, were frequently reported by evaluations. 71,113,167,189–191,193 For example, Cross et al. ,191 in their evaluation (deemed to be of low reliability and usefulness) of the CFS intervention, reported the following:
Many teachers reported not being able to find sufficient time in their teaching curriculum to complete the eight learning activities. Some teachers indicated they could find almost no implementation time.
In the evaluations, judged to be of low reliability and usefulness, of the DARE Plus intervention and the PPP intervention,190,193 whole-school elements were described as being the most challenging and time-consuming to organise. Mitchell et al.’s193 evaluation of PPP, of low reliability and usefulness, reported that:
One reason for the weakness in the parent involvement component may be that developing close communications between parent and school is a time-consuming task.
Similarly, Bosma et al.’s190 evaluation of DARE Plus reported the following:
Community organizing takes time. While the youth and adult components complemented each other, it was challenging for organizers to balance the workload. The DARE Plus Project community organizer positions were 75% time … but these… could… have been full-time. Many organizers put in additional time on behalf of the project.
Staff in one study suggested that the time commitment for implementation should have been more clearly laid out at the outset. 71
School staff could struggle to marshal finite time (and other resources) when they were expected to deliver a new intervention simultaneously alongside other innovations or initiatives. These situations diffused the resources available for any one intervention and eroded agents’ ability to commit the time needed to support effective decision-making and delivery. Anyon et al.’s189 evaluation of the Responsive Classroom intervention, for example, reported the following:
A school leader noted that ‘It’s not one new thing; it’s always five new things that we’re working on. I think the attention span is tested.’ … The simultaneous implementation of new initiatives limited the amount of professional development that could focus exclusively on RC [Responsive Classroom].
Other studies emphasised the more general limited capacity among teaching staff who were taking on intervention implementation on top of their usual roles. As was reported by one of the facilitators in the evaluation of Learning Together:
[Teacher has] … been given so much to do I think in his day-to-day role, it’s just another thing for him to do. He does it, he does it with very good grace ….
Bonell et al. 167
Leadership resources
A key resource factor in determining whether or not interventions were collectively enacted with fidelity was whether or not those charged with leading the intervention were in possession of leadership resources such as a budget, the ability to direct other staff or the ability to modify policies or systems. 69,71,73,88,167,189,190,195 Schools that gave intervention leadership roles to powerful staff consistently achieved better levels of implementation, according to several evaluations. Bonell et al.’s69 evaluation of the HSE intervention, for example, reported the following:
Action-team members reflected that this work had progressed well because it was led by a senior staff-member with the knowledge and capacity to get things done: ‘I think that importantly there are key staff … for example [the assistant-head] … who’s really pushed that idea of student-led stuff forward’
Teacher, action-team member, Woodbridge, Bonell et al. 69
Power and authority could be formal or informal, the latter reflecting individuals or groups having a long track record at the school, strong relationships and an informal ability to persuade people to make things happen. 167,194 Warren et al. ’s194 evaluation of Learning Together, for example, reported the following:
In another school, despite there being no senior leaders on the group, the lead had worked for a long time at the school and was… respected and liked by… students and staff. Thus, it was possible to galvanise action without the formal involvement of senior leaders in some cases.
When leadership was delegated to less powerful staff, these struggled to ensure wider staff participation or to enact changes to policies or systems. When leadership commitment to intervention activities was limited or inconsistent, there may have been less collective vision and impetus for implementation, as reported in the evaluation of the Responsive Classroom intervention. 189 Lack of senior-level support could also affect the drawing down of material and cognitive resources to support intervention activities. 167,194 Warren et al. ’s194 evaluation of Learning Together reported that:
In other schools, the chair received little support from the head-teacher or other school leaders: ‘The head teacher there was completely uninterested when I came to give a talk to the senior leadership team’.
Facilitator, Warren et al. 194
For example, some decisions made by action groups were stalled or rejected by other agents within the school system, such as head teachers or school-leadership teams. 167,194 Bonell et al.,167 in their evaluation of Learning Together, reported the following:
According to the facilitator [some schools] repeatedly revised minor issues, such as school’s policy on make-up, but was largely unable to confront more substantive issues relating to student behaviour, health or wellbeing. In [one school] the AG [Action Group] proposed numerous actions that were rejected by the head teacher.
Staff/school relational and culture resources
Another recurrent subtheme was that interventions could be better or more widely implemented in school systems characterised by strong connections between staff or with strong cultures of innovation. 63,66,110,167,189 In schools with strong connections, those agents leading interventions could draw on existing relational resources such as mutual support, observation and learning to support enactment, rather than attempting to develop this from a low baseline. Malloy et al.’s110 evaluation of Positive Action, judged to be of high reliability and low usefulness, reported:
Stronger affiliation among teachers likely led to more opportunities to share ideas about PA [Positive Action] materials and observe other teachers as they carried out PA activities outside of the classroom. This may have influenced teachers’ use of these supplementary program components.
Bond et al. ’s66 evaluation of the Gatehouse Project similarly reported the importance of networks connecting staff in enabling collective action:
To plan and implement such wide-reaching change clearly required schools to have considerable support networks, both within the school and often from outside.
A culture of teacher autonomy, as reported in the evaluation of Friendly Schools,169,197 could undermine collective action, because it was difficult for those leading an intervention to encourage the consistent enactment of new practices that deviated from locally understood norms and expectations of staff roles. Similarly, Anyon et al.’s189 evaluation of the Responsive Classroom intervention reported the following:
School staff observed that RC [Responsive Classroom], a schoolwide intervention, ran counter to the school’s culture of individuality. For example, one teacher noted: ‘One … characteristic of [the school is] … there’s a lot of autonomy in terms of how teachers run their classrooms.’
A staff culture of innovation could also support collective implementation. Such cultures could encourage staff to take the time to identify who would implement the intervention and then enact this with fidelity. 63,113
Reflexive monitoring
Whole-school interventions took time to build as school agents gradually came to define their roles in intervention processes and develop the commitment and cognitive capacity they needed to facilitate implementation. Reflexive monitoring, whether through formal or informal processes, was important in determining the extent to which implementation built or dissipated over time.
Intervention capability for reflexive monitoring
Processes of reflexive monitoring were best supported when interventions included this as an integral component. 66,69,88,167 Studies indicated that interventions were particularly successful when they included an action group that reviewed data, identified priorities, oversaw delivery and reflected on the results. Such processes enabled members to reflexively monitor what was being enacted and with what consequences. Evaluations suggested that this gave participants the permission and resources to try different things, persist with what was perceived as working and refine or reject what was perceived to go less well. This approach allowed staff to abandon activities viewed as unsuccessful without rejecting the intervention overall. For example, Bond et al.’s66 evaluation of the Gatehouse Project reported that:
This common purpose gave permission for teachers to try new strategies such as substantially restructuring student and teacher teams … [I]n one school, teachers worked together to reorganize classes into small groups of four or five learners and teachers into teaching teams to promote a collaborative and an academic environment.
These steps required staff to meet frequently to review information and monitor consequences. 190
As part of processes of reflexive monitoring, quick wins evidencing positive outcomes can help maintain and further build coalitions and commitment, and collective impetus to implement further intervention activities. Bosma et al.’s190 evaluation of DARE Plus, for example, reported that:
Teams with the highest numbers of activities were those that developed the most momentum, with each success leading to new ideas for working on issues.
As well as groups, ongoing support from training, facilitation or coaching could also support reflexive monitoring by providing an opportunity for reflection and/or an outsider perspective. The importance of an external facilitator was, for example, described as follows in Bond et al.’s66 evaluation of the Gatehouse Project:
The support that [critical friend] provided in the staff room, in staff meetings, has been invaluable. We wouldn’t be where we are now, because I’d never recognized the value of having a person who is not a practicing teacher in the school at the moment.
Collective reflexive monitoring to refine implementation
Reflexive monitoring could be a collective action oriented towards refining how an intervention was implemented. 66,193,194 For example, in the case of two interventions, over time, staff in some schools opted to recruit fewer disengaged or disadvantaged students to participate in intervention activities. 167,193 Mitchell193 reported of the PPP:
In 1989–90, school counsellors made a concerted effort to select at-risk, as well as able, students for the peer helper program … In 1990–91, counselors were more selective in choosing peer helpers and as a result, fewer high-risk students were selected as peer helpers.
When external facilitators were removed in the Learning Together intervention, this resulted in the overall fidelity of implementation declining, but some intervention components became mainstreamed so that their ‘form’ was modified at the same time as their ‘function’ became integrated within school policies and systems, as Warren et al.’s194 evaluation reported:
The absence of [facilitator] has been incredibly significant because she … was able to tie it in all the time to the agenda. And was a touchstone I suppose really for that. And then … so that … I think that was a loss.
Senior leadership team member, Warren et al. 194
Reflexive monitoring reinforcing implementation
A final subtheme that emerged from process evaluations was that reflexive monitoring could reinforce the conditions necessary for further implementation. 66,69,167,193 Staff and students in schools recognised through processes of reflexive monitoring that interventions had diverse consequences for different parts of the school systems, many of which were unanticipated. Instituting action teams with diverse membership could lead to staff with previously discrete roles coming to think more holistically about student development and well-being. For example, Bond et al. ’s66 evaluation of the Gatehouse Project reported that:
There was also evidence of changing professional identity – teachers shifted their position from being a teacher of a subject or program to placing the young person and learning at the center of practice.
Similarly, involving students in decision-making or being surveyed about their needs could transform staff and student attitudes by suggesting that the school was becoming a more participative institution. Bonell et al.’s69 evaluation of the HSE intervention reported:
I was speaking to some younger students … they said, ‘I can’t believe this is happening and we’re having a say in the rules …’. And they think, ‘if I make the rules, then I shouldn’t go against my own rules’.
Student action team member, Woodbridge, Bonell et al. 69
Consultation with stakeholders
Policy and practice stakeholders found the results of the process evaluation synthesis plausible. In particular, it was felt that this confirmed the importance of context, and locally tailoring interventions to fit this. This was because different schools have different student populations and different needs, so that a one-size-fits-all approach would not work. It was also suggested that whole-school interventions should build on existing school priorities and capacities. Interventions should involve actions that school leaders are prepared to support. Different issues and even different outcomes will build ownership and stimulate action in different schools. Which issue and outcomes are chosen may not matter as long as these stimulate action because these interventions appear to have broad effects across multiple outcomes.
Stakeholders supported the emphasis in the process evaluation synthesis on needs assessment and action groups being key components. Some suggested that, rather than merely assessing need, initial assessments also needed to consider school capacity, existing initiatives and priorities.
Participants supported our emphasis on building staff engagement with interventions. Staff needed to be persuaded of the reasons why implementing an intervention was their job. Intervention activities that provided a smarter means of doing one’s current job were more attractive to staff than those creating new jobs. However, other participants indicated that, even when interventions provided another means of doing one’s current work, this would nonetheless add to workloads, at least in the short term, so that building staff buy-in was important. Several participants suggested that some school staff would resist interventions aiming to transform school learning or discipline, adding to the importance of building staff belief before implementation commenced.
The ALPHA group of young researchers felt that both teacher commitment and the acceptability of interventions to young people were key in ensuring successful implementation. They noted limited evidence on student perspectives on interventions in our results. There were elements of restorative practice, for example, that they thought some students might find unacceptable. The ALPHA participants recommended involving young people in the development of future interventions.
Chapter 7 Results: synthesis of outcome evaluations
Reports included in the outcome evaluation synthesis
We included 23 outcome evaluations of 20 interventions described across 4852,61,64,67,68,73,80,85,87,103,108,109,115,117,123,124,161–188,197–200 reports, published between 1986 and 2021. Of the included studies, 18 were RCTs and 5 were quasi-experimental studies. Twenty-one studies reported on violence outcomes,52,61,64,67,68,73,80,85,87,103,108,109,115,117,123,124,161–173,177–188,197–200 12 on substance use52,61,64,67,68,87,103,108,109,115,117,123,124,162–168,174–177,179–188,198 and 7 on educational attainment. 52,61,64,80,85,108,109,123,124,162–164,172–174,179,198
See Table 3 for a list of all reports included in the outcome evaluation synthesis; see Appendix 6, Table 13, for characteristics and quality assessment of included outcome evaluations.
Syntheses presented
In this chapter, we present meta-analyses of intervention effects by outcome (violence perpetration, violence victimisation, observed violence, substance use and academic attainment), and then by intervention type (interventions promoting student participation in school policy decisions; interventions promoting student relationships with teachers, but not student participation in school policy decisions; interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers; interventions promoting parent involvement only), by follow-up time (up to 1 year post baseline and > 1 year post baseline) and, finally, by study design (RCT and non-randomised evaluation). Owing to the number of trials and outcome included, we present the narrative synthesis in Appendix 7.
Where relevant, we also report meta-analyses of RCTs for violence perpetration, violence victimisation and substance use, considering substance use both overall and stratified by substance type: alcohol, tobacco, illicit drug use and ‘omnibus’ substance use outcomes (e.g. frequency or prevalence of a range of substances). We also present meta-analyses for interventions overall by outcome and time point.
Our syntheses deviate in several ways from those planned in the protocol. These are summarised in Appendix 1, Table 8. Given that many interventions spanned multiple age groups and included multiyear longitudinal follow-up, we also did not attempt to synthesise evidence by age group. We do not present meta-analyses for observed violence or for academic attainment: the former because of a mix of informants for this outcome, and the latter because it became apparent in the course of the review that there is an additional body of evidence reporting the effects on educational attainment of the sorts of whole-school interventions as defined in this review, but these studies were not included in the present review because they did not also report effects on violence or substance use. Therefore, our synthesis of effects on educational attainment from the studies included in the present review should be considered as non-comprehensive and hypothesis-generating, rather than as comprehensive and hypothesis-testing.
Meta-analyses include only RCTs because the analytic methods that were used in included non-randomised evaluations meant that effect sizes from these evaluations could not systematically be computed for inclusion in a meta-analysis. For example, non-randomised studies used inappropriate accounting for clustering (Project PATHE,174 CDP162–164,198), used only one comparison school (Project PATHE)174 or used matching with no clear origin points for intervention exposure, for example comparing cross-sectional samples in schools that had implemented an intervention for several years versus schools that had not (Flay et al. 85 and Flay and Allred173).
We undertook random-effects modelling because of the statistical heterogeneity of included studies, but we did not undertake sensitivity checks with fixed-effects models, as this would have been redundant given the evidence identified; this is because the clinical heterogeneity of included interventions, and outcome measures and follow-up times, meant that it would be imprudent to ignore any between-study variance, however slight. Put another way, even in meta-analyses with low heterogeneity (as measured using I2), we believe that the interpretation of meta-analyses should always be closer to a random-effects model (average of the distribution of effects) than to a fixed-effects model (the common effect, with variation only due to sampling error). However, we did undertake sensitivity analysis of meta-analyses with only two studies using a multilevel model with a compound symmetry matrix of 0.8 within study, to address the instability of these models in the robust variance estimation framework, which was our primary approach. Furthermore, meta-analyses were frequently too small in number of effect sizes to explore heterogeneity where I2 was > 50% or to undertake meta-regressions considering moderation.
Finally, we did not explore moderation of effects between studies using meta-regression or qualitative comparative analysis because of the insufficient number of comparable studies. Study-level differences in effectiveness were too inconsistent and the number of studies too small to calibrate studies into effective and ineffective sets; hence, qualitative comparative analysis was not possible.
Risk of bias and quality of studies
For RCTs, our risk-of-bias assessment covered a range of domains, as prescribed by the Cochrane Collaboration Risk of Bias tool. 138
Appendix 6, Table 13, shows the results of our risk-of-bias assessment for each outcome evaluation study. Figure 9 shows a risk-of-bias graph by domain for RCTs. Figure 10 shows a risk-of-bias graph by domain for non-randomised studies.
Risk of bias of randomised controlled trials
Sequence generation
Of the 18 trials included, 752,61,80,87,108,109,165–168,172,175–177,179–181,184,197 had adequate sequence generation and 1164,67,68,103,115,117,123,124,161,169–171,178,182,183,185–188,199,200 did not state how sequence generation was undertaken.
Allocation concealment
Of the 18 trials included, 1552,61,64,67,68,87,103,108,109,115,117,123,124,161,169–171,175–188,199,200 did not state how allocation concealment was undertaken. The remaining three80,165–168,172,197 trials provided information on how allocation was concealed, which was generally through the use of an independent statistician.
Blinding of participants, personnel and outcome assessors
Only one trial165–168 reported blinding of participants, study personnel and outcome assessors in regard to treatment allocation. A further 13 trials64,67,68,87,103,115,117,123,124,161,169,170,177,178,182,183,185–188,197,199,200 did not report information with which to make a judgement. Three trials52,61,80,108,109,172,175,176,179 did not include blinding of participants, personnel or outcome assessors. In one trial,180,181,184 participants and personnel were not blinded, but outcome assessors were.
Complete outcome data
Ten trials64,80,87,103,117,123,124,165–170,172,177,178,185–188,199,200 presented complete outcome data, whereas six trials52,61,67,68,108,109,115,171,175,176,179–181,184,197 did not and two trials161,182,183 were unclear. Specifically, one trial rated as not having complete outcome data52,61,108,109,179 did not have complete outcome data in the interim findings report,109 with subsequent reports from this trial rated as ‘unclear’ for completeness of outcome data. Complete outcome data were defined as balanced retention in trial arms with attrition of < 30%. Of the six trials without complete outcome data, three trials67,68,115,171,180,181,184 also had notably imbalanced attrition between arms.
No selective outcome reporting
Only one trial185–188 had evidence of selective outcome reporting, whereas six trials80,161,165–168,172,175,176,180,181,184,197 did not, with remaining trials at unclear risk of bias because of selective outcome reporting. For the one trial with evidence of selective outcome reporting, outcomes for bullying perpetration required further contact with authors to retrieve relevant data.
Reduction of other sources of bias
Two trials80,170,172 did not reduce other sources of bias, whereas six trials reduced other sources of bias by using longitudinal analysis methods with baseline adjustment,87,165–168,177 using appropriate estimators to account for non-normal distributions,64,123,124,161,185–188 and multiple imputation. 197
Quality of non-randomised studies
Bias due to confounding
Of the five non-randomised evaluations included, two162–164,174,198 were rated as at critical risk of bias due to confounding, because of inappropriate control for clustering and insufficient control for confounding. Two85,173 were rated as at moderate risk of bias because confounding was expected, but appropriate confounding domains were included. One73 was rated as having a low risk of bias as confounding domains were included, results were adjusted for clustering and school allocation was quasi-random.
Bias in selection of participants for the study
One non-randomised evaluation162–164,198 was rated as being at serious risk of bias due to selection of participants for the study because the long-term follow-up163 reached a relatively small fraction of the full sample, with systematic differences between the original and follow-up samples. Two85,173 were rated as being at moderate risk of bias as start and follow-up of the intervention did not coincide for all participants. Two73,174 were rated as having a low risk of bias as analyses were prospective and follow-up coincided for all participants.
Bias in classification of interventions
Three non-randomised evaluations73,162–164,174,198 were rated as being at low risk of bias from classification of interventions. Two non-randomised evaluations85,173 were rated as being at moderate risk of bias because of how exposure was operationalised in these studies (e.g. as having implemented the intervention for > 4 years).
Bias due to deviations from intended interventions
All five evaluations were rated as being at low risk of bias due to deviations from intended interventions.
Bias due to missing data
Three evaluations73,85,173 were rated as being at low risk of bias due to missing data. One evaluation162–164,198 was rated as being at moderate risk of bias due to missing data, given high numbers of missing data and the lack of an analysis strategy to account for this. One evaluation73 did not present enough information to reach a judgement in this domain.
Bias in measurement of outcomes
Three evaluations73,162–164,174,198 were rated as being at moderate risk of bias in this domain because of the possibility of knowledge of assignment influencing outcome measurement. Two evaluations85,173 were rated as being at low risk of bias because of the use of administrative data for outcomes.
Bias in selection of the reported result
One evaluation73 was reported as being at low risk of bias in this domain. The remaining four85,162–164,173,174,198 were rated as being at moderate risk of bias because, although outcome reporting was reasonably complete, there was no point of comparison to confirm this.
Effects on violence perpetration
Nine outcome evaluations presented findings for this outcome up to 1 year post baseline: eight RCTs87,169–171,177,178,180–188,200 and one non-randomised evaluation. 73 Fourteen outcome evaluations presented findings for this outcome at time points > 1 year post baseline: 13 RCTs52,61,64,87,103,108,109,117,123,124,165–171,177,179–188,197,199) and 1 non-randomised evaluation. 162–164,198
Interventions promoting student participation in school policy decisions
Within this intervention category, three RCTs87,177,178,180,181,184,200 and one non-randomised evaluation73 presented outcomes up to 1 year post baseline. Seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,177,179–181,184,199 and one non-randomised evaluation162–164,198 presented outcomes > 1 year post baseline.
Up to 1 year post baseline
Findings from three RCTs87,177,178,180,181,184,200 and one non-randomised evaluation73 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence perpetration at up to 1 year post baseline (Figure 11). This was reflected in a meta-analysis suggesting an overall non-significant reduction in violence perpetration, but with significant heterogeneity between studies. A meta-analysis including four effect sizes from three studies87,177,178,180,181,184,200 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline [odds ratio (OR) 0.83, 95% confidence interval (CI) 0.65 to 1.08]. However, this finding had substantial heterogeneity, with I2 estimated at 62.7%.
More than 1 year post baseline
Findings from seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,177,179–181,184,199 and one non-randomised evaluation162–164,198 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence perpetration at > 1 year post baseline (Figure 12). This was reflected in a meta-analysis suggesting a non-significant and heterogeneous reduction in violence perpetration. A meta-analysis including 32 effect sizes from seven studies52,61,64,87,103,108,109,117,123,124,165–168,177,179–181,184,199 yielded a non-significant pooled estimate of interventions’ impact in reducing violence perpetration > 1 year post baseline (OR 0.69, 95% CI 0.47 to 1.03). This pooled effect was estimated with substantial heterogeneity, represented in an I2 of 83.9%.
Interventions promoting student relationships with teachers, but not student participation in school policy decisions
In this intervention category, two RCTs170,171 presented outcomes both up to 1 year post baseline and > 1 year post baseline.
Up to 1 year post baseline
Findings from two RCTs170,171 suggested possible, but inconsistent, impacts on violence perpetration of interventions promoting student relationships with teachers, but not student participation in school policy decisions, at up to 1 year post baseline (Figure 13). This was reflected in a meta-analysis suggesting a non-significant and moderately heterogeneous reduction in violence perpetration. A meta-analysis including nine effect sizes from two studies170,171 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline (OR 0.77, 95% CI 0.14 to 4.21). This finding had a moderate amount of heterogeneity, with an I2 of 37.1%. A sensitivity analysis model for this meta-analysis using fixed effects did not converge; thus, we could not compare robustness under different estimation methods.
More than 1 year post baseline
Findings from two RCTs170,171 suggested unlikely and inconsistent impacts on violence perpetration of interventions promoting student relationships with teachers, but not student participation in school policy decisions, > 1 year post baseline (Figure 14). This was reflected in a meta-analysis suggesting a non-significant and moderately heterogeneous reduction in violence perpetration. A meta-analysis including nine effect sizes from two studies170,171 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.88, 95% CI 0.10 to 7.68). This finding had a moderate amount of heterogeneity, with an I2 of 56.2%. However, a sensitivity analysis using fixed effects suggested an impact of similar magnitude, but with borderline significance (OR 0.85, 95% CI 0.73 to 1.00). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.
Interventions promoting student engagement in learning, but not student participation in decision-making or improving relationships with teachers
In this intervention category, two RCTs182,183,185–188 presented outcomes both up to 1 year post baseline and > 1 year post baseline.
Up to 1 year post baseline
Findings from two RCTs182,183,185–188 suggested that interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, were unlikely to affect violence perpetration up to 1 year post baseline (Figure 15). This was reflected in a non-significant meta-analysis with minimal heterogeneity. A meta-analysis including two effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of violence perpetration at up to 1 year post baseline (OR 0.90, 95% CI 0.78 to 1.05). This finding did not include substantial heterogeneity, with an I2 of 0%.
More than 1 year post baseline
Findings from two RCTs182,183,185–188 did not suggest clear impacts on violence perpetration of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, > 1 year post baseline (Figure 16). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including two effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.77, 95% CI 0.01 to 74.3). This finding included substantial heterogeneity, with an I2 of 94.9%. A sensitivity analysis also did not suggest a possible significant impact of interventions (OR 0.78, 95% CI 0.47 to 1.28).
Interventions promoting parent involvement only
In this intervention category, two RCTs169,197 presented included outcomes. Only one (Friendly Schools) 169 presented findings up to 1 year post baseline; both presented findings > 1 year post baseline.
Up to 1 year post baseline
At the end of the first school year of implementation of Friendly Schools,169 control group students (n = 863) and Friendly Schools students (n = 984) were not significantly different in their odds of bullying peers every few weeks (OR 0.66, 95% CI 0.39 to 1.13) or bullying peers at all (OR 1.15, 95% CI 0.81 to 1.63). This intervention targeted grade-4 students, who were 8–9 years of age at the trial’s start. ORs are expressed as the increase in odds of bullying in the control group, that is the first estimate, but not the second estimate, reflects a numerical benefit to the control group. Because only one RCT169 reported outcomes in this category, no meta-analysis was possible.
More than 1 year post baseline
Findings from two RCTs169,197 suggested that interventions promoting parent involvement only were unlikely to affect violence perpetration > 1 year post baseline (Figure 17). This was reflected in a non-significant and minimally heterogeneous meta-analysis finding. A meta-analysis including eight effect sizes from two studies169,197 suggested a non-significant impact in increasing the odds of violence perpetration at up to 1 year post baseline (OR 1.04, 95% CI 0.997 to 1.09). This finding included minimal heterogeneity, with an I2 of 0%. A sensitivity analysis generated a similar finding (OR 1.04, 95% CI 0.94 to 1.15).
Overall meta-analysis
Overall meta-analyses suggested that interventions promoting commitment to school generate small, but statistically significant, impacts, potentially of public health significance, in reducing violence perpetration at up to 1 year post baseline (OR 0.85, 95% CI 0.76 to 0.96). This analysis drew on 17 effect sizes from eight studies169–171,177,182,183,188,200 and included a moderate amount of heterogeneity, with an I2 of 57.4% (Figure 18). Interventions promoting commitment to school also generate small but statistically significant impacts, also potentially of public health significance, in reducing violence
perpetration > 1 year post baseline (OR 0.79, 95% CI 0.65 to 0.98). This analysis drew on 54 effect sizes from 13 studies64,103,108,109,166,168–171,177,180,182,188,197,199 and included a substantial amount of heterogeneity, with an I2 of 81.8% (Figure 19).
Violence victimisation
Nine outcome evaluations presented findings for this outcome up to 1 year post baseline: eight RCTs67,68,103,115,117,169–171,178,180,181,184–188,200 and one non-randomised evaluation. 73 Twelve outcome evaluations presented findings for this outcome at time points > 1 year post baseline: 11 RCTs67,68,80,103,115,117,161,165–172,180,181,184,197,199 and one non-randomised evaluation. 162–164,198
Interventions promoting student participation in school policy decisions
Within this intervention type, three RCTs103,117,178,180,181,184,200 and one non-randomised evaluation73 presented findings for this outcome up to 1 year post baseline. Five RCTs80,103,117,165–168,172,180,181,184,199 and one non-randomised evaluation162–164,198 presented findings for this outcome > 1 year post baseline.
Up to 1 year post baseline
Findings from three RCTs103,117,178,180,181,184,200 suggested possible, but inconsistent, impacts on violence victimisation at up to 1 year post baseline of interventions promoting student participation in school policy decisions (Figure 20). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including nine effect sizes from three studies103,117,178,180,181,184,200 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.79, 95% CI 0.42 to 1.08). This finding included substantial heterogeneity, with an I2 of 92.8%.
More than 1 year post baseline
Findings from five RCTs80,103,117,165–168,172,180,181,184,199 and one non-randomised evaluation73 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions on violence victimisation > 1 year post baseline (Figure 21). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 45 effect sizes from five studies80,103,117,165–168,172,180,181,184,199 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.75, 95% CI 0.49 to 1.15). This finding included substantial heterogeneity, with an I2 of 90.0%.
Interventions promoting student relationships with teachers, but not student participation in school policy decisions
Within this intervention type, three RCTs67,68,115,170,171 presented findings for this outcome up to 1 year post baseline. These three trials also presented findings for this outcome > 1 year post baseline.
Up to 1 year post baseline
Findings from three RCTs67,68,115,170,171 suggested possible, but inconsistent, impacts on violence victimisation at up to 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions (Figure 22). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 10 effect sizes from three studies67,68,115,170,171 suggested a non-significant impact in reducing the odds of violence victimisation at up to 1 year post baseline (OR 0.86, 95% CI 0.52 to 1.43). This finding included substantial heterogeneity, with an I2 of 66.9%.
More than 1 year post baseline
Findings from three RCTs67,68,115,170,171 suggested possible, but inconsistent, impacts on violence victimisation > 1 year post baseline of interventions promoting student relationships with teachers, but not student participation in school policy decisions (Figure 23). This was reflected in a non-significant and moderately heterogeneous meta-analysis. A meta-analysis including 14 effect sizes from three studies67,68,115,170,171 suggested a non-significant impact in reducing the odds of violence victimisation > 1 year post baseline (OR 0.89, 95% CI 0.73 to 1.07). This finding included moderate heterogeneity, with an I2 of 35.0%.
Interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers
Within this intervention type, only one RCT185–188 presented findings for this outcome; this was at about 5.5 months post baseline. At this time point,186 students in the intervention group were less likely to report bullying victimisation than students in the control group (β = –0.76, SE 0.33, total n = 1323), but this was moderated by student engagement in the main analysis. We calculated an overall mean difference (MD) between groups in bullying victimisation of –0.05 at the sample mean for student engagement, which we estimated as unlikely to be statistically significant. Because only one RCT reported outcomes in this category, no meta-analysis was possible.
Interventions promoting parent involvement only
Within this intervention type, one RCT169 presented findings for this outcome up to 1 year post baseline. Both this trial and a second RCT197 also presented findings > 1 year post baseline.
Up to 1 year post baseline
At the end of the first school year of implementation (end of grade 4) of Friendly Schools,169 control group students (n = 863) and intervention group students (n = 984) were not significantly different in their odds of being bullied every few weeks (OR 1.16, 95% CI 0.87 to 1.54) or in their odds of being bullied at all (OR 1.49, 95% CI 1.14 to 1.94). ORs are expressed as the increase in odds of bullying in the control group; that is both estimates reflect a numerical benefit to the intervention group. Because only one RCT reported outcomes in this category, no meta-analysis was possible.
More than 1 year post baseline
Findings from two RCTs169,197 suggested that interventions promoting parent involvement only were unlikely to reduce violence victimisation > 1 year post baseline (Figure 24). This was reflected in a non-significant and substantially heterogeneous meta-analysis. A meta-analysis including 12 effect sizes from two studies169,197 suggested a non-significant impact in reducing the odds of violence perpetration > 1 year post baseline (OR 0.94, 95% CI 0.10 to 9.06). This finding included substantial heterogeneity, with an I2 of 81.5%. A sensitivity analysis suggested a similar finding (OR 0.92, 95% CI 0.71 to 1.20).
Overall meta-analyses
Overall meta-analyses suggested that interventions promoting commitment to school generate small but statistically significant impacts, and, potentially, impacts of public health significance, in reducing violence victimisation at up to 1 year post baseline (OR 0.84, 95% CI 0.72 to 0.98). This analysis drew on 22 effect sizes from eight studies67,117,169–171,181,185,200 and included a substantial amount of heterogeneity, with an I2 of 81.1% (Figure 25). Interventions promoting commitment to school also generate small but statistically significant impacts, and, potentially, impacts of public health significance, in reducing violence victimisation > 1 year post baseline (OR 0.85, 95% CI 0.73 to 0.99). This analysis drew on 71 effect sizes from 11 studies67,80,103,117,161,166,168–172,180,197,199 and included a substantial amount of heterogeneity, with an I2 of 80.8% (Figure 26).
Violence observed
One outcome evaluation, a RCT,169 presented evidence for this outcome up to 1 year post baseline. Six outcome evaluations presented evidence for this outcome > 1 year post baseline: three RCTs64,123,124,165–169 and three non-randomised evaluations. 85,162–164,173,198 Owing to the range of study designs and informants for this outcome, we did not undertake a meta-analysis. In particular, RCTs drew either on student report or on teacher report, which may be incommensurate and would not generate an interpretable pooled effect; non-randomised evidence used analytic methods not amenable to meta-analysis, and drew on official reports rather than teacher or student report.
Interventions promoting student participation in school policy decisions
Within this intervention type, two RCTs64,123,124,165–168 and three non-randomised evaluations85,162–164,173,198 presented findings for this outcome > 1 year post baseline. Collectively, both RCTs and the non-randomised evidence suggested an impact on reducing observed violence of interventions promoting student participation in school policy decisions.
Interventions promoting parent involvement only
Within this intervention type, one RCT (Friendly Schools)169 presented findings for this outcome both up to 1 year post baseline and > 1 year post baseline. At the end of the first school year from baseline, control group students (n = 863) were more likely to see someone being bullied (OR 1.36, 95% CI 1.03 to 1.81) than Friendly Schools students (n = 984). This pattern continued at the end of the second school year from baseline (OR 1.48, 95% CI 1.14 to 1.92) and at the end of the third year from baseline (OR 1.67, 95% CI 1.25 to 2.24).
Substance use
Five outcome evaluations presented findings for this outcome up to 1 year post baseline, all RCTs. 103,117,180–188 Twelve outcome evaluations presented findings for this outcome > 1 year post baseline: 10 RCTs52,61,64,67,68,87,103,108,109,115,117,123,124,165–168,175–177,179–188 and two non-randomised evaluations. 162–164,174,198 When possible, we report meta-analyses both overall and stratified by substance type: alcohol, tobacco, illicit drug use, and ‘omnibus’ substance use outcomes (e.g. frequency or prevalence of a range of substances).
Interventions promoting student participation in school policy decisions
Within this intervention type, two RCTs103,117,180,181,184 presented findings on this outcome up to 1 year post baseline. Seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,175–177,179–181,184 and two non-randomised evaluations162–164,174,198 presented findings on this outcome > 1 year post baseline.
Up to 1 year post baseline
Findings from two RCTs103,117,180,181,184 suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions in reducing substance use at up to 1 year post baseline. This was reflected in a non-significant and moderately heterogeneous meta-analysis overall, with similar findings for alcohol outcomes and tobacco outcomes separately, but meta-analyses with a small number of studies were sensitive to estimation method.
A meta-analysis including 16 effect sizes from two studies103,117,180,181,184 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.16 to 4.30). This finding included moderate heterogeneity, with an I2 of 49.8% (Figure 27). However, a sensitivity analysis suggested a similar, but statistically significant, effect (OR 0.81, 95% CI 0.68 to 0.97). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.
A separate analysis of alcohol outcomes drew on eight effect sizes from two studies103,117,180,181,184 (Figure 28). This suggested a non-significant impact in reducing the odds of alcohol use at up to 1 year post baseline (OR 0.87, 95% CI 0.29 to 2.61), with some heterogeneity (I2 = 19.8%). However, a sensitivity analysis suggested a significant effect (OR 0.89, 95% CI 0.83 to 0.95).
A separate analysis of smoking outcomes drew on six effect sizes from two studies103,117,180,181,184 (Figure 29). This suggested a non-significant impact in reducing the odds of smoking at up to 1 year post baseline (OR 0.81, 95% CI 0.09 to 7.26), with substantial heterogeneity (I2 = 68.9%). However, a sensitivity analysis suggested a significant effect (OR 0.82, 95% CI 0.67 to 0.99).
More than 1 year post baseline
Findings from seven RCTs52,61,64,87,103,108,109,117,123,124,165–168,175–177,179–181,184 and two non-randomised evaluations162–164,174,198 suggested possible, but inconsistent, impacts of interventions promoting student involvement in school policy decisions in reducing substance use > 1 year post baseline. This was reflected in a non-significant and substantially heterogeneous meta-analysis, with similar findings for alcohol outcomes and illicit drug use outcomes separately, and less evidence of impact on tobacco outcomes.
A separate analysis of omnibus substance use outcomes suggested a larger, but still substantially heterogeneous and non-significant, effect.
A meta-analysis including 55 effect sizes from seven studies52,61,64,87,103,108,109,117,123,124,165–168,175–177,179–181,184 suggested a non-significant impact in reducing the odds of substance use > 1 year post baseline (OR 0.81, 95% CI 0.57 to 1.15). This finding included substantial heterogeneity, with an I2 of 78.1% (Figure 30).
A separate analysis of alcohol outcomes drew on 22 effect sizes from six studies52,64,117,166,175,180 (Figure 31). This suggested a non-significant impact in reducing the odds of alcohol use > 1 year post baseline (OR 0.81, 95% CI 0.56 to 1.18), with substantial heterogeneity (I2 = 77.3%).
A separate analysis of smoking outcomes drew on 16 effect sizes from six studies (Figure 32). 52,64,117,166,175,180 This suggested a non-significant impact in reducing the odds of smoking > 1 year post baseline (OR 0.92, 95% CI 0.66 to 1.28), with substantial heterogeneity (I2 = 78.8%).
A separate analysis of illicit drug use outcomes drew on 10 effect sizes from five studies52,64,166,175,180 (Figure 33). This suggested a non-significant impact in reducing the odds of illicit drug use > 1 year post baseline (OR 0.74, 95% CI 0.37 to 1.46), with substantial heterogeneity (I2 = 84.3%).
A separate analysis of omnibus substance use outcomes drew on seven effect sizes from three studies52,64,87 (Figure 34). This suggested a non-significant impact in reducing the odds of alcohol use at up to 1 year post baseline (OR 0.56, 95% CI 0.24 to 1.28), with moderate heterogeneity (I2 = 48.1%).
Interventions promoting student relationships with teachers, but not student participation in school policy decisions
Within this intervention type, one RCT67,68,115 presented findings on this outcome both up to 1 year post baseline and > 1 year post baseline.
Findings for substance use in the Gatehouse Project trial were presented across three different publications drawing on two different types of analytic sample: the ‘original’ study cohort67,68 and a set of sequential cohorts of students in year 8 that did not overlap with the original study cohort. 115 Only the sequential cohorts present findings for omnibus substance use outcomes. Because only one RCT reported outcomes in this category, no meta-analysis was possible.
Up to 1 year post baseline
In the original study cohort of the Gatehouse Project,67 findings were presented at up to 1 year post baseline for the odds of any drinking, regular drinking, binge drinking, any smoking, regular smoking and any cannabis use in the preceding 6 months. At the end of the first intervention year (end of year 8), the odds were not significantly different between the intervention and control groups for any drinking (OR 1.00, 95% CI 0.78 to 1.28), regular drinking (OR 1.09, 95% CI 0.77 to 1.57) or binge drinking (OR 0.95, 95% CI 0.69 to 1.32). Intervention students were not significantly less likely to smoke at all (OR 0.89, 95% CI 0.72 to 1.12), but they were less likely than control students to be regular smokers (OR 0.66, 95% CI 0.46 to 0.95). Differences in cannabis use were not significant (OR 0.98, 95% CI 0.69 to 1.40). The exact numbers of students participating in measurement waves were not available.
More than 1 year post baseline
Substance use findings in the original study cohort of the Gatehouse Project trial were further presented at the end of the second school year from baseline and at the end of the third school year from baseline, corresponding to the end of year 10. 67 Intervention students were not significantly different from control group students in the following odds: their odds of any drinking at the end of the second school year (OR 1.00, 95% CI 0.78 to 1.28) or at the end of the third school year (OR 0.96, 95% CI 0.69 to 1.33); their odds of being a regular drinker at the end of the second school year (OR 1.05, 95% CI 0.70 to 1.57) or at the end of the third school year (OR 1.13, 95% CI 0.77 to 1.66); or their odds of binge drinking at the end of the second school year (OR 0.99, 95% CI 0.70 to 1.38) or at the end of the third school year (OR 1.02, 95% CI 0.71 to 1.46). A similar pattern of null results was found for the odds of any smoking at the end of the second school year (OR 0.92, 95% CI 0.63 to 1.33) or at the end of the third school year (OR 0.91, 95% CI 0.67 to 1.24). Intervention students were not different from controls on any cannabis use in the previous 6 months at the end of the second school year (OR 1.06, 95% CI 0.75 to 1.49) or at the end of the third school year (OR 0.81, 95% CI 0.57 to 1.16). A subsequent analysis68 probed incident and prevalent cannabis use at the last follow-up. Prevalence of any cannabis use in the previous 6 months was not significantly different between the intervention (n = 1155) and control (n = 990) groups (OR 0.80, 95% CI 0.57 to 1.13), nor was incidence of any cannabis use in the previous 6 months different between intervention (n = 1062) and control (n = 941) groups (OR 0.81, 95% CI 0.56 to 1.15). Patterns were similar when comparing intervention and control groups on prevalence of weekly cannabis use (OR 0.74, 95% CI 0.45 to 1.20) and on incidence of weekly cannabis use (OR 0.73, 95% CI 0.43 to 1.25). Finally, the sequential cohort analysis115 used an omnibus substance use outcome to compare two different cohorts of year-8 students, in 1999 and 2001, after 2 years of exposure to the intervention. Neither the 1999 cohort, which drew on 1158 intervention group students and 1428 control group students (OR 0.84, 95% CI 0.61 to 1.18), nor the 2001 cohort, which drew on 966 intervention group students and 1497 control group students (OR 0.85, 95% CI 0.65 to 1.12), demonstrated a significant impact of the Gatehouse Project intervention in reducing substance use.
Interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers
Within this intervention type, two RCTs182,183,185–188 presented findings on this outcome, both up to 1 year post baseline and > 1 year post baseline.
Up to 1 year post baseline
Findings from two RCTs182,183,185–188 suggested possible, but inconsistent, impacts on reducing substance use of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers, at up to 1 year post baseline (Figure 35). This was reflected in a non-significant and substantially heterogeneous meta-analysis, but meta-analyses with a small number of studies were sensitive to estimation method.
A meta-analysis including three effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.76, 95% CI 0.20 to 2.86). This finding included substantial heterogeneity, with an I2 of 66.2%. However, a sensitivity analysis suggested a significant impact of similar magnitude (OR 0.80, 95% CI 0.67 to 0.95). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.
A separate meta-analysis of alcohol outcomes drew on two effect sizes from two studies182,183,185–188 (Figure 36). This suggested a non-significant impact on reducing alcohol use at up to 1 year post baseline (OR 0.82, 95% CI 0.59 to 1.15), with substantial heterogeneity (I2 = 66.0%).
More than 1 year post baseline
Findings from two RCTs182,183,185–188 suggested possible, but inconsistent, impacts on reducing substance use of interventions promoting student engagement in learning, but not student participation in decision-making or relationships with teachers > 1 year post baseline (Figure 37). This was reflected in a non-significant and substantially heterogeneous meta-analysis.
A meta-analysis that included eight effect sizes from two studies182,183,185–188 suggested a non-significant impact in reducing the odds of substance use at up to 1 year post baseline (OR 0.67, 95% CI 0.04 to 10.30). This finding included substantial heterogeneity, with an I2 of 84.5%. However, a sensitivity analysis suggested a finding of similar magnitude, albeit statistically significant (OR 0.68, 95% CI 0.51 to 0.91). The sparseness of evidence thus suggests that any conclusion of effectiveness is tenuous and sensitive to estimation method.
A separate meta-analysis of alcohol outcomes drew on five effect sizes from two studies182,183,185–188 (Figure 38). This suggested a non-significant impact on reducing alcohol use > 1 year post baseline (OR 0.70, 95% CI 0.02 to 21.89), with substantial heterogeneity (I2 = 87.3%). A sensitivity analysis generated a more precise CI, but did not change the conclusion (OR 0.70, 95% CI 0.48 to 1.03).
Overall meta-analyses
Overall meta-analyses are presented first for all substances and then, when appropriate, by substance type.
Across all substance use outcomes
Overall, meta-analyses suggested that interventions promoting commitment to school generate small, but statistically significant, impacts, and, potentially, impacts significant to public health, on reducing substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.70 to 0.97). This analysis drew on 25 effect sizes from five studies67,117,181,182,186 and included a moderate amount of heterogeneity, with an I2 of 54.0% (Figure 39). Interventions promoting commitment to school also generate small, but statistically significant, impacts, and, potentially, impacts significant to public health, on reducing substance use > 1 year post baseline (OR 0.79, 95% CI 0.62 to 0.998). This analysis drew on 81 effect sizes from 10 studies64,67,68,87,108,109,115,117,166,168,175,180,182,185 and included a substantial amount of heterogeneity, with an I2 of 76.9% (Figure 40).
Alcohol outcomes
Alcohol use outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on 14 effect sizes from five studies67,117,181,182,186 (Figure 41) suggested a small, and statistically non-significant, reduction in alcohol use, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.88, 95% CI 0.74 to 1.04). This finding had a moderate amount of heterogeneity (I2 = 42.5%). An analysis drawing on 35 effect sizes from nine studies52,64,67,117,166,175,180,182,186 (Figure 42) suggested a small, and statistically non-significant, reduction in alcohol use, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.81, 95% CI 0.63 to 1.03). This finding had substantial heterogeneity (I2 = 75.6%).
Smoking outcomes
Smoking outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on eight effect sizes from four studies67,117,181,182 (Figure 43) suggested a statistically non-significant reduction in smoking, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.77, 95% CI 0.54 to 1.09). This finding had substantial heterogeneity (I2 = 70.7%). An analysis drawing on 21 effect sizes from eight studies52,64,67,117,166,168,175,180,182 (Figure 44) suggested a statistically non-significant reduction in smoking, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.87, 95% CI 0.69 to 1.10). This finding had substantial heterogeneity (I2 = 77.3%).
Illicit drug use
Illicit drug use outcomes were meta-analysed for up to 1 year post baseline and for > 1 year post baseline. An analysis drawing on three effect sizes from two67,181 studies (Figure 45) suggested a small, and statistically non-significant, reduction in illicit drug use, caused by interventions promoting commitment to school, up to 1 year post baseline (OR 0.89, 95% CI 0.16 to 5.01). This finding had a low amount of heterogeneity (I2 = 3.1%) and was imprecisely estimated. A sensitivity analysis suggested a similar conclusion as to the magnitude and significance of the effect (OR 0.85, 95% CI 0.66 to 1.08). An analysis drawing on 16 effect sizes from six studies52,64,67,68,166,175,180 (Figure 46) suggested a statistically non-significant reduction in illicit drug use, caused by interventions promoting commitment to school, > 1 year post baseline (OR 0.76, 95% CI 0.46 to 1.24). This finding had substantial heterogeneity (I2 = 80.7%).
Omnibus substance use outcomes
Omnibus substance use outcomes were meta-analysed for > 1 year post baseline only (Figure 47). An analysis drawing on nine effect sizes from four studies52,64,87,109,115 suggested a statistically non-significant reduction in general substance use > 1 year post baseline (OR 0.67, 95% CI 0.32 to 1.42). This finding had substantial heterogeneity (I2 = 63.3%).
Academic attainment
All included outcome evaluations reporting academic attainment were of the same intervention type: promoting student participation in school policy decisions. All outcomes were reported >1 year post baseline. Three RCTs52,61,64,80,108,109,123,124,172,179 and four non-randomised evaluations85,162–164,173,174,198 contributed to this synthesis. Because our analyses on this outcome were primarily hypothesis-generating, we did not undertake meta-analyses. Specifically, interventions included in this systematic review were required to include substance use and violence, but not academic attainment. Thus, it is possible, if not likely, that evaluations of interventions promoting student commitment to school with outcomes on academic attainment, but not substance use and violence, exist. As a result, our synthesis of these outcomes cannot provide a conclusive test of the hypothesis that interventions promoting student commitment to school improve academic attainment, but they can generate hypotheses in this regard.
Findings from included studies did not suggest clear evidence of impact on academic attainment. Only one of the three RCTs suggested unambiguously positive and significant impacts on test scores; moreover, one RCT suggested a mixed pattern of positive and negative effects, and two of the non-randomised evaluations suggested a clear pattern of null effects.
Grading of Recommendations Assessment, Development and Evaluation analyses
We summarised the results of the synthesis using a summary-of-findings table, focusing on overall meta-analyses (Table 6). Certainty in findings was rated as low to very low for all outcomes, owing to risk-of-bias ratings for included studies and high values of I2 in meta-analyses. The findings for observed violence outcomes were rated as being of very low quality because of the mix of informants and scales used, especially as these were not observational in nature.
Certainty assessment | Impact, OR (95% Cl) | Certainty | Importance | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Studies (n) | Study design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | |||||
Violence perpetration | |||||||||||
Up to 1 year post baseline | |||||||||||
8 | Randomised trials | Seriousa | Seriousb | Not serious | Not serious | None | 0.85 (0.76 to 0.96), drawing on 17 effect sizes | ⊕⊕⚪⚪ Low | Critical | ||
> 1 year post baseline | |||||||||||
13 | Randomised trials | Seriousa | Very seriousc | Not serious | Not serious | None | 0.75 (0.65 to 0.98), drawing on 54 effect sizes | ⊕⚪⚪⚪ Very low | Critical | ||
Violence victimisation | |||||||||||
Up to 1 year post baseline | |||||||||||
8 | Randomised trials | Seriousa | Very seriousc | Not serious | Not serious | None | 0.84 (0.72 to 0.98) | ⊕⚪⚪⚪ Very low | Critical | ||
> 1 year post baseline | |||||||||||
11 | Randomised trials | Seriousa | Very seriousc | Not serious | Not serious | None | 0.85 (0.73 to 0.99) | ⊕⚪⚪⚪ Very low | Critical | ||
Substance use | |||||||||||
Up to 1 year post baseline | |||||||||||
5 | Randomised trials | Seriousa | Seriousb | Not serious | Not serious | None | 0.83 (0.70 to 0.97) | ⊕⊕⚪⚪ Low | Critical | ||
> 1 year post baseline | |||||||||||
10 | Randomised trials | Seriousa | Very seriousc | Not serious | Not serious | None | 0.79 (0.62 to 0.998) | ⊕⚪⚪⚪ Very low | Critical | ||
Observed violence | |||||||||||
Up to 1 year post baseline | |||||||||||
1 | Randomised trials | Seriousa | Not serious | Very seriousd | Not serious | None | Evidence from one randomised trial suggests reductions in observed violence up to 1 year post baseline | ⊕⚪⚪⚪ Very low | Important | ||
> 1 year post baseline | |||||||||||
3 | Randomised trials | Seriousa | Very seriouse | Very seriouse | Not serious | None | Evidence from three randomised trials suggests reductions in observed violence > 1 year post baseline | ⊕⚪⚪⚪ Very low | Important |
Consultation with stakeholders
Stakeholders regarded the findings on intervention effects as small, but nonetheless important, expected and plausible. They viewed this category of intervention as involving a broad approach with broad benefits. It was viewed as a more holistic and pragmatic approach than delivering a different curriculum for each area of health. Several stakeholders suggested that violence might be regarded by some school staff as a key outcome that is in schools’ interest to address because it places a burden on school discipline systems and is a source of destabilisation. Participants differed as to whether or not substance use was similarly perceived as a priority within schools. Some thought this was much less of a problem for schools and was declining in prevalence, whereas other stakeholders felt that this remained an important topic to address, partly because of cuts to substance use prevention funding. Several participants also felt that this type of intervention was also likely to be effective in promoting other aspects of physical and mental health.
Several participants commented that, as well as reducing risk behaviours, these interventions were attractive to schools because they could contribute to positive development, for example through their effects on commitment to school. This outcome was regarded as important in itself and also as a means to increase engagement with learning, attendance, aspirations and attainment. It was viewed as an important issue for schools, especially after the pandemic, which has seen school engagement fall among some students. The mixed evidence of effects on educational attainment elicited various responses among stakeholders. Some thought that such interventions would positively affect attainment because of their effects on student maturity, independence and school commitment, and that the mixed effects may have reflected use of different measures. Other participants, however, felt that one would not necessarily expect educational attainment benefits based on the theory of change. Several participants thought that this type of intervention was likely to generate wider impacts not measured in the studies reviewed, but which are nonetheless important to schools. These included staff spending less time dealing with misbehaviour.
One stakeholder suggested that the review’s results would be useful to schools in resisting pressure to adopt zero-tolerance approaches to discipline and instead adopting more inclusive approaches. Other participants commented, however, that there was no contradiction between involving students in decisions and using learning approaches to discipline while, at the same time, being clear as to which behaviours would not be tolerated.
A number of organisations and sectors were suggested by stakeholders as targets for knowledge exchange. These included the Alcohol Education Trust; the Anna Freud Centre; the Association of School and College Leaders; the Chartered College of Teaching; Healthy Schools London leads in London local authorities; the Home Office Serious Violence Unit; local authority directors of education; multiacademy trust leaders; the National Association of Head Teachers; the National Society for the Prevention of Cruelty to Children; the Office for Standards in Education, Children’s Services and Skills; Place2Be; pupil referral unit leaders; the PSHE Association; secure estate educational providers; Stonewall; Terrence Higgins Trust; and YoungMinds.
The ALPHA group of young researchers was intrigued, and some members were surprised, by the findings of the impact of interventions on substance use and violence outcomes. Similar to the policy stakeholders, the ALPHA group appreciated the focus of included interventions on addressing the school environment as a broader mechanism for addressing student health. For some, health education nevertheless remained an important facet of interventions that was perceived to have a profound impact on some adolescents. Participants suggested that schools should invest in improving relationships between students, between students and teachers, and with the rest of the school community to promote student health, as well as academic outcomes. Creating a friendlier, more welcoming and caring school environment was seen to be of positive benefit to staff and students in itself and was also thought likely to be linked to positive mental health outcomes not captured by this review. The ALPHA participants also thought that schools should work on increasing and improving mechanisms for student involvement to make school provision more student-centred and relevant. However, they thought that this might run counter to current practice in some schools and suggested that, in their own experience, student involvement in schools had sometimes appeared tokenistic.
The ALPHA participants suggested that findings should be shared with the police and other law enforcement agencies, school health practitioners, head teachers and school staff, youth charities specialising in violence or substance use, education and health ministers, and parents and carers of adolescents. The ALPHA group suggested that findings would need to be carefully condensed and made accessible and appropriate for each audience.
Chapter 8 Moderation and mediation analysis
About this chapter
In this chapter, we present narrative syntheses of moderation and subgroup analyses, organised by outcome and population, and narrative synthesis of mediation by student commitment to school, organised by outcome.
Included reports
Nineteen reports were included in the synthesis of moderator analysis,61,64,68,80,85,87,103,108,117,163,166,168,171–173,176,177,181,186 covering 14 studies of 11 interventions. Three reports were included in the synthesis of mediator analysis,124,165,184 covering three studies of three interventions.
Moderation
Violence perpetration
Eight studies, all RCTs, reported moderation or subgroup analyses relating to violence perpetration. 64,87,103,108,166,168,171,177,181,186 Moderators related to age, sex, socioeconomic position and baseline risk. Evidence was inconsistent across trials for moderation on any of these factors and was sparse for all moderators except sex.
Age
Of the relevant studies, two RCTs report moderation or subgroup analyses by age. 171,177 Evidence for moderation by age on violence perpetration was sparse and inconsistent.
The RCT of the AAYP school/community intervention177 used a longitudinal model to examine moderation of intervention impact on violence perpetration by age over 4 school years. Although the main effect of the school and community intervention in the AAYP was a significantly reduced rate of increase in violence overall, compared with the control intervention (β = –0.155, SE 0.051; p = 0.002), the interaction term for intervention and age (β = –0.150, SE 0.056; p = 0.007) suggested an even stronger effect among older students. In practical terms, children 1 standard deviation below the mean (i.e. aged 9.79 years) did not experience effects in reducing the rate of violence increase over time (β = 0.045, SE 0.105; p = 0.66), but children 1 standard deviation above the mean (i.e. aged 12.47 years) did experience effects (β = –0.356, SE 0.074; p < 0.001). The RCT of FSFF171 stratified analysis into children in grade 4 and children in grade 6 and compared them on bullying perpetration outcomes at the end of the first and second school years from baseline. These effects are presented in depth in Appendix 7. Analysis included comparisons of both medium-intensity and low-intensity interventions against high-intensity interventions on outcomes of bullied others versus not and bullied others frequently versus not. No formal interaction tests were presented; however, a pattern of non-significant effects and inconsistent patterning of differential effects between grade-4 and grade-6 students across outcomes did not suggest any evidence of moderation by age in this trial.
Sex
Of the relevant studies, six RCTs report moderation or subgroup analyses by sex. 64,87,103,108,166,168,181 Evidence for moderation by sex of violence perpetration was inconsistent, with some studies87,103,108,166,168 suggesting patterns of differential benefits favouring boys and others108,181 suggesting patterns of differential benefits favouring girls (Figure 48). Two studies87,103 did not report formal interaction tests, which made drawing conclusions on this body of evidence more challenging.
In the AAYP school/community intervention RCT,87 estimates for growth in violence over 4 school years were presented stratified by sex. A formal interaction test was not presented; however, boys in the school and community intervention group experienced decreased growth in violent behaviours, compared with the control group, with a relative reduction by the end of grade 8 of 47% (p = 0.02). However, girls in the intervention group experienced a non-significant increase in violent behaviours, compared with girls in the control group.
Similarly, results for DARE Plus103 were stratified by sex, but without a formal interaction test. At the end of the second school year from baseline, boys receiving DARE Plus103 had a significant decrease in physical violence, compared with boys in the control group (M = 3.58 vs. M = 4.23; p = 0.03), whereas girls had a non-significant increase (M = 2.05 vs. M = 2.14; p = 0.64). Boys had a marginally significant decrease in verbal violence (M = 6.44 vs. M = 7.12; p = 0.06), whereas girls had a non-significant decrease (M = 5.77 vs. M = 5.92; p = 0.65), and neither boys (M = 1.70 vs. M = 1.97; p = 0.24) nor girls (M = 0.43 vs. M = 0.34; p = 0.56) were significantly different on weapon-carrying. This suggests some, albeit tenuous, evidence that DARE Plus produces favourable effects for boys.
Learning Together168 compared intervention effects at 36 months among boys and girls on an overall score of aggression, as measured by the Edinburgh Study of Youth Transitions and Crime (ESYTC) scale, and the modified aggression subscale score. There was some suggestion of a greater effect among boys than girls on the ESYTC scale score [boys: β = –0.33 (95% CI –0.73 to 0.06); girls: β = 0.04 (95% CI –0.32 to 0.39)], with marginal significance for the interaction term (p = 0.0890). However, there was significant evidence of interaction (p = 0.0029) on the modified aggression subscale score [boys: β = –0.53 (95% CI –0.89 to –0.18); girls: β = –0.03 (95% CI –0.37 to 0.31)]. A subsequent analysis168 examined moderation in perpetration of cyberbullying and wider aggression at both 24 and 36 months post baseline. There was no evidence of moderation on wider aggression at either time point or on cyberbullying perpetration at 36 months; however, at 24 months, the differential impact of the intervention on cyberbullying perpetration was significant (p = 0.002) for boys (OR 0.61, 95% CI 0.41 to 0.89), compared with girls (OR 1.19, 95% CI 0.85 to 1.67).
The Chicago trial of Positive Action108 considered moderation by sex for both violence-related behaviours and bullying behaviours using a longitudinal model over 6 school years. No significant moderation was reported for violence-related behaviours; however, the intervention had a larger impact by the study’s end in reducing violence among girls (d = –0.51) than among boys (d = –0.23), as suggested by significant three-way interactions between intervention, time and sex.
The Hawaii trial of Positive Action64 presented violence perpetration outcomes for boys and girls both as individual behaviours and as a summary score; however, only the summary score included a formal test of interaction. Individual violent behaviours did not show a clear pattern of greater effect for boys or girls. Although carrying a knife (boys: OR 0.27; girls: OR 0.51) and threatening to cut or stab someone (boys: OR 0.30; girls: OR 0.48) both reflected numerically greater intervention benefits for boys, the opposite was the case for cutting or stabbing someone (boys: OR 0.29; girls: OR 0.25), carrying a gun (boys: OR 0.33; girls: OR 0.58) or shooting at someone (boys: OR 0.14; girls: OR 0.22). The overall interaction test on the sum score of violent behaviours suggested a greater, but non-significant, intervention impact in reducing violent behaviours among boys than girls [interaction incidence rate ratio (IRR) 0.67, 90% CI 0.35 to 1.28; p = 0.158].
Finally, in the trial of SEHER,181 violence perpetration was examined at 8 months post baseline. In the lay counsellor SEHER ‘Mitra’ (LSM) (leader) group, compared with the control group, girls had a greater reduction in perpetration than boys [girls: OR 0.59 (95% CI 0.33 to 1.04); boys: OR 0.81 (95% CI 0.53 to 1.21)]. This yielded a significant interaction effect (p = 0.006). In contrast, there was no significant difference between boys and girls (p = 0.444) in the teacher SEHER Mitra (TSM) group, compared with the control group [girls: OR 1.61 (95% CI 0.91 to 2.85); boys: OR 1.51 (95% CI 1.00 to 2.29)].
Socioeconomic position
Only one trial (Learning Together)166,168 considered moderation by socioeconomic position for violence perpetration. The main analysis did not find evidence of moderation by socioeconomic position at 36 months post baseline, with intervention effects favouring students with low socioeconomic position on the ESYTC scale [low: MD –0.32 (95% CI –0.78 to 0.14); high: MD 0.12 (95% CI –0.24 to 0.49)], but without significant interaction (p = 0.0750), and intervention effects favouring students with high socioeconomic position on the modified aggression subscale [low: MD –0.04 (95% CI –0.43 to 0.36); high: MD –0.27 (95% CI –0.61 to 0.07)], but again without significant interaction (p = 0.2176). Family affluence was not a moderator of cyberbullying perpetration or wider forms of aggression at 24 or 36 months post baseline. 168
Baseline risk
Two trials, Cooperative Learning186 and Learning Together,166 considered moderation by baseline risk for violence perpetration. In Cooperative Learning,186 baseline risk was operationalised as student engagement with school. At about 5.5 months post baseline, a significant interaction between intervention condition and student engagement was found for bullying perpetration (β = 0.09, SE 0.04; p < 0.05), suggesting that the intervention was more effective at reducing violence perpetration among those with low baseline levels of school engagement.
In Learning Together,166 baseline risk was operationalised as experience of bullying victimisation, measured by the Gatehouse Bullying Scale (GBS), and aggression, measured by the ESYTC scale. Although there was a numerical trend towards greater intervention effect among students with high levels of baseline bullying victimisation, this was not significant at 36 months (p = 0.4422) for the overall ESYTC scale score [low levels of victimisation: MD 0.02 (95% CI –0.34 to 0.37); high levels of victimisation: MD –0.17 (95% CI –0.63 to 0.29)], nor was it significant (p = 0.8100) for the modified aggression subscale [low levels of victimisation: MD –0.18 (95% CI –0.52 to 0.17); high levels of victimisation: MD –0.22 (95% CI –0.62 to 0.18)]. However, baseline aggression significantly moderated intervention effects on the ESYTC scale at 36 months (p < 0.0001), with evidence of an intervention-generated increase in aggression among students with low baseline aggression (MD 0.71, 95% CI 0.33 to 1.10), and an intervention-generated decrease in aggression among students with high baseline aggression (MD –0.65, 95% CI –1.03 to –0.27). This moderation was in the same direction for the modified aggression subscale score, albeit not statistically significant (p = 0.0933), with no evidence of impact on students with low baseline aggression (MD 0.03, 95% CI –0.33 to 0.39), and numerical, but not statistical, impact on students with high baseline aggression (MD –0.25, 95% CI –0.60 to 0.10).
Violence victimisation
Seven studies reported moderation or subgroup analyses relating to violence victimisation. Six of these were RCTs. 80,103,117,166,168,171,172,181,186 One non-randomised evaluation, the CDP,163 reported a relevant moderator analysis. Moderators related to age, sex, socioeconomic position and baseline risk. Evidence was sparse and inconsistent across trials for moderation related to any of these factors.
Age
Two studies reported subgroup analyses of violence victimisation outcomes by age: one RCT171 and one non-RCT. 163 Although neither study presented a formal interaction test, there was little evidence of moderation by age.
The RCT of FSFF171 stratified analysis into children in grade 4 and children in grade 6 and compared them on bullying victimisation outcomes at the end of the first and second school years from baseline. These effects are presented in depth in the narrative synthesis in Appendix 7. Analysis included comparisons of both medium-intensity and low-intensity interventions against high-intensity intervention on outcomes of bullied others versus not, and bullied others frequently versus not, and demonstrated relatively few significant effects either by intervention or within year group. No formal interaction tests were presented; however, inconsistent patterning of differential effects between grade-4 and grade-6 students across outcomes did not suggest any evidence of moderation by age in this trial. An analysis of the effectiveness of the CDP163 on a measure of violence victimisation stratified children by grades 6, 7 and 8. No formal interaction test was presented, but the magnitude of differences was relatively similar between children in grades 6 (intervention M = 1.88, control M = 2.00), 7 (intervention M = 1.88, control M = 1.89) and 8 (intervention M = 1.78, control M = 1.184), with an overall effect that was marginally significant (p < 0.10). This suggests little evidence of moderation by age.
Sex
Four RCTs presented moderation or subgroup analyses by sex for violence victimisation outcomes. 80,103,117,166,168,172,181 Evidence was highly inconsistent across studies. Although the DARE Plus trial and peer violence outcomes from the GST trial suggested greater intervention impacts for boys, findings from the SEHER RCT indicated a pattern of greater impacts for girls. Findings from the Learning Together trial did not provide evidence of moderation favouring either boys or girls (see Figure 48).
In the main report of the DARE Plus RCT,117 physical victimisation was analysed via longitudinal model and stratified by sex, although no formal interaction test was presented. Among boys, DARE Plus generated a significant (one-tailed p = 0.02) decrease in the rate of growth of physical victimisation (β = –0.10 vs. β = 0.03), whereas rates of growth among girls were virtually identical between conditions (both β = 0.00, one-tailed p = 0.45). In a subsequent analysis,103 differences in overall victimisation at the end of the second school year from baseline were not significant between the DARE Plus and control (p = 0.11) groups for boys (M = 7.99 vs. M = 8.62), nor were they significant for girls (M = 5.16 vs. M = 5.33; p = 0.62).
In the main report of the GST RCT,172 the primary outcome was physical violence from school staff in the previous week, as reported by students. The intervention had a significantly greater impact (p = 0.043) among boys (OR 0.34, 95% CI 0.21 to 0.56) than girls (OR 0.46, 95% CI 0.29 to 0.74). A subsequent report80 considered violence outcomes separately and included peer violence outcomes. Summary analyses of violence, including emotional, physical and sexual violence, suggested that the intervention had a greater effect among boys than among girls for violence from staff or peers in the previous week [boys: OR 0.34 (95% CI 0.22 to 0.53) vs. girls: OR 0.55 (95% CI 0.36 to 0.84); p < 0.0001] or the previous term [boys: OR 0.20 (95% CI 0.11 to 0.37) vs. girls: OR 0.44 (95% CI 0.24 to 0.82); p < 0.0001]. Similar patterns were found for any staff violence in the previous week [boys: OR 0.34 (95% CI 0.21 to 0.54) vs. girls: OR 0.49 (95% CI 0.31 to 0.77); p = 0.009] or previous term [boys: OR 0.21 (95% CI 0.12 to 0.37) vs. girls: OR 0.45 (95% CI 0.26 to 0.80); p < 0.0001]. Differences between boys and girls for emotional violence from staff were in similar directions, but not significant for either the previous week [boys: OR 0.68 (95% CI 0.40 to 1.16) vs. girls: OR 0.85 (95% CI 0.52 to 1.40); p = 0.354] or the previous term [boys: OR 0.60 (95% CI 0.40 to 9.92) vs. girls: OR 0.76 (95% CI 0.50 to 1.14); p = 0.218]. As in the main report, which included physical violence in the previous week, differences in staff violence in the previous term were significantly different (p < 0.0001) for boys (OR 0.20, 95% CI 0.11 to 0.36), compared with girls (OR 0.44, 95% CI 0.25 to 0.78). However, restricting to severe physical violence did not yield a similar or significant pattern of results, for either the previous week [boys: OR 0.77 (95% CI 0.19 to 3.13) vs. OR 0.31 (95% CI 0.11 to 0.91); p = 0.227] or the previous term [boys: OR 0.38 (95% CI 0.16 to 0.86) vs. OR 0.70 (95% CI 0.34 to 1.45); p = 0.174]. The impacts on sexual violence from school staff towards students in the previous term were not different (p = 0.648) between boys (OR 0.85, 95% CI 0.27 to 0.70) and girls (OR 1.20, 95% CI 0.46 to 3.10). Differences in intervention effect on peer violence were not significant for any form of violence, for either the previous week [boys: OR 0.62 (95% CI 0.43 to 0.89) vs. girls: OR 0.77 (95% CI 0.54 to 1.09); p = 0.192] or the previous term [boys: OR 0.61 (95% CI 0.43 to 0.88) vs. girls: OR 0.74 (95% CI 0.52 to 1.05); p = 0.184]. When stratified by form of violence, effects were very similar for emotional violence, in both the previous week [boys: OR 0.59 (95% CI 0.42 to 0.83) vs. girls: OR 0.77 (95% CI 0.55 to 1.08); p = 0.731] and the previous term [boys: OR 0.67 (95% CI 0.48 to 0.95) vs. girls: OR 0.68 (95% CI 0.48 to 0.95); p = 0.964]. There was similarly little evidence of difference in physical violence in the previous week [boys: OR 0.82 (95% CI 0.51 to 1.30) vs. girls: OR 0.72 (95% CI 0.48 to 1.12)] or in the previous term [boys: OR 0.63 (95% CI 0.44 to 0.90) vs. girls: OR 0.88 (95% CI 0.62 to 1.26); p = 0.064]. However, although relying on few cases, there was some marginally significant signal of differential impact on sexual violence in the previous week [boys: OR 0.38 (95% CI 0.07 to 1.96) vs. girls; OR 2.33 (95% CI 0.73 to 7.51); p = 0.061] and in the previous term [boys: OR 0.64 (95% CI 0.15 to 2.72) vs. girls: OR 3.39 (95% CI 1.22 to 9.39); p = 0.056].
In the main report of Learning Together,166 GBS scores at 36 months did not suggest a difference (p = 0.6113) in intervention effect between boys (MD –0.04, 95% CI –0.08 to 0.001) and girls (MD –0.03, 95% CI –0.06 to 0.01). Subsequent analyses168 did not find that sex moderated cyberbullying victimisation at 24 or 36 months.
Finally, in the trial of SEHER,181 violence victimisation, as measured by frequency of bullying, any victimisation and experience of forced sex, was examined at 8 months post baseline. In the LSM group, both boys (d = –0.39, 95% CI –0.53 to –0.25) and girls (d = –0.51, 95% CI –0.76 to –0.26) experienced a lower frequency of bullying, but the numerically greater impact for girls was not statistically significant (p = 0.111). A similar pattern was evident for any victimisation [boys: OR 0.65 (95% CI 0.46 to 0.90); girls: OR 0.57 (95% CI 0.31 to 1.02)], but this difference was statistically significant (p = 0.018). Finally, although girls (OR 1.07, 95% CI 0.68 to 1.68) and boys (OR 1.27, 95% CI 0.85 to 1.90) had numerically different intervention impacts on the odds of forced sex, this difference was, again, not significant (p = 0.594). In the TSM group, neither boys (d = 0.01, 95% CI –0.13 to 0.16) nor girls (d = –0.04, 95% CI –0.29 to 0.21) experienced significant intervention impacts on the frequency of bullying; an interaction test did not suggest evidence of effect modification (p = 0.388). However, although both boys (OR 1.16, 95% CI 0.81 to 1.62) and girls (OR 1.75, 95% CI 0.97 to 3.14) experienced numerical, but not statistically significant, increases in any victimisation as a result of the intervention, there was evidence of a differential intervention effect favouring boys (p = 0.0001). As in the LSM group, neither boys (OR 0.89, 95% CI 0.57 to 1.38) nor girls (OR 1.06, 95% CI 0.65 to 1.74) experienced any intervention impacts on forced sex; a test of interaction did not find evidence of a differential intervention effect (p = 0.517).
Socioeconomic position
Only one trial, Learning Together,166,168 considered moderation by socioeconomic position for violence perpetration. In the main trial report,166 GBS scores at 36 months did not suggest a difference (p = 0.8944) in intervention effect between students with low levels of family affluence (MD –0.02, 95% CI –0.07 to 0.03) and students with high levels of family affluence (MD–0.03, 95% CI –0.06 to 0.01). Subsequent analyses168 did not find that family affluence moderated cyberbullying victimisation at 24 or 36 months.
Baseline risk
Three RCTs presented moderation or subgroup analyses by baseline risk for violence victimisation outcomes. 166,172,186
In Cooperative Learning,186 baseline risk was operationalised as student engagement with school. At about 5.5 months post baseline, a significant interaction between intervention condition and student engagement was found for bullying victimisation (β = 0.21, SE 0.09; p < 0.05), suggesting that the intervention was more effective at reducing victimisation among those with low baseline levels of school engagement.
In the main report of the GST trial,172 schools with a high prevalence of physical violence at baseline (OR 0.39, 95% CI 0.21 to 0.75) were not different from schools with a low prevalence of physical violence at baseline (OR 0.41, 95% CI 0.22 to 0.78) on the primary outcome of physical violence from school staff in the previous week (p = 0.9192).
In the main report of Learning Together,166 baseline risk was operationalised as experience of bullying victimisation, measured by the GBS, and aggression, measured by ESYTC scale; violence victimisation was also measured by the GBS. Students with high levels of baseline victimisation benefited from the intervention (MD –0.41, 95% CI –0.45 to –0.36) more than students with low levels of baseline victimisation (MD 0.15, 95% CI 0.12 to 0.18), for whom there was some evidence of an intervention-generated increase in victimisation. This difference was significant (p < 0.0001). Similarly, students with high levels of baseline aggression experienced greater declines in bullying victimisation as a result of the intervention (MD –0.06, 95% CI –0.10 to –0.02) than students with low levels of baseline aggression (MD 0.01, 95% CI –0.03 to 0.05), with a significant interaction effect (p = 0.0024).
Violence observed
Four studies85,163,168,173 presented moderation or subgroup analyses relating to observed violence outcomes. One of these168 was a RCT. Three85,163,173 were non-randomised evaluations. Evidence was sparse across all moderators, including age, sex, ethnicity and socioeconomic deprivation, precluding any firm conclusions.
Age
Only the report of the CDP163 reported subgroup analyses of intervention impacts on observed violence by sex. Drawing on teachers’ ratings of whether or not students insult others, get others into trouble, start fights or destroy others’ property, the analysis stratified children by grades 6, 7 and 8. No formal interaction tests were presented, and overall intervention impacts were not significant; grade-level means suggested benefits for students in grade 6 (M = 1.88 vs. M = 2.12 for intervention vs. control students, respectively), but harms for students in grade 7 (M = 1.93 vs. M = 1.68 for intervention vs. control students, respectively) and grade 8 (M = 1.94 vs. M = 1.90 for intervention vs. control students, respectively).
Sex
Only the Learning Together trial168 presented moderation of observed violence outcomes by sex. In this study, sex moderated observed aggression at 24, but not at 36, months post baseline. Girls (MD –0.15, 95% CI –0.26 to –0.04), but not boys (MD 0.01, 95% CI –0.11 to 0.13), reported lower levels of observed aggression; this interaction was significant (p = 0.02) (see Figure 48).
Ethnicity
Two studies presented moderation of observed violence outcomes by ethnicity. 85,173 In Positive Action Nevada,85 ethnicity (defined as percentage of African American students) did not moderate number of violent incidents, either in total per school or per 1000 students or when broken down to student-to-student violence, student-to-staff violence or possession of weapons; exact estimates of moderation or tests of moderation were not presented. In Positive Action Florida,173 schools with higher proportions of African American students experienced greater intervention effects in reductions of violent incidents per 100 students; in schools below the median for percentage of African American students, Positive Action was associated with a mean reduction in violent incidents of 0.9, whereas, in schools above the median, the mean reduction was 3.8. Although the interaction term was significant, an exact test was not presented.
Socioeconomic position
Two studies presented moderation of observed violence outcomes by socioeconomic position. 85,168 In the Learning Together RCT,168 family affluence moderated observed aggression at 24, but not at 36, months post baseline. At 24 months post baseline, intervention impacts on observed aggression exhibited a U-shaped relationship with family affluence; at both low levels (MD –0.18, 95% CI –0.62 to 0.25) and high levels (MD –0.14, 95% CI –0.25 to –0.03) of family affluence, the intervention reduced students’ reports of observed aggression, but this was not the case for students with a middle level of affluence (MD 0.04, 95% CI –0.09 to 0.18). The interaction test was significant (p = 0.03). In Positive Action Nevada,85 socioeconomic position, defined as school-level percentage of students with free/reduced-price lunch, did not moderate observed violence outcomes, although exact estimates of moderation or tests of moderation were not presented.
Substance use
Eight studies presented moderation or subgroup analyses for substance use outcomes. These included seven RCTs. 64,68,87,117,166,168,176,181 One non-randomised study163 presented subgroup analyses for substance use outcomes. Moderators related to age, sex, socioeconomic position and baseline risk; evidence for all moderators was sparse and there was no clear evidence of differential effects for any moderator.
Age
Only one study163 presented subgroup estimates by age for substance use outcomes. An analysis of tobacco use, alcohol use, marijuana use or other illicit drug use in the previous 30 days was stratified by grade. No formal interaction tests were presented, and all intervention impacts were non-significant; scrutiny of grade-level estimates for each outcome revealed nearly identical prevalence between intervention and control group students. In both intervention and control groups, 6% of grade-6 students and 10% of grade-7 students reported tobacco use, whereas 12% of grade-8 intervention students and 13% of grade-8 control students reported tobacco use. Six per cent of grade-6 intervention students and 8% of grade-6 control students reported alcohol use, with similar absolute differences for grade-7 students (10% vs. 12% for the intervention and control groups, respectively) and grade-8 students (17% vs. 19% for the intervention and control groups, respectively). Marijuana use findings were inconsistent in direction between grade 6 (2% vs. 3% for the intervention and control groups, respectively), grade 7 (5% vs. 7% for the intervention and control groups, respectively) and grade 8 (9% vs. 8% for the intervention and control groups, respectively), but revealed minimal differences. Differences in illicit drug use were also minuscule for grade 6 (8% vs. 9% for the intervention and control groups, respectively), grade 7 (6% for both groups) and grade 8 (5% vs. 6% for the intervention and control groups, respectively).
Sex
Six RCTs reported moderation or subgroup analyses by sex for substance use outcomes. 64,87,117,166,168,176,181 Three studies87,117,166,168 demonstrated a pattern of intervention impacts favouring boys, whereas one study181 demonstrated a pattern of intervention impacts favouring girls. Two studies64,176 did not provide evidence of an intervention effect gradient by sex (see Figure 48).
In the RCT of the AAYP school/community intervention,87 estimates for growth in substance use over 4 school years were presented stratified by sex. A formal interaction test was not presented; however, boys in the school and community intervention group experienced decreased growth in substance use, compared with the control group, with a relative reduction by the end of grade 8 of 35% (p = 0.05). Girls in the intervention group experienced a non-significant decreased growth in substance use, compared with girls in the control group (48% vs. 52%, respectively; p = 0.86).
In the main report of DARE Plus,117 substance use outcomes [specifically, alcohol use in the previous year and previous month, ever drunk (i.e. intoxicated) and current smoker] were analysed via longitudinal models and stratified by sex, although no formal interaction tests were presented. Across outcomes, differences between groups were similar in direction and magnitude between boys and girls, although only effects for boys reached statistical significance. For alcohol use in the previous year, boys in DARE Plus had a marginally significant (one-tailed p = 0.04) decrease in rate of growth, compared with boys in the control group (β = 0.19 vs. β = 0.26, respectively). An intervention effect for girls was not reported on this outcome (β = 0.23 vs. β = 0.25 for the intervention and control groups, respectively; one-tailed p = 0.36). For alcohol use in the previous month, boys’ rate of growth was significantly lower (one-tailed p = 0.01) in the DARE Plus group (β = 0.08) than in the control group (β = 0.14); for girls, the difference was of similar magnitude and direction (β = 0.08 vs. β = 0.12 for the intervention and control groups, respectively), but not significant (one-tailed p = 0.15). The growth rate for the outcome of ever drunk was not different for boys across both groups (β = 0.11 vs. β = 0.15 for the intervention and control groups, respectively; one-tailed p = 0.07), nor for girls (β = 0.07 vs. β = 0.12 for the intervention and control groups, respectively; one-tailed p = 0.11). Finally, boys in the DARE Plus group experienced a decrease in growth of current smoking, compared with boys in the control group (β = 0.18 vs. β = 0.31 for the intervention and control groups, respectively; one-tailed p = 0.02); girls did not demonstrate an intervention effect on this outcome (β = 0.22 vs. β = 0.28 for the intervention and control groups, respectively; one-tailed p = 0.25).
Although the main report of Learning Together166 tested a range of substance use outcomes at 36 months post baseline, moderation analyses by sex were presented for outcomes of ever smoked, ever drank alcohol and illicit drug use (a three-category variable comprising ‘no’, ‘yes, offered but did not accept’ and ‘yes, offered and accepted’). There was clear evidence of moderation by sex for smoking and alcohol use, with greater intervention effects for boys. For the outcome of ever smoked, boys in the intervention group reported lower odds of smoking than boys in the control group (OR 0.33, 95% CI 0.22 to 0.50), whereas differences were not significant for girls (OR 0.87, 95% CI 0.60 to 1.25), with a significant interaction test (p < 0.0001). A similar pattern was in evidence for alcohol use [boys: OR 0.52 (95% CI 0.38 to 0.70); girls: OR 0.95 (95% CI 0.71 to 1.26)], again with a significant interaction test (p = 0.0002). Estimates of intervention impact were closer between boys and girls for the illicit drug use outcome [boys: OR 0.44 (95% CI 0.29 to 0.68); girls: OR 0.57 (95% CI 0.38 to 0.86)]. Although the intervention was statistically effective for both groups on this outcome, there was no evidence of interaction (p = 0.2350). A subsequent analysis168 examined e-cigarette use and found that Learning Together was more effective for boys than for girls at both 24 months [boys: OR 0.45 (95% CI 0.30 to 0.67) vs. girls: OR 0.80 (95% CI 0.83 to 1.55); p = 0.014] and 36 months [boys: OR 0.35 (95% CI 0.23 to 0.53) vs. girls: OR 0.94 (95% CI 0.63 to 1.40); p < 0.001].
The Hawaii trial of Positive Action64 presented substance use outcomes for boys and girls both as individual behaviours and as a summary score; however, only the summary score included a formal test of interaction. Individual substance use behaviours did not show a clear pattern of greater effect for boys or girls; smoking a cigarette (boys: OR 0.66; girls: OR 0.38) and using an illegal drug (boys: OR 0.34; girls: OR 0.15) both reflected greater intervention benefits for girls, but the opposite was the case for getting drunk on alcohol (boys: OR 0.24; girls: OR 0.40) and getting high on drugs (boys: OR 0.18; girls: OR 0.25), with nearly identical effects for drinking any alcohol (boys: OR 0.48; girls: OR 0.47). The overall interaction test on the sum score of substance use behaviours suggested a non-significant differential intervention impact in reducing substance use among boys, compared with girls (interaction IRR 1.07, 95% CI 0.65 to 1.80; p = 0.402). Teacher reports of smoking (boys: OR 0.42; girls: OR 0.78), drinking (boys: OR 0.66; girls: OR 1.16) or illicit drug use (boys: OR 0.21; girls: OR 0.42) for individual students did, however, suggest a pattern of greater intervention effects for boys, albeit with no formal interaction tests.
In the SEHER trial,181 substance use outcomes were evaluated at 8 months post baseline, and included smoking tobacco, chewing tobacco, drinking alcohol and other substance use. In the LSM arm, compared with the control arm, effects were consistently greater for girls than for boys. Girls demonstrated a greater intervention-generated decrease in smoking tobacco (OR 0.56, 95% CI 0.25 to 1.23) than boys (OR 1.16, 95% CI 0.69 to 1.92), with a significant interaction test (p < 0.0001). Effects were numerically similar for chewing tobacco [girls: OR 0.54 (95% CI 0.29 to 0.98); boys: OR 1.12 (95% CI 0.65 to 1.91)], with a significant interaction test (p < 0.0001); for drinking alcohol [girls: OR 0.53 (95% CI 0.27 to 1.03) vs. boys: OR 1.05 (95% CI 0.62 to 1.78); p = 0.020]; and for other substance use [girls: OR 0.64 (95% CI 0.26 to 1.57) vs. boys: OR 0.99 (95% CI 0.72 to 1.34); p = 0.002]. Differences in intervention impact between boys and girls were all non-significant when comparing the TSM group with the control group. For alcohol and tobacco outcomes, effects were numerically greater for girls than for boys, but tended to be closer in magnitude than in the LSM arm: smoking tobacco – [girls: OR 0.45 (95% CI 0.19 to 1.05) vs. boys: OR 0.74 (95% CI 0.51 to 1.07) (p = 0.362)], chewing tobacco – [girls: OR 0.49 (95% CI 0.26 to 0.93) vs. boys: OR 0.60 (95% CI 0.33 to 1.07) (p = 0.341)] and drinking alcohol – [girls: OR 0.45 (95% CI 0.21 to 0.93) vs. boys: OR 0.80 (95% CI 0.45 to 1.43) (p = 0.157)]. However, in the TSM arm, intervention impacts for other substance use were greater for boys (OR 0.74, 95% CI 0.51 to 1.07) than for girls (OR 0.91, 95% CI 0.35 to 2.31), albeit with a non-significant interaction test (p = 0.501).
Finally, in the whole-of-school intervention,176 moderation was tested by gender for a range of substance use outcomes measured at 35 months post baseline. There was no evidence of moderation (p > 0.05 in every case) by gender for any substance use outcomes, including recent tobacco use [boys: OR 1.37 (95% CI 0.77 to 2.42); girls: OR 1.38 (95% CI 0.78 to 2.43)], amount of tobacco use [boys: MD –4.56 (95% CI –19.35 to 10.23); girls: MD –0.33 (95% CI –13.69 to 13.03)], recent alcohol use [boys: OR 1.09 (95% CI 0.72 to 1.66); girls: OR 1.12 (95% CI 0.72 to 1.73)], recent risky alcohol use [boys: OR 0.80 (95% CI 0.55 to 1.16); girls: OR 1.21 (95% CI 0.82 to 1.79)], amount of alcohol consumed [boys: MD –1.40 (95% CI –4.29 to 1.49); girls: MD 0 (95% CI –3.01 to 3.00)], any marijuana use [boys: OR 1.04 (95% CI 0.65 to 1.67); girls: OR 1.23 (95% CI 0.74 to 2.04)] or any other substance use [boys: OR 0.96 (95% CI 0.51 to 1.82); girls: OR 1.77 (95% CI 0.80 to 3.92)].
Socioeconomic position
Two studies166,168,176 presented moderation analyses by socioeconomic position. Neither analysis suggested a pattern of differential effects.
There was little evidence of moderation by family affluence in Learning Together,166 with no significant evidence of interaction for outcomes of ever smoked, ever drank alcohol and illicit drug use. For the outcome of ever smoked, there was a numerically greater intervention benefit for students of lower socioeconomic status at baseline (OR 0.51, 95% CI 0.32 to 0.81) than for students of high economic status at baseline (OR 0.66, 95% CI 0.46 to 0.95), but no significant interaction effect (p = 0.3003). For the outcome of ever drank alcohol, the numerical pattern was similar [low socioeconomic status: OR 0.59 (95% CI 0.41 to 0.85); high socioeconomic status: OR 0.81 (95% CI 0.61 to 1.08)], with a marginally significant interaction test (p = 0.0869). Intervention effect estimates were nearly identical for illicit drug use [low socioeconomic status: OR 0.50 (95% CI 0.31 to 0.81); high socioeconomic status: OR 0.54 (95% CI 0.36 to 0.80)], with no interaction effect (p = 0.7599). A subsequent analysis focusing on e-cigarette use168 did not find evidence of moderation by family affluence at 24 or 36 months post baseline. In the whole-of-school intervention,176 moderation was tested by family affluence for a range of substance use outcomes measured at 35 months post baseline. Only intervention impacts on amount of tobacco was moderated by family affluence (p < 0.01), with marginally significant intervention impacts on students with a low socioeconomic position (MD –12.89, 95% CI –26.00 to 0.23), but a non-significant intervention-generated increase among students of high socioeconomic position (MD 16.36, 95% CI –1.03 to 33.76). There was no evidence of moderation (p > 0.05 in every case) by family affluence for any other substance use outcomes, including recent tobacco use [low socioeconomic status: OR 1.40 (95% CI 0.79 to 2.48); high socioeconomic status: OR 1.35 (95% CI 0.65 to 2.81), recent alcohol use (low socioeconomic status: OR 1.06 (95% CI 0.70 to 1.61); high socioeconomic status: OR 1.21 (95% CI 0.71 to 2.07)], recent risky alcohol use [low socioeconomic status: OR 1.06 (95% CI 0.72 to 1.55); high socioeconomic status: OR 0.84 (95% CI 0.52 to 1.35)], amount of alcohol consumed [low socioeconomic status: MD –0.91 (95% CI –3.65 to 1.84); high socioeconomic status: MD –0.59 (95% CI –4.20 to 3.02)], any marijuana use [low socioeconomic status: OR 1.30 (95% CI 0.79 to 2.15); high socioeconomic status: OR 0.85 (95% CI 0.46 to 1.56)] or any other substance use [low socioeconomic status: OR 0.47 (95% CI 0.72 to 3.01); high socioeconomic status: OR 0.82 (95% CI 0.34 to 1.94)].
Baseline risk
Randomised controlled trials of the Gatehouse Project, Learning Together and whole-of-school interventions68,166,176 present moderation analyses for substance use by baseline risk. The baseline risk variables considered included school engagement, baseline substance use,176 and bullying victimisation and aggression. 166
In the Gatehouse Project,68 incidence and prevalence of cannabis use were compared over strata defined by school engagement and baseline tobacco use. By year 10 (after 3 school years of the intervention), there was no evidence of an interaction effect between intervention group and baseline tobacco smoking for either prevalence (p = 0.22) or incidence (p = 0.25) of any cannabis use. Weekly cannabis use was also considered as an outcome. Although there was no evidence of moderation by baseline tobacco smoking for prevalence of weekly cannabis use (p = 0.10), baseline tobacco smoking did moderate intervention effects on incident weekly cannabis use (p = 0.04), with evidence of a significant intervention effect for baseline non-smokers (OR 0.50, 95% CI 0.26 to 0.98). School engagement did not yield a significant moderation test for intervention effects for prevalence of any cannabis use (p = 0.84), incidence of any cannabis use (p = 0.55), prevalence of weekly cannabis use (p = 0.97) or incidence of weekly cannabis use (p = 0.84).
In Learning Together,166 moderation of intervention effects for outcomes of ever smoked, ever drank alcohol and illicit drug use at 36 months post baseline was tested using baseline risk stratifiers of bullying victimisation and aggression. Baseline levels of bullying victimisation (i.e. low or high) did not moderate intervention effects for smoking [low: OR 0.66 (95% CI 0.46 to 0.96) vs. high: OR 0.52 (95% CI 0.33 to 0.80); p = 0.2887]; alcohol consumption, although an interaction test was marginally significant [low: OR 0.81 (95% CI 0.61 to 1.07) vs. high: OR 0.57 (95% CI 0.41 to 0.81); p = 0.0541]; or illicit drug use [low: OR 0.46 (95% CI 0.31 to 0.68) vs. high: OR 0.58 (95% CI 0.37 to 0.93); p = 0.2895]. However, baseline levels of aggression (i.e. low or high) did moderate intervention impacts on smoking [low: OR 0.93 (95% CI 0.58 to 1.49) vs. high: OR 0.46 (95% CI 0.32 to 0.68); p = 0.0053] and alcohol use [low: OR 1.08 (95% CI 0.78 to 1.49) vs. high: OR 0.56 (95% CI 0.41 to 0.74); p = 0.0009], with greater intervention effects for students with a high level of baseline aggression. However, baseline levels of aggression did not moderate intervention impacts on illicit drug use [low: OR 0.62 (95% CI 0.40 to 0.96) vs. high: OR 0.50 (95% CI 0.34 to 0.74); p = 0.3137].
Finally, in the whole-of-school intervention,176 moderation of each substance use outcome was tested by whether or not students were baseline users of the relevant substance. There was no evidence of moderation (p > 0.05 in every case) for any substance use outcomes by baseline use of that substance, including recent tobacco use [non-user: OR 1.39 (95% CI 0.83 to 2.31); user: OR 1.34 (95% CI 0.60 to 2.97)], amount of tobacco use [non-user: MD –5.16 (95% CI –16.55 to 6.23); user: MD 6.83 (95% CI –11.76 to 25.43)], recent alcohol use [non-user: OR 1.22 (95% CI 0.76 to 1.95); user: OR 0.94 (95% CI 0.63 to 1.42)], recent risky alcohol use [non-user: OR 0.99 (95% CI 0.69 to 1.41); user: OR 0.79 (95% CI 0.52 to 1.21)], amount of alcohol consumed [non-user: MD –1.05 (95% CI –3.76 to 1.67); user: MD –0.95 (95% CI –4.10 to 2.20)] or any marijuana use [non-user: OR 1.37 (95% CI 0.13 to 14.23); user: OR 1.09 (95% CI 0.69 to 1.72)]. No moderation analysis was possible for the outcome of any other substance use.
Academic attainment
Four studies, the Chicago RCT of Positive Action,61 and the non-randomised evaluations of the CDP163 and of Positive Action in Nevada, Hawaii85 and Florida,173 presented moderation or subgroup analyses for academic attainment outcomes. Moderators included age, sex, ethnicity and socioeconomic position. Evidence was sparse, but generally suggested no gradient of effectiveness for ethnicity or socioeconomic position.
Age
Only one study, the CDP,163 presented subgroup estimates by age for academic attainment outcomes. The analysis of grade-point average and of achievement test score was stratified by grade. No formal interaction tests were presented, and all intervention impacts were non-significant. There was no clear pattern of magnitude or direction of effect by grade for either outcome. For example, for grade-point average, grade-6 students in the intervention group had a higher score than those in the control group (M = 2.52 vs. M = 2.39, respectively), but this effect was reversed for students in grade 7 (M = 2.34 vs. M = 2.39 for the intervention and control groups, respectively), and then reversed again for students in grade 8 (M = 2.50 vs. M = 2.43 for the intervention and control groups, respectively), with no effect reaching statistical significance.
Sex
Only the Chicago trial of Positive Action61 presented moderation of academic attainment outcomes by sex. Findings for moderation by sex of intervention impacts on student self-reported grades were not presented, but probably did not suggest differential impacts, given null effects overall. Impacts on teacher-rated academic performance were greater for boys than for girls, although the amount of this was not quantified. End-point effect sizes comparing intervention impacts for boys and girls on standardised test performance suggested a greater increase for boys (d = 0.33) than for girls (d = 0.11) on reading, but a greater impact for girls (d = 0.41) than for boys (d = 0.31) on mathematics. However, these differences were unlikely to be significant (see Figure 48).
Ethnicity
Moderation of academic attainment outcomes by ethnicity was tested in the non-randomised evaluation of Positive Action in Nevada and Hawaii. 85 Subgroup analyses were presented in the RCT of Positive Action in Chicago. 61 Findings were inconsistent, but generally appeared to be null.
In the study of Positive Action Nevada,85 ethnicity (defined as percentage of African American students) did not appear to moderate academic achievement on standardised tests, either in terms of mathematics; reading; language; science; or combined mathematics, reading and language scores. Similarly, in the Nevada and Hawaii analysis,85 ethnicity (defined as percentage of Japanese or Chinese students) did not appear to moderate academic achievement in standardised tests of mathematics or reading, or a combined measure of both. In the Chicago trial of Positive Action,61 end-point effect sizes comparing intervention impacts for subgroups were presented for standardised tests of reading and mathematics. African American students had larger effect sizes than the general population for both reading (d = 0.50 vs. d = 0.22, respectively) and mathematics (d = 0.55 vs. d = 0.38, respectively), but moderation tests, which were not formally presented, were unlikely to be significant, given a pattern of null effects for longitudinal models.
Socioeconomic position
Moderation of academic attainment outcomes by socioeconomic position was tested in the non-randomised evaluations of Positive Action in Nevada and Hawaii85 and in Florida,173 and subgroup analyses were presented in the Chicago trial of Positive Action. 61 Although reporting was poor, no studies suggested effectiveness gradients by socioeconomic position.
Although reporting was unclear in the non-randomised evaluations of Positive Action in Nevada and Hawaii,85 it appeared that socioeconomic position (defined as school-level percentage of students receiving free or reduced-price lunch) did not moderate intervention effects on academic attainment as measured by standardised test performance. In the matched controls analysis of schools in the Florida evaluation of Positive Action,173 the percentage of students receiving free/reduced-price lunch did not moderate intervention impacts on standardised tests of academic achievement, although the exact outcome used was unclear. In the Chicago trial of Positive Action,61 end-point effect sizes comparing intervention impacts for subgroups were presented for standardised tests of reading and mathematics. Students receiving free or reduced-price lunch were not different from the general population in terms of effect sizes for reading (d = 0.23 vs. d = 0.22, respectively) or mathematics (d = 0.42 vs. d = 0.38, respectively).
Mediation by student commitment to school
Three studies presented mediation analyses of intervention effects by measures of school-belonging and/or commitment. 124,165,184 All interventions were classed as promoting student participation in school policy decisions. Findings relating to intervention impacts on mediators are presented first, followed by results of mediation models for each outcome.
Mediators
In the Learning Together RCT,165 analyses tested whether or not student view of school climate, measured at 24 months using the Beyond Blue School Climate Questionnaire (including, as relevant to this analysis, student sense of belonging and student commitment to academic values), mediated various outcomes at 36 months. Evidence of intervention impacts on mediators was tenuous, with some evidence of an impact on student sense of belonging (MD 0.04, 95% CI 0.01 to 0.07), but not on student commitment to academic values (MD 0.00, 95% CI –0.02 to 0.03). In the SEHER trial,184 school climate, measured at 8 months post baseline using the same questionnaire as in Learning Together, was tested for mediation of outcomes at 17 months post baseline. Sense of belonging (MD 1.573; p < 0.001) and commitment to school (MD 0.720; p < 0.001) were both significantly linked to intervention status, as was an overall measure of school climate (MD 6.639; p < 0.001). Finally, Positive Action Hawaii124 included a test of whether or not school commitment, operationalised as ‘academic behaviours’ (e.g. ‘work hard in school’, ‘set goals’, ‘try to be your best’) and rated by both student and teachers, mediated outcomes. Analyses were undertaken in grade 5 using mediator and outcome data from the same time point. Both student (MD 0.273, SE 0.039) and teacher (MD 0.125, SE 0.045) ratings of academic behaviours were significantly greater in the Positive Action arm than in the control arm.
Violence perpetration
All three trials presented mediation analyses for violence perpetration. Findings were inconsistent by measure of school commitment. Findings were inconsistent across studies.
In Learning Together,165 analyses tested whether or not student view of school climate, measured at 24 months using the Beyond Blue School Climate Questionnaire, mediated impacts on aggression perpetration (as measured by ESYTC scale) and bullying perpetration (modified aggression subscale). Although student view of school climate as a whole was linked to both aggression perpetration (adjusted MD –2.57, 95% CI –2.96 to –2.18) and bullying perpetration (adjusted MD –1.53, 95% CI –1.74 to –1.33), including student view of school climate in regression models for either aggression perpetration or bullying perpetration did not appear to attenuate intervention impacts, suggesting that student view of school climate did not mediate violence perpetration outcomes. In the SEHER trial,184 school climate, measured at 8 months post baseline, was tested for mediation of violence perpetration outcomes at 17 months post baseline. Neither sense of belonging nor commitment to school was linked to the outcome. However, student report of school climate as a whole significantly mediated reductions in violence perpetration (indirect effect β = –0.081, 95% CI –0.100 to –0.034), reflecting 15.43% of the total intervention effect on this outcome. Finally, Positive Action Hawaii124 included a test of whether or not school commitment mediated violence perpetration, measured as a count of violence-related behaviours. A full mediation model demonstrated that student reports of academic behaviours accounted for a 42.3% reduction in violent behaviours (p < 0.01), with teacher-rated academic behaviours accounting for a 16.0% reduction in violent behaviours (p < 0.05).
Violence victimisation
Two trials, Learning Together165 and SEHER,184 presented mediation analyses for violence victimisation. In Learning Together,165 analyses of mediation on bullying victimisation at 36 months post baseline did not demonstrate attenuation of intervention impacts when student view of school climate was included in regression models, suggesting that student view of school climate did not mediate violence victimisation outcomes. In the SEHER trial,184 sense of belonging (indirect effect β = –0.033, 95% CI –0.057 to –0.007), but not commitment to school, mediated intervention impacts on bullying victimisation. An overall measure of school climate had a significant indirect effect on victimisation, accounting for 14.87% of intervention impacts on victimisation.
Substance use
Two trials, Learning Together165 and Positive Action Hawaii,124 presented mediation analyses for substance use. In Learning Together,165 analyses of mediation on substance use outcomes (ever smoked, ever drank alcohol, ever been really drunk, ever been offered illicit drugs) at 36 months post baseline did not demonstrate attenuation of intervention impacts when student view of school climate was included in regression models, suggesting that student view of school climate did not mediate substance use outcomes. In Positive Action Hawaii,124 student-rated academic behaviours completely mediated the impact of Positive Action on count of substance use behaviours, accounting for a 44.6% reduction in substance use and generating a statistically non-significant direct effect of the intervention. Academic behaviours rated by teachers generated only partial mediation, but still accounted for a 14.9% reduction in substance use behaviours.
Chapter 9 Results: synthesis of economic evaluations
About this chapter
In this chapter, we present the results of the review and synthesis of economic evidence.
Included studies
Three reports relating to two economic evaluations were eligible for inclusion in the review. Both focused on interventions aiming to involve students in school decision-making, but differed considerably in terms of context and the outcome measures that they reported. The first, by Greco et al.,89 evaluated the cost-effectiveness of the GST to reduce episodes of violence by staff in Ugandan primary schools. The second, reported initially as a cost–consequences analysis167 and later as a CUA,196 evaluated the cost-effectiveness of Learning Together, an intervention designed to reduce bullying and aggression in English secondary schools.
Quality of included studies
The quality of each study was assessed on 10 main items, containing between two and six sub-items: 31 questions in total (see Appendix 8, Table 14). The level of reviewer agreement was high for both studies, at 87% on the sub-items for both studies. Overall, reviewer agreement for the 10 main items was 100%. Both studies were judged to be of high quality. The only main item judged to have not met the required reporting level was that assessing adjustment for differential timing in the analysis of costs and consequences, as reported by Greco et al. ,89 for which the analysis time horizon was 18 months. Although this technically indicates that discounting should have been undertaken, it is noted that this would have had a negligible impact on the results.
Summaries of the included studies
Good School Toolkit
This evaluation assessed the cost-effectiveness of the GST in 42 primary schools in the Luwero District of Uganda, between June 2012 and July 2014. 89 It was performed within a cluster RCT and did not include any decision modelling. The intervention’s objective was to reduce physical violence against children by staff. This is described in more detail in Chapter 3.
The evaluation was implicitly a cost-effectiveness analysis as the outcome measure was the number of cases of physical violence by a school staff member in the previous week. The time horizon for the analysis was 18 months and no discounting was performed. The analysis was performed from a provider perspective, in this instance the non-governmental organisation Raising Voices. Thus, it did not include other potential (averted) costs to the health or education sectors. The development costs of the GST and the start-up and implementation costs were included, collected retrospectively. Costs were initially recorded in Ugandan shillings, but were converted to 2015 US dollars using an average exchange rate from the Bank of Uganda, and inflated using the International Monetary Fund’s consumer price index. Little is specified with regard to activities in the control arm, although it is referred to as a do-nothing alternative.
The analysis used generalised linear models to estimate risk differences and 95% CIs using an intention-to-treat approach, after accounting for school-level clustering. It is unclear how missing data were handled and a probabilistic analysis was not performed.
The base-case results showed that the total cost of the GST was US$397,233, with an estimated reduction of 1620 cases of physical violence by staff. This cost has been equated to being the incremental cost of the GST (i.e. the cost of the control arm was assumed to be zero), producing an incremental cost-effectiveness ratio (ICER) of US$245 per case of physical violence averted. Only a limited number of one-way sensitivity analyses were reported, in which the development costs of the GST and its resale value were varied by ± 50%, and using the upper and lower CIs for the number of cases averted, with the results ranging between US$162 and US$548 per case of physical violence averted.
Learning Together
The second study is a cost–consequences analysis and CUA of Learning Together, which aimed to reduce school-based bullying and aggression. 167,196 The study used data collected as part of a cluster RCT in 40 state secondary schools in south-east England in 2014–7. The RCT enrolled school children at the end of year 7 (i.e. children aged 11–12 years). The time horizons for the economic evaluation were 2 and 3 years, the latter being the trial duration; no decision modelling was undertaken to extrapolate the results. Outcomes in the CUA were expressed as quality-adjusted life-years (QALYs), via use of the Child Health Utility-9 Dimensions (CHU-9D) index; note that the QALYs did not include potential benefits to staff, but only to students. Costing was performed from a public sector perspective, and included costs to the education, health and police sectors. The intervention costs included trainers’ and facilitators’ time, and time spent by school staff managing cases of aggression/bullying and formulating action plans. Costs (in Great British pounds) and QALYs were both discounted at 3.5% per annum, although the price year is not stated. The analysis was based on an intention-to-treat approach. Unadjusted and adjusted analyses were presented, with the latter adjusting for differences in the following variables: baseline measures of outcome; sex; ethnicity; level of family affluence; and school-level stratifying factors, such as level of deprivation, measured by the proportion of students eligible for free school meals. The data were analysed using multilevel linear regression models accounting for clustering at the school level, and assuming that costs and QALYs were missing at random. A probabilistic analysis was reported.
Results from the cost–consequences analysis showed that the primary outcome favoured the intervention arm (adjusted MD in GBS score –0.03, 95% CI –0.06 to 0.00). Although the general direction of an effect favoured the intervention on the ESYTC scale, the CHU-9D index and the staff Short Form questionnaire-12 items physical health score, differences were not statistically significant. The intervention arm cost approximately £62 (95% CI –£165 to £288) more per student than the control arm at year 3. In the CUA, at 2 years, the mean costs per student were £493 and £650 for the control and intervention groups, respectively, and the mean QALYs per student were 1.6833 and 1.6834 for the control and intervention groups, respectively. The equivalent values at year 3 were £667 and £719, and 2.4858 and 2.4937, for the control and intervention groups, respectively. This produced mean ICERs at year 2 in the unadjusted and adjusted analyses, respectively, of £10,214 and £13,284, indicating a 65% probability that the intervention was the most cost-effective option when a £20,000 per additional QALY critical threshold value was assumed. The ICERs decreased to £1905 and £1875 for the unadjusted and adjusted analyses, respectively, when a 3-year time horizon was used, with the probability of cost-effectiveness increasing to 90%. One-way sensitivity analyses showed that the results were robust to changes in the costing perspective and assumptions regarding the cost of the intervention. The characteristics and findings of included economic studies are summarised in Table 7.
Item | Description | |
---|---|---|
Programme | Learning Together (Bonell et al.167/Legood et al.196) | The GST (Greco et al.89) |
RQ | To test the hypothesis that the Learning Together intervention is cost-effective in managing bullying/aggression in English secondary schools (years 8–10), compared with standard school-based approaches | To assess the costs and cost-effectiveness of the GST: an intervention for reducing violence in primary schools in Uganda |
Intervention |
|
|
Comparator(s) and whether or not this represents standard practice in the UK | Schools (in the south-east England state education system) randomised to the control group continued with their normal practice and received no additional input | The study involved a two-arm cluster RCT. Little is said in the economic report or main effectiveness evaluation as to what constituted the control, although it is noted to be a ‘do-nothing’ alternative (note that this is unlikely to represent standard UK practice). Control schools also received the first visit |
Base-case population characteristics and analysed subgroups | 6667 students in 40 different schools were randomised (by school). Pupils enrolled were boys and girls aged 11–12 years in secondary schools in the state education system (excluding schools in special measures). Staff outcomes were also measured. Baseline data for the primary and secondary outcomes appear balanced (e.g. Gatehouse Bullying Scale, the ESYTC scale, the SDQ, SWEMWBS and overall PedsQL™) and in terms of other school characteristics (e.g. Special Educational Needs, English as a foreign language, student absence and Office for Standards in Education, Children’s Services and Skills rating) | They are not in the economic report, although the trial included only Ugandan primary schools in the Luwero District. In the main RCT report, it is stated that, at baseline, levels of each primary and secondary outcome were similar across groups, and a detailed table is provided that supports this conclusion. For example, sex, level of disability (none, some) and age did not significantly differ |
Form of economic evaluation | CCA and CUA | Cost-effectiveness analysis |
If CUA, were QALYs reported? | Yes. The CHU-9D measure was used to assess health-related quality of life as part of the economic evaluation. Utility values were calculated using data collected from the CHU-9D questionnaire at baseline and at 24 and 36 months | No CUA was reported |
Primary outcome measure(s) for the economic evaluation |
|
Cost-effectiveness was estimated as the cost per self-reported case of physical violence averted in the previous week in primary schools (the main outcome of the trial) |
Methods used to value health states and other benefits | Utilities from CHU-9D were calculated using preference weights elicited from adults in the UK | N/A, as outcomes such as QALYs and DALYs were not reported |
Methods and sources of information used to estimate resource use | The cost analysis was performed from a public sector perspective (health, education and police costs). Resource use was collected alongside the main RCT, and through invoices from the trainers and facilitators (which included preparation and organisation time) |
|
Did the study include start-up provider costs? | Training (facilitator and staff) costs were included; they are an integral part of the intervention | The costs included in the evaluation were said to be from the perspective of the provider: the NGO Raising Voices |
Did the study include ongoing provider costs? | Yes, including staff time spent dealing with bullying and aggression, further training and curriculum changes | Yes, these are referred to as ‘implementation costs’ (and included in-school capacity development, technical support for schools, school-led activities and monitoring and evaluation) |
Did the study include provider costs per contact? | Yes, at the school level for the intervention: an additional £47–58 for training, facilitators and school staff over the 3 years. The unit cost of an inpatient stay was £298. The unit costs of police contact were £267 and £457 for those not arrested and those arrested, respectively | Yes. Unit costs estimated included cost per primary school pupil in intervention school |
Did the study include costs to patients? | No | No |
Currency and price year | GBP; price year not stated | Costs in Ugandan shillings were converted to 2015 US dollars using the IMF consumer price index |
Details of model used and key structural issues and assumptions | No decision modelling was performed | No decision modelling was performed |
Justification for model used | N/A | N/A |
Base-case time horizon | 24 and 36 months | 18 months |
Base-case discount rates for costs and benefits | 3.5% for costs and outcomes | The costs of the GST were annualised using a discount rate of 9.26%. However, there is no mention of discounting future costs and benefits. Note that it is probably fine not to do so given the relatively short time horizon |
Statistical test(s) and CI(s) for stochastic data | Multilevel modelling allowing for clustering at the school level. Unadjusted and adjusted (baseline outcome measures, sex, ethnicity, SES and school stratifying factors) | 95% CIs are reported for estimates of risk difference using generalised linear models, after accounting for school-level clustering using an intention-to-treat approach |
Sensitivity analyses | Exclusion of NHS and police costs was tested in a sensitivity analysis. Further analysis excluded staff training costs in terms of time spent (as they might not be deemed ‘additional’). The discount rates were also varied | Univariate sensitivity analysis was undertaken on the design and development costs of the intervention, the ‘resale’ value of the GST and the number of cases of violence in the preceding week estimated from the generalised linear model |
Base-case ICER | The intervention was associated with higher unadjusted and adjusted costs, although the differences were not statistically significant. The difference in QALYs was also not statistically significant. The adjusted ICERs were £13,284 (95% CI –£32,175 to £58,743) and £1875 (95% CI –£12,945 to £16,695) per QALY gained at 2 and 3 years, respectively, also corresponding to a 65% and a 90% probability of being cost-effective at a critical threshold value of £20,000 per QALY gained | US$245 per episode of violence averted in the previous week, decreasing to US$97 when only the annual implementation costs were included (which is taken to mean running costs if implemented elsewhere) |
ICERs for specified subgroups | NR | NR |
Author conclusions |
|
The authors stated that there was evidence that the GST reduced episodes of violence and that the costs of achieving this compare favourably with other violence prevention programmes. However, they also state that further research is needed on the effectiveness and cost-effectiveness of other violence prevention programmes (to put the results of this economic evaluation into context), and that broader outcome measures are needed |
Critical appraisal of the included studies
A strength of both studies is that they were embedded in high-quality cluster RCTs and the comparator arms appear to represent ‘standard’ school-level antibullying practices in their respective contexts. The results from the study by Greco et al. 89 are difficult to interpret, as it would require knowledge of the critical value to avoid a case of physical violence by staff; the authors do not explicitly state whether or not they believe that the intervention is cost-effective, although they acknowledge that their outcome measure is limited in terms of potentially capturing the full effects of the intervention.
Both studies analysed data using appropriate statistical techniques. 89,167,196 Greco et al. 89 did not specify how missing data were handled in their study, and the implication of assuming that data were missing at random in the studies by Bonell et al. 167 and Legood et al. 196 is unclear. In Legood et al.,196 future costs and QALYs were discounted using nationally recommended rates. Greco et al. 89 did not apply any discounting, although this is likely to have had minimal impact on the results because of the relatively short time horizon. Neither study attempted to extrapolate the trial results beyond the observed period, but both authors concluded that consideration of a longer time horizon was likely to have increased cost-effectiveness if sustained intervention effects were assumed.
The evaluation of Learning Together is more useful from a UK decision-making perspective, although it is not clear that the critical QALY threshold204 recommended by the National Institute for Health and Care Excellence (NICE) (in this case, £20,000 per additional QALY) should be applied to interventions that are likely to be funded through the educational budget, as opposed to the NHS budget, as the authors did in this study. However, expressing outcomes in terms of a generic measure of health outcomes, such as QALYs, and using the NICE threshold value, is still a sensible approach. Legood et al. 196 also reported probabilistic results and considered a wider cost perspective and a broader range of outcomes than Greco et al. ,89 although it is noted that separate RCT reports for both studies are available.
The differences in the costs and QALYs between the control and intervention arms in the two reports from the economic evaluation of Learning Together167,196 were not statistically significant at the 5% level. For example, the incremental QALYs in all scenarios was, at most, 0.024 (95% CI –0.113 to 0.0325). Although this finding is noteworthy, it is acknowledged that economic evaluations, and their interpretation, are generally agnostic as to the size of the absolute differences in costs and benefits. The upper and lower CIs for the incremental costs and number of cases of physical violence averted reported by Greco et al. 89 did not include zero. No subgroup analyses or value-of-information results were reported in either economic evaluation.
The two economic evaluations of Learning Together167,196 both note that a limitation with their analyses was that some of the cost reductions in the treatment arm might not have been captured in the analyses. Indeed, a later analysis showed that the intervention was associated with lower disciplinary and truancy rates. 168 This would serve only to increase the cost-effectiveness of the Learning Together intervention.
The ICERs in the CUA were generally robust to alternative assumptions regarding the cost of the intervention and the cost perspective. This said, the results were sensitive to the time horizon, with the 2- and 3-year results indicating probabilities that the intervention is cost-effective of 65% and 90%, respectively. However, the latter result is arguably the most appropriate to consider as it based on the entire trial follow-up period. Greco et al. 89 reported a more limited set of one-way sensitivity analyses, although the implication of different assumptions in terms of decision-making is difficult to ascertain without knowing the critical value of averting a case of physical violence.
Chapter 10 Discussion
About this chapter
In this chapter, we reflect on the results from the review and syntheses presented in the preceding chapters to answer the review questions, assess the limitations in the evidence and consider implications for future research and policy.
Summary of key findings
Intervention typology
The first RQ asked what whole-school interventions that promote student commitment to school to prevent student substance use and violence have been evaluated and what intervention subtypes are apparent.
We identified four discrete intervention subtypes, two of which were further subdivided, based on the approaches they took.
-
The first subtype focused on developing student participation in school policy, rule-making or other school-wide decisions.
-
This was further subdivided into those interventions in which students sat on a school decision-making group with staff, and sometimes parents and members of the wider community, and those that used other mechanisms to involve students in school decision-making.
-
-
A second set of interventions did not involve student participation in decision-making, but did focus on improving staff–student relationships.
-
This was subdivided into interventions aiming to improve student/staff relationships via introducing restorative practices or other non-punitive disciplinary methods and those that took other approaches to improving staff–student relationships.
-
-
The third distinct category of interventions we identified did not involve student participation in decisions or aim to improve staff–student relationships, but instead focused on increasing student engagement in learning.
-
We identified a fourth intervention subtype that did not address any of the above areas but that did focus on increasing parental involvement in school.
Synthesis of theories of change
The first RQ also asked how closely do intervention theories of change align with the theory of human functioning and school organisation. The majority of descriptions of theories of change were judged to be of low to medium quality. We undertook a thematic content analysis using a priori codes derived from the theory of human functioning and school organisation, and refined and augmented these with themes arising inductively from the included theories of change. We organised the synthesis by intervention categorised.
We synthesised together interventions developing student participation (except Positive Action because this had a dissimilar theory of change to other interventions in this category), those enhancing staff–student relationships and those involving parents. This was because these intervention types had commonalities in their theories of change. Despite only one intervention in these categories being explicitly informed by the theory of human functioning and school organisation,46,71,88,165–168,194,195 the theories of change of these interventions aligned closely with the theory of human functioning and school organisation, with recurrent themes that reciprocally translated with each other and with the theory of human functioning and school organisation.
Several theories could be interpreted as involving the reframing of school organisational management to involve students, parents and/or other members of the community in school-level decision-making. This was theorised in ways that could be interpreted, using the starting codes from the theory of human functioning and school organisation, as making schools more student-centred and eroding boundaries among and between staff, students, parents and community members.
Several theories considered modifications to schools’ instructional orders (i.e. teaching and learning), differing as to whether or not this was limited to promoting student engagement by improving teaching methods and study skills, or more radically reframing teaching to give students a more active role in their learning. Considering the former, changes to teaching management were theorised in ways that could be interpreted as eroding boundaries between students and teachers, and among students. Interventions could increase teacher collaboration and accountability via systems such as team teaching and increased peer support. Interventions that introduced culturally specific learning were theorised in ways that could be interpreted as enabling the recentring of teaching on student needs and eroding boundaries between schools and local communities. Interventions that taught students new study skills were theorised to develop meta-learning skills, interpretable as eroding boundaries between different academic subjects. Interventions could increase the career focus in learning to reframe this on students’ perceived future needs. Considering more radical changes to teaching, some interventions were theorised to reorient learning so that new knowledge was evaluated in the light of students’ existing knowledge, which we interpreted as eroding boundaries between academic education and students’ broader development, and between school and diverse local community cultures. Interventions involving co-operative learning or increasing peer or parent roles in learning were theorised in ways that we interpreted as eroding boundaries between the roles of teachers, students and parents. All these approaches to reframing the instructional order were theorised to engender what we interpreted as school commitment, as well as practical reasoning and affiliation, among students.
We also applied codes from the theory of human functioning and school organisation to understand the theories of change of interventions addressing how schools managed discipline, corresponding to the concept of the regulatory order in our starting theory. Interventions differed as to whether they merely aimed to modify how staff managed discipline (e.g. enhancing classroom management, staff visibility and consistency of enforcement) or instead gave students a role in revising discipline systems (e.g. by rewriting school rules). Theorisations reciprocally translated with our a priori concepts of eroding boundaries and reframing provision: the former group aiming to recentre the regulatory order on the needs of students, and the latter group transforming the role of students from the subjects of the regulatory order to become its co-creators (thus eroding the boundary between students’ and teachers’ roles). Some interventions were underpinned by theories suggesting an even more fundamental reframing that, in effect, eroded the boundary between schools’ regulatory and instructional orders. Such interventions aimed to move from punitive discipline towards more learning-based discipline via teaching social and emotional skills or using restorative practice. As with the instructional order, some interventions involved parents in discipline, eroding boundaries between teacher and parent roles.
Several theories focused on building relationships between schools and local communities via voluntary work, either of students in communities or of community members in schools, all of which were interpretable as eroding boundaries between schools and their local communities.
A key recurring theme across theories of change was that the abovementioned processes were theorised not merely to minimise risk behaviours, but to contribute towards students’ positive overall development. Interventions generally sought to achieve this via engendering student commitment to school by ensuring that school provision was reframed to meet students’ needs and by eroding boundaries between and among staff and students. Across theories of change, attributes aligning with practical reasoning and affiliation were theorised to enable students to choose healthier, over riskier, behaviours. Some theories of change also suggested that increases in student commitment to school might affect reducing risk behaviours via increasing commitment to pro-school peer groups and decreasing commitment to antischool groups and antisocial risk behaviours. Thus, as a result of synthesising themes from the included theories of change, we were able to refine our understanding of the theory of human functioning and school organisation.
The refined theory of change for the review is presented as a logic model (see Figure 1). The elaborated theory of change provides a list of specific intervention activities that can modify school classification and training. In terms of school management and organisation, these include the following: use data on student needs and preferences to inform policy, as well as involving students, parents or other community members directly in decisions. In terms of teaching and learning, these include the following: increasing teacher collaboration and accountability, teaching students meta-learning skills, increasing the career focus within teaching, building academic learning on existing knowledge informed by local cultures, instituting co-operative learning methods, using peer tutoring and involving parents in learning. In terms of discipline, these include the following: enhancing classroom management, staff visibility and consistency of enforcement of rules; student participation in writing rules; and instituting learning-based discipline, for example via restorative practice or social and emotional skills education. In terms of community engagement, these include student volunteering in community or community volunteering in school. The elaborated theory of change also refines the concept of reframing so that this includes the process of building learning on existing student knowledge through ‘constructivist’ approaches to education.
The elaborated theory of change also proposes additional ways in which such reclassification might erode boundaries in the school. Additional boundaries that might be eroded are those between staff (via more collaboration and accountability among teachers); between learning and broader development (e.g. via the teaching of social and emotional skills in lessons); between the roles of teachers, students and parents (e.g. via peer tutoring, and involving parents in learning and discipline); and between the instructional and regulatory orders (e.g. via learning-based discipline methods, such as restorative practice).
The amended theory of change also recognises that whole-school interventions might prevent violence and substance use among students in other, more direct, ways. These include by providing health education that teaches students the skills needed to avoid risk behaviours, and by identifying and curtailing risk behaviours via school discipline systems that increase conformity to the regulatory order. This can engender positive student development and provide the knowledge, attitudes and norms supportive of avoiding risk behaviours.
In terms of commitment, the elaborated theory of change proposes that the abovementioned actions enable students to commit, not to separate instructional and regulatory orders, but to a joint instructional and regulatory order, which can actually have been co-created, via the abovementioned activities, by staff and students. Whereas the starting theory suggested that such mechanisms might particularly benefit socioeconomically disadvantaged students, our elaborated theory broadened this to include students from marginalised or diverse cultural groups, as well as delinquent or academically disengaged students. Our refined theory of change also suggests that activities such as restorative practice can actively reintegrate students with instructional/regulatory orders when they have previously been excluded from them. Commitment is theorised to reduce involvement in violence and substance use both via students developing the practical reasoning and positive affiliations to avoid this and by discouraging students from engaging in peer groups in which risk behaviours are normative. Included theories of change suggested little about how mechanisms might play out differently by context other than the theory of change, directly informed by the theory of human functioning and school organisation, proposing that mechanisms might be more successful in generating benefits for the most socioeconomically disadvantaged students and in schools with more capacity to deliver interventions.
The theories of change for interventions developing student engagement in learning/social skills aligned much less obviously with the theory of human functioning and school organisation. These interventions were theorised to simultaneously increase students’ school engagement and reduce risk behaviours through activities primarily focused on social skills curricula. Interventions did include whole-school elements, but these were secondary to the curriculum. Although interventions in this category did aim to modify teaching to increase engagement in learning, the primary mechanism of reducing risk behaviours was a direct one, whereby students learning social skills would reduce their involvement in bullying and/or substance use.
Synthesis of process evaluations
The second RQ asked what factors relating to setting, population and intervention influence the implementation of such interventions. Studies ranged widely in quality from those judged to be of both high reliability and high usefulness in addressing the RQs69,71,88,167,194,195 to those judged as being of low quality and low usefulness. 113,190,191,193
We used May’s155 general theory of implementation as a framework to inform our analysis because this theory aligned with emergent themes in our analysis. The theory postulates that interventions come to be enacted via providers going through processes of sense-making (understanding the intervention), cognitive participation (committing to be involved in delivery), collective action (collaborating with others to identify each individual’s role in implementation) and reflexive monitoring (formally and informally assessing the success of implementation and determining further actions). This provided a useful heuristic through which to interpret our findings.
Evaluations suggested that school staff were more likely to understand what was required in implementing an intervention when provided with good-quality materials and ongoing support. Studies reported that school staff could sometimes wilfully or unintentionally interpret interventions in unintended ways: wilfully when this fitted with school priorities (e.g. using the intervention to promote academic achievement rather than preventing bullying), unintentionally when some staff lacked the expertise and guidance to understand the intervention.
Evaluations suggested that various factors influenced how much school staff were prepared to commit to enacting intervention activities. Staff assessed interventions in terms of their ease of integration with existing practices. Interventions that could be locally tailored or had the potential to build on existing work were reported as more likely to secure staff commitment. The lack of local adaptability of an intervention was particularly undermining for whole-school elements, such as proposed changes to school policies or discipline systems. Providing local data as an integral element of an intervention was reported as helping to build buy-in. Staff were also more likely to commit to interventions when external providers appeared credible and had direct experience of schools. Students were reportedly more likely to commit to participate in an intervention if this offered opportunities for active roles or expressing their views. This required that interventions enabled genuine participation supported by staff facilitation. When this was not the case, this could result in student disengagement.
Whether or not interventions succeeded in engendering staff commitment was also reported to be affected by the school context. School leaders were more likely to commit to an intervention that addressed an issue they were already interested in tackling, for example providing a way to respond to a new government policy or inspection requirements. Schools were more likely to commit when there was already a recognition of the need for change, for example because of poor exam results or a new head wanting to transform a school. Similarly, teachers with a prior interest in the topic addressed by an intervention, such as social and character education or restorative practice, were more likely to commit to implementing such interventions.
Evaluations also examined how much those in schools worked together to divide up responsibilities for implementation and deliver interventions with fidelity. To facilitate this, interventions needed to be locally workable, fitting with school timetables and providing staff and students with clear guidance. Interventions that included planning groups (consisting of staff, and possibly also students, parents or other community members) were reported as particularly successful in ensuring collective action to enact interventions. Diverse participation in such groups could support implementation by ensuring that decisions were pragmatic and by maintaining commitment. Studies also reported that these groups could also help to ensure that intervention activities added up to a co-ordinated process of integrated school transformation, rather than merely a disparate set of initiatives.
Interventions with better synergies between intervention components appeared to be implemented more successfully. Synergy appeared to occur when the consequences of enacting one intervention activity (e.g. training or provision of local data) helped provide the conditions required for the implementation of other components. Some evaluations reported a lack of synergy between curriculum and whole-school components. In some cases, classroom curriculum activities were enacted, but whole-school changes intended to reinforce what students learned in these classes were incompletely delivered. In other cases, whole-school elements that aimed to build on existing school achievements were enacted, but curriculum elements were not delivered with fidelity because these were judged by staff to be unworkable.
The extent to which staff in schools could come together to collectively enact interventions also depended on local resources. Time was in chronic short supply in many schools, which undermined the ability of staff to implement interventions. Another key resource was whether or not those charged with leading the intervention were empowered to do this. Such leadership resources could include the appropriate budget, the ability to direct other staff or the ability to modify policies or systems. Another key resource was whether or not school systems possessed strong connections between staff and cultures of innovation. Conversely, a culture of teacher autonomy could undermine collective action.
Evaluations also examined how schools reflexively monitored implementation. Implementation was reported as taking time to build as school staff and students gradually came to define their roles in intervention processes and develop the commitment and cognitive capacity they needed to effectively facilitate implementation. Reflexive monitoring, whether through formal or informal processes, was important in determining the extent to which implementation built up or dissipated over time. Interventions were particularly successful when they included an action group that oversaw delivery and reflected on the results. These processes enabled members to reflexively monitor what was being enacted and with what consequences. This could give participants the permission and resources to try different things, persist with what was perceived as working and refine or reject what was perceived to go less well. ‘Quick wins’ evidencing positive outcomes could help maintain collective impetus to implement further intervention activities. Ongoing support from training, facilitation or coaching could also support reflexive monitoring by providing an opportunity for reflection and/or an outsider perspective.
Syntheses of outcome evaluations
The third RQ asked, overall and by intervention subtype, what are the effects on student substance use, violence and educational attainment of whole-school interventions aiming to promote student commitment.
Overall meta-analyses suggested that interventions promoting commitment to school generate what we as researchers and our policy stakeholders regarded as small, but statistically significant, impacts, potentially of public health significance, in reducing violence perpetration at up to 1 year post baseline (OR 0.85, 95% CI 0.76 to 0.96) and > 1 year post baseline (OR 0.79, 95% CI 0.65 to 0.98). Considering intervention subtypes, meta-analyses suggested possible, but inconsistent, impacts of interventions promoting student participation in school policy decisions and interventions promoting student relationships with teachers on violence perpetration at up to 1 year post baseline and > 1 year post baseline. Meta-analyses suggested interventions promoting student engagement in learning and of interventions promoting parental involvement were unlikely to affect violence perpetration up to 1 year post baseline, or > 1 year post baseline.
Overall meta-analyses suggested that interventions promoting commitment to school generate small, but statistically significant, impacts in reducing violence victimisation at up to 1 year post baseline (OR 0.84, 95% CI 0.72 to 0.98) and > 1 year post baseline (OR 0.85, 95% CI 0.73 to 0.99). In terms of intervention subtypes, evidence suggested possible, but inconsistent, impacts on violence victimisation of interventions promoting student participation in decision-making and of interventions promoting student relationships with teachers at up to 1 year post baseline and > 1 year post baseline. Studies suggested that interventions promoting parental involvement were unlikely to affect violence victimisation up to 1 year post baseline and > 1 year post baseline.
Studies of intervention effects on observed violence were not meta-analysed because of the heterogeneity of study designs, but evidence suggested that interventions promoting student participation in decision-making significantly reduced observed violence at > 1 year post baseline. Interventions promoting parental involvement significantly reduced observed violence at up to 1 year post baseline and > 1 year post baseline.
Overall meta-analyses suggested that interventions promoting commitment to school generate small, but statistically significant, impacts in reducing substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.70 to 0.97) and > 1 year post baseline (OR 0.79, 95% CI 0.62 to 0.998). Meta-analyses of such interventions suggested small and statistically non-significant reductions in measures of specific outcomes (i.e. alcohol use, smoking, illicit drug use, general substance use) at up to 1 year post baseline and > 1 year post baseline.
Considering intervention subtypes, evidence suggested possible, but inconsistent, impacts in reducing substance use of interventions promoting student participation in decision-making at up to 1 year post baseline and > 1 year post baseline, with similar findings for alcohol outcomes and illicit drug use outcomes separately, and less evidence of impact on tobacco outcomes. An analysis of omnibus substance use outcomes separately suggested a larger, but still substantially heterogeneous and non-significant, effect. One RCT reported on the effects of an intervention promoting student relationships with teachers: up to 1 year post baseline there were no significant effects on alcohol or cannabis outcomes and no significant effects on any smoking, but a significant effect on regular smoking; at > 1 year post baseline, there were no significant effects on alcohol, tobacco or cannabis use outcomes. Findings from two randomised trials suggested possible, but inconsistent, impacts in reducing substance use of interventions promoting student engagement in learning, but a non-significant impact on reducing alcohol use at up to 1 year post baseline. Evidence suggested possible, but inconsistent, impacts in reducing substance use of interventions promoting student engagement in learning > 1 year post baseline, and a non-significant impact of such interventions on reducing alcohol use > 1 year post baseline.
The syntheses of effects on academic attainment are exploratory, given that these focus only on the subset of whole-school interventions aiming to improve academic attainment via promoting student commitment to school that also report health outcomes. All such studies were of interventions promoting student participation in decision-making and reporting attainment > 1 year post baseline. Findings did not suggest clear evidence of impact on academic attainment.
Syntheses of moderation and mediation analyses
The fifth RQ asked are the effects of interventions on student substance use and violence mediated by student commitment to school, or moderated by setting or population?
Evidence was inconsistent across trials for moderation of effects on violence perpetration and on violence victimisation related to age, sex, socioeconomic position and baseline risk, and was sparse for all moderators except for sex in moderating effects on violence perpetration. Regarding observed violence, evidence was sparse across all moderators, including age, sex, ethnicity and socioeconomic deprivation, precluding any firm conclusions. Regarding substance use, moderators related to age, sex, socioeconomic position and baseline risk; evidence for all moderators was sparse and there was no clear evidence of differential effects. For educational attainment, moderators included age, sex, ethnicity and socioeconomic position; evidence was sparse, but generally suggested no gradient of effectiveness for ethnicity or socioeconomic position.
All studies examining mediation were of interventions promoting student participation in decision-making. In two out of three studies, measures of commitment (involving student reports of school climate and of academic-related behaviours) mediated intervention effects on violence perpetration. 124,184 A third study reported that a measure of student-reported school climate did not mediate intervention effects on violence perpetration. 165 Regarding violence victimisation, one study reported that a measure of student belonging, but not one of student academic commitment, mediated intervention effects on victimisation perpetration,184 whereas another study reported that neither of these measures mediated intervention effects on victimisation. 165 Regarding substance use, one study reported that a measure of student academic behaviours mediated intervention effects on substance use behaviours,124 whereas another study reported that neither of these measures mediated intervention effects on tobacco, alcohol or drug use. 165
Synthesis of economic evaluations
The fourth RQ asked what is the cost-effectiveness of such interventions, overall and by intervention subtype. Evidence was sparse, but there was some evidence of cost-effectiveness of two interventions that aimed to promote student involvement in decision-making, although it should be noted that the GST focused on prevention of violence by staff towards children, whereas the focus in the Learning Together study was on violence by children.
There was reasonable evidence to suggest that the Learning Together intervention is likely to be cost-effective in the urban/semi-urban UK context in which it was implemented. There is some reason to believe that these results might be generalisable to other UK settings. Although of high quality overall and providing some evidence, the cost-effectiveness of the GST in Uganda is less clear. Without knowledge of the economic value of averting a case of violence, its results are more difficult to interpret within a decision-making context. Furthermore, the relevance of the evidence to the UK is also less clear.
Deviations from the protocol
Deviations from the protocol are listed in Appendix 1, Table 8. We amended the list of databases that were originally intended to be searched on the advice, informed by initial pilot searches, of the information scientist. In assessing the quality of economic evaluations using an adapted version of the Drummond et al. 150 checklist, we expanded a number of questions to ensure that information that was particularly relevant to this review was extracted. Both stakeholder groups were consulted only once, rather than twice, as planned. This was because of disruption to the project arising from the COVID-19 pandemic and the challenges this raised for those working in public health and education.
As explained in Chapter 7, in the synthesis of outcomes, we were not able to structure the synthesis by age group because many interventions spanned multiple age groups and included multiyear longitudinal follow-up. The synthesis of outcome evidence also modified the way in which follow-up times were categorised to better align with those provided in the studies reviewed. We did not conduct meta-analyses for observed violence (because of the heterogeneity of measures) or for academic attainment (because the evidence is only a subset of the studies examining this outcome, i.e. those that also examine violence or substance use outcomes). Meta-analyses included only randomised trials because the analytic methods that were used in included non-randomised evaluations meant that effect sizes from these evaluations could not systematically be computed for inclusion in a meta-analysis, as explained in Chapter 7. We undertook random-effects models, but did not undertake sensitivity checks with fixed-effects models because it would be imprudent to ignore between-study variance, and we concluded that the interpretation of meta-analyses should always be closer to a random-effects model (average of the distribution of effects) than to a fixed-effects model (the common effect, with variation only due to sampling error). We did not explore moderation of effects between studies using meta-regression or qualitative comparative analysis because of the insufficient number of comparable studies. Study-level differences in effectiveness were too inconsistent and the number of studies too small to calibrate studies into effective and ineffective sets, hence qualitative comparative analysis was not possible.
Limitations
Search and study selection
The original review searches involved multiple sources and methods, and they aimed to maximise sensitivity. However, the updated searches were necessarily narrower because of the limits imposed by the COVID-19 pandemic. However, the sources that yielded all of the included study reports found as a result of the original electronic searches were included in the updated searches, so we think it unlikely that any studies were missed because of this reduced scope.
Synthesis of theories of change
The synthesis was limited by the quality of the existing theory reports, which sometimes did not describe clear pathways from intervention activities to intended outcomes.
The assessment of reported theories of change did not include assessment of the parsimony of theories of change, because we have found in past reviews that this is very difficult to consistently operationalise as a criterion of quality assessment; however, this is an important feature of theories of change. We also did not aim to systematically assess the evidence base for each of the scientific theories underpinning the intervention theories of change, because this was outside the scope of this review and would require assessing not only the evidence for the scientific theory, but also the evidence for the application of that theory to the outcomes targeted in the intervention theories of change it underpins.
Synthesis of process evaluations
The process evaluation synthesis was limited by the size and quality of eligible reports. Studies were generally weak in exploring participant accounts, in particular those of students.
Syntheses of outcome evaluations and moderation/mediation analyses
The overall certainty in findings from outcome evaluations was rated as being low to very low for all outcomes. Although the number of studies found was adequate for examining the effects on self-reported violence and substance use of the overall category of whole-school interventions aiming to promote student commitment to school, there was an insufficient number of studies to examine effects by intervention subtype or to examine effects on observed violence with precision. We can draw no firm conclusions about the effects of such interventions on educational attainments because studies of such interventions that report effects on violence or substance use are likely to be only a subset of whole-school interventions that aim to raise attainment by addressing student commitment to school. Few studies examined moderation by factors such as gender, ethnicity, socioeconomic status or baseline risk to assess the potential for such interventions to reduce or increase health inequalities or be transferable to different populations or settings. Even fewer examined whether or not intervention effects on violence or substance use were mediated by measures of student commitment.
This hindered our ability to assess whether or not these interventions were underpinned by the theorised mechanisms of modifying school environments to increase student commitment to school, and hence reduce violence and substance use.
Synthesis of economic evaluations
The synthesis of economic evaluations was limited to two studies eligible for inclusion in this review. Thus, only a small number of studies was identified, and they were performed in notably different contexts (the UK and Uganda). Although the overall quality of the two evaluations was considered to be high, interpreting the results from the evaluation of the GST was difficult as its results are presented in terms of the additional cost of averting a case of violence, rather than a disability-adjusted life-year, or gaining a QALY. The time horizon for both studies was relatively short, 36 months at most, meaning that neither study considered the potential longer-term impact of reducing school-based violence. Furthermore, the relevance to the UK of the evidence from the GST study is also less clear.
Conclusions
Whole-school interventions aiming to promote student commitment to school to prevent violence and substance use can be categorised into those that promote student participation in school decisions, better staff–student relationships, student engagement in learning and parental involvement in school. Despite not generally being explicitly informed by the theory of human functioning and school organisation, most such interventions, other than some that focus primarily on teaching student social skills, are underpinned by theories of change that closely align with constructs in this theory. Factors influencing how well these interventions were implemented include whether or not interventions were tailorable, workable, well explained and had synergistic components. Interventions with action groups comprising staff/students, etc. and providing local data were well implemented. Implementation was also affected by whether or not schools accepted the need for school transformation and whether or not staff had the necessary time, budgets, authority and connections for delivery.
This category of whole-school intervention appears effective in preventing violence victimisation and perpetration, and substance use. However, the effects of such interventions appear to be small, as judged by ourselves and by the policy stakeholders. Although such interventions may play a significant role in broader efforts by schools and society to reduce young people’s involvement in violence and substance use, they are unlikely on their own to offer large changes in population behaviours. Economic evaluations were sparse, but provide some evidence that such interventions are cost-effective. There was sparse evidence on intervention effect moderators and mediators, but this did not point to consistent differences in effect.
This review also provides some insights into the validity of the theory of human functioning and school organisation. 46 The synthesis of intervention theories of change suggested some refinements to an overall theory of change for whole-school interventions aiming to prevent violence and substance use via increasing student commitment to school. However, it did not suggest significant changes to the theory of human functioning and school organisation itself. Although only one intervention was explicitly informed by this theory,46 intervention theories of change aligned closely with the theory. Therefore, their effectiveness provides some empirical support for the theory.
The finding of a lack of moderation of intervention effects on violence and substance use by socioeconomic status does, however, undermine the emphasis within the theory of human functioning and school organisation that reframing provision and eroding boundaries within schools will be particularly significant for students of lower socioeconomic status in building school commitment and preventing health risk behaviours. However, we should interpret the results cautiously, given that only three studies examined moderation of effects on violence or substance use by socioeconomic status. The finding that not all studies reported mediation of intervention effects on violence and substance use by various measures of student commitment to school might also be taken as evidence against the theory of human functioning and school organisation. However, we would stress that only three studies examined this question. 124,165,184 Furthermore, we would also point out that the study that reported that a measure of student commitment did not mediate intervention effects on violence and substance use (which some of the authors of this review worked on) later produced another analysis that does lend support to mediation by student commitment to school. 205 This analysis, published after our searches were completed, and therefore not included in our review, reported a moderated mediation analysis. This found that a measure of student commitment to school was a mediator of intervention effects on bullying, but that such moderation occurred in only a subset of schools. This subset comprised schools with fewer institutional challenges at baseline (lower rates of bullying, leadership rated highly by government inspectors and high rates of student inclusion), hypothesised as enabling these schools to deliver intervention components most likely to build student commitment to reduce involvement in risk behaviours. 205 Thus, all existing studies examining mediation of the interventions forming the focus of the present review do provide some evidence of mediation by measures of student commitment to school. On balance then, this review provides empirical support for the theory of human functioning and school organisation but might cautiously suggest that the mechanisms this theory proposes work equally for all students, regardless of socioeconomic status.
Implications for policy, practice and future research
The finding that this category of whole-school interventions has small, but significant, effects on reducing violence and substance use, together with the finding from the stakeholder consultation that there is considerable policy and practice interest in this intervention type, suggest that this is an area warranting further research in an attempt to further optimise interventions to maximise their effectiveness and to optimise the quality of evaluations to maximise their usefulness. Such research should address primary and secondary school phases. It should be conducted in the UK to provide evidence most directly relevant to informing policy here, but should also occur in other countries, including low- and middle-income countries for which there is currently little pertinent evidence. These findings also suggest that schools might deliver such interventions where there is evidence for their effectiveness, particularly when such evidence is from similar settings and populations.
Interventions might be further optimised by ensuring that they are well theorised, for example by being informed by the theory of human functioning and school organisation. This would help ensure that interventions have the potential to promote student commitment to school by addressing the different mechanisms via which this might happen. These might include mechanisms such as reframing provision on measured and/or expressed student needs, improving relationships between staff and students through processes such as joint decision-making bodies or restorative practices, eroding boundaries between academic learning and broader development via innovative teaching methods, and eroding boundaries between schools and their local communities by students volunteering in communities and parents or other community members volunteering in schools. Interventions should be co-theorised with school staff and students to ensure that theories of change and intervention materials use terminology that is acceptable to schools, informed by our stakeholder consultation. Effectiveness is also likely to be improved by ensuring that interventions address not only ‘upstream’ institutional influences on violence or substance use, such as school environment, but also more ‘downstream’ individual influences, such as student knowledge, skills and norms. Interventions might also be optimised by designing them to be maximally implementable, for example via ensuring good guidance, senior leadership, providing local needs’ data and developing collaborative co-ordinating bodies. Our stakeholders supported the finding that interventions are likely to be most feasible when they use standardised procedures, including needs assessment and collaborative decision-making, to ensure that interventions are tailored to local needs, capacities and priorities.
Evaluations might be optimised by piloting interventions and evaluation methods prior to larger studies of effectiveness. Effectiveness studies should employ large, RCT designs, which are important to estimate what may sometimes be small, but nonetheless important, intervention effects while minimising bias and confounding. Studies should include process evaluations to examine intervention acceptability and fidelity, as well as factors affecting this. This should contribute to intervention refinements and inform assessments of potential intervention transferability to other settings and populations. Informed by our consultation, studies should also assess intervention impacts on educational outcomes such as school engagement, aspirations and attainment, as well as staff time spent addressing misbehaviour. Process evaluations should draw on quantitative and qualitative data, and explore staff and student views. Finally, future interventions should examine moderation and mediation of intervention effects. Moderation analyses are important for exploring heterogeneity of effects to assess the extent to which an intervention might address health inequalities or be transferable to different populations. Mediation analyses are important to assess not merely whether or not interventions work, but how they work (and whether or not increasing student commitment to school is central to this).
Acknowledgements
The authors gratefully acknowledge members of the patient and public involvement stakeholder group who took part in consultations during the course of this review. The authors are also grateful to the individuals and organisations involved in developing the proposal for and approach to this review: Professor George Patton, Public Health England, Department for Education, Education Endowment Foundation and Healthy Schools London.
Contributions of authors
Ruth Ponsford (https://orcid.org/0000-0003-2612-0249) (Assistant Professor) managed the study and led the data extraction; the quality assessment; and the syntheses of intervention types, theory and process data.
GJ Melendez-Torres (https://orcid.org/0000-0002-9823-4790) (Professor of Clinical and Social Epidemiology) led the quantitative analyses, data extraction, quality assessment and synthesis of evidence from outcome evaluations.
Alec Miners (https://orcid.org/0000-0003-1850-1463) (Associate Professor in Health Economics) advised on the methods of quality appraisal and synthesis for cost-effectiveness evidence, and led these components.
Jane Falconer (https://orcid.org/0000-0002-7329-0577) (Professional Services User Support Librarian) planned and implemented the searches, managing and de-duplicating found references and writing up the results of these.
Chris Bonell (https://orcid.org/0000-0002-6253-6498) (Professor of Public Health Sociology) was the principal investigator and directed the review, overseeing all stages and components. He was directly involved with other investigators in screening; data extraction; quality assessment; and syntheses of intervention types, theory, qualitative and economic evaluation evidence. He oversaw searching and the synthesis of statistical evidence.
All authors contributed to the review methods and all contributed to, read and approved the final manuscript.
Data-sharing statement
All data underpinning this report are available on request from the corresponding author.
Ethics approval
The research involved no human participants and drew solely on evidence in the public realm, so ethics approval was not required.
Publication
Ponsford R, Falconer J, Melendez-Torres GJ, Bonell C. Whole-school interventions promoting student commitment to school to prevent substance use and violence: synthesis of theories of change. Health Educ J 2022;81:614–37.
Disclaimers
This report presents independent research funded by the National Institute for Health and Care Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the PHR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the PHR programme or the Department of Health and Social Care.
References
- Bonell C, Jamal F, Harden A, Wells H, Parry W, Fletcher A, et al. Systematic review of the effects of schools and school environment interventions on health: evidence mapping and synthesis. Public Health Res 2013;1. https://doi.org/10.3310/phr01010.
- Bier MC, Zwarun L, Warren VF. Getting universal primary tobacco use prevention into priority area schools a media literacy approach. Health Promot Pract 2011;12:152S-158S. https://doi.org/10.1177/1524839911414887.
- Jones SM, Brown JL, Lawrence Aber J. Two-year impacts of a universal school-based social-emotional and literacy intervention: an experiment in translational developmental research. Child Dev 2011;82:533-54. https://doi.org/10.1111/j.1467-8624.2010.01560.x.
- Kupersmidt JB, Scull TM, Benson JW. Improving media message interpretation processing skills to promote healthy decision making about substance use: the effects of the middle school media ready curriculum. J Health Commun 2012;17:546-63. https://doi.org/10.1080/ 10810730.2011.635769.
- Lynskey MT, Fergusson DM. Childhood conduct problems, attention deficit behaviors, and adolescent alcohol, tobacco, and illicit drug use. J Abnorm Child Psychol 1995;23:281-302. https://doi.org/10.1007/BF01447558.
- Resnick MD, Bearman PS, Blum RW, Bauman KE, Harris KM, Jones J, et al. Protecting adolescents from harm. Findings from the National Longitudinal Study on Adolescent Health. JAMA 1997;278:823-32. https://doi.org/10.1001/jama.278.10.823.
- Tobler AL, Komro KA, Dabroski A, Aveyard P, Markham WA. Preventing the link between SES and high-risk behaviors: ‘value-added’ education, drug use and delinquency in high-risk, urban schools. Prev Sci 2011;12:211-21. https://doi.org/10.1007/s11121-011-0206-9.
- Bonell CP, Fletcher A, Jamal F, Wells H, Harden A, Murphy S, et al. Theories of how the school environment impacts on student health: systematic review and synthesis. Health Place 2013;24:242-9. https://doi.org/10.1016/j.healthplace.2013.09.014.
- Brooks F, Klemera E, Chester K, Magnusson J, Spencer N. HBSC England National Report: Findings from the 2018 HBSC Study for England 2020.
- Nutt DJ, King LA, Phillips LD. Independent Scientific Committee on Drugs. Drug harms in the UK: a multicriteria decision analysis. Lancet 2010;376:1558-65. https://doi.org/10.1016/S0140-6736(10)61462-6.
- NHS Digital . Statistics on Alcohol: England, 2013 2013.
- Home Office . The Government’s Alcohol Strategy 2012.
- Droomers M, Schrijvers CT, Casswell S, Mackenbach JP. Occupational level of the father and alcohol consumption during adolescence; patterns and predictors. J Epidemiol Community Health 2003;57:704-10. https://doi.org/10.1136/jech.57.9.704.
- Hingson RW, Heeren T, Edwards EM. Age at drinking onset, alcohol dependence, and their relation to drug use and dependence, driving under the influence of drugs, and motor-vehicle crash involvement because of drugs. J Stud Alcohol Drugs 2008;69:192-201. https://doi.org/10.15288/jsad.2008.69.192.
- Viner RM, Taylor B. Adult outcomes of binge drinking in adolescence: findings from a UK national birth cohort. J Epidemiol Community Health 2007;61:902-7. https://doi.org/10.1136/jech.2005.038117.
- World Health Organization . Evidence-Based Strategies and Interventions to Reduce Alcohol-Related Harm. A60 14 2007.
- Masterman PW, Kelly AB. Reaching adolescents who drink harmfully: fitting intervention to developmental reality. J Subst Abuse Treat 2003;24:347-55. https://doi.org/10.1016/S0740-5472(03)00047-3.
- Department of Health and Social Care . Healthy Lives, Healthy People: Our Strategy for Pubic Health in England 2010.
- Allender S, Balakrishnan R, Scarborough P, Webster P, Rayner M. The burden of smoking-related ill health in the UK. Tob Control 2009;18:262-7. https://doi.org/10.1136/tc.2008.026294.
- Nash R, Featherstone H. Cough Up: Balancing Tobacco Income and Costs in Society. London: Policy Exchange; 2010.
- Dunstan S. General Lifestyle Survey Overview: A Report on the 2010 General Lifestyle Survey. Newport: Office for National Statistics; 2012.
- Hibell B, Guttormsson U, Ahlström S, Balakireva O, Bjarnason T, Kokkevi A, et al. The 2011 ESPAD Report: Substance Use Among Students in 36 European Countries 2011.
- Department of Health and Social Care . Smoking Kills: A White Paper on Tobacco 1998.
- Ferguson DM, Boden JM, Horwood LJ. Cannabis use and other illicit drug use: testing the cannabis gateway hypothesis. Addiction 2006;101:556-69. https://doi.org/10.1111/j.1360-0443.2005.01322.x.
- Hall W. The mental health risks of adolescent cannabis use. PLOS Med 2006;3. https://doi.org/10.1371/journal.pmed.0030039.
- Moore TH, Zammit S, Lingford-Hughes A, Barnes TR, Jones PB, Burke M, et al. Cannabis use and risk of psychotic or affective mental health outcomes: a systematic review. Lancet 2007;370:319-28. https://doi.org/10.1016/S0140-6736(07)61162-3.
- Parrott AC, Lees A, Garnham NJ, Jones M, Wesnes K. Cognitive performance in recreational users of MDMA of ‘ecstasy’: evidence for memory deficits. J Psychopharmacol 1998;12:79-83. https://doi.org/10.1177/026988119801200110.
- Beautrais AL, Joyce PR, Mulder RT. Cannabis abuse and serious suicide attempts. Addiction 1999;94:1155-64. https://doi.org/10.1046/j.1360-0443.1999.94811555.x.
- Charlton J, Kelly S, Dunnell K, Evans B, Jenkins R. Suicide deaths in England and Wales: trends in factors associated with suicide deaths. Popul Trends 1993:7134-42.
- Thomas J, Kavanagh J, Tucker H, Burchett HED, Tripney J, Oakley A. Accidental Injury, Risk-taking Behaviour and the Social Circumstances in Which Young People (Aged 12–24) Live: A Systematic Review. London: EPPI-Centre; 2007.
- Calafat A, Blay N, Juan M, Adrover D, Bellis MA, Hughes K, et al. Traffic risk behaviors at nightlife: drinking, taking drugs, driving, and use of public transport by young people. Traffic Inj Prev 2009;10:162-9. https://doi.org/10.1080/15389580802597054.
- Drugs Strategy Directorate . Updated Drug Strategy 2002 2002.
- Jayakody A, Sinha S, Roberts H, Viner R. Smoking, Drinking, Drug Use, Mental Health and Sexual Behaviour in Young People in East London. London: Department of Health and Social Care, Teenage Pregnancy Unit; 2005.
- Jessor R, Donovan JE, Costa FM. Beyond Adolescence: Problem Behaviour and Young Adult Development. Cambridge: Cambridge University Press; 1994.
- Krug EG, Mercy JA, Dahlberg LL, Zwi AB. The world report on violence and health. Lancet 2002;360:1083-8. https://doi.org/10.1016/S0140-6736(02)11133-0.
- Scott S, Knapp M, Henderson J, Maughan B. Financial cost of social exclusion: follow up study of antisocial children into adulthood. BMJ 2001;323. https://doi.org/10.1136/bmj.323. 7306.191.
- Beinart S, Anderson B, Lee SY, Utting D. Youth at Risk? A National Survey of Risk Factors, Protective Factors and Problem Behaviour among Young People in England, Scotland and Wales (JRF Findings 432). York: Joseph Rowntree Foundation; 2002.
- Bender D, Lösel F. Bullying at school as a predictor of delinquency, violence and other anti-social behaviour in adulthood. Crim Behav Ment Health 2011;21:99-106. https://doi.org/10.1002/cbm.799.
- Olweus D. Bullying at School: What We Know and What We Can Do. Malden, MA: Blackwell; 1993.
- Sainsbury Centre for Mental Health . Diversion: A Better Way for Criminal Justice and Mental Health 2009.
- Faggiano F, Vigna-Taglianti FD, Versino E, Zambon A, Borraccino A, Lemma P. School-based prevention for illicit drugs use: a systematic review. Prev Med 2008;46:385-96. https://doi.org/10.1016/j.ypmed.2007.11.012.
- Foxcroft DR, Tsertsvadze A. Cochrane review: universal school-based prevention programs for alcohol misuse in young people. Evid Based Child Health 2012;7:450-575. https://doi.org/10.1002/ebch.1829.
- Thomas RE, McLellan J, Perera R. School-based programmes for preventing smoking. Cochrane Database Syst Rev 2013;4. https://doi.org/10.1002/14651858.CD001293.pub3.
- Wells J, Barlow J, Stewart-Brown S. A systematic review of universal approaches to mental health promotion in schools. Health Educ 2003;103:197-220. https://doi.org/10.1108/09654280310485546.
- Marmot MG. Fair Society, Healthy Lives. The Marmot Review. Strategic Review of Health Inequalities in England Post-2010. London: The Marmot Review; 2010.
- Markham WA, Aveyard P. A new theory of health promoting schools based on human functioning, school organisation and pedagogic practice. Soc Sci Med 2003;56:1209-20. https://doi.org/10.1016/S0277-9536(02)00120-X.
- Youth Justice Board for England and Wales . National Evaluation of the Restorative Justice in Schools Programme 2004.
- Bonell C, Wells H, Harden A, Jamal F, Fletcher A, Thomas J, et al. The effects on student health of interventions modifying the school environment: systematic review. J Epidemiol Community Health 2013;67:677-81. https://doi.org/10.1136/jech-2012-202247.
- Bonell C, Parry W, Wells H, Jamal F, Fletcher A, Harden A, et al. The effects of the school environment on student health: a systematic review of multi-level studies. Health Place 2013;21:180-91. https://doi.org/10.1016/j.healthplace.2012.12.001.
- Patton G, Sawyer SM, Santelli JS, Ross DA, Afifi R, Allen NB, et al. Our future: a Lancet commission on adolescent health and wellbeing. Lancet 2016;387:2423-78. https://doi.org/10.1016/S0140-6736(16)00579-1.
- Public Health England . The Link Between Pupil Health and Wellbeing and Attainment: A Briefing for Head Teachers, Governors and Staff in Education Settings 2014.
- Lewis KM, Bavarian N, Snyder FJ, Acock A, Day J, Dubois DL, et al. Direct and mediated effects of a social-emotional and character development program on adolescent substance use. Int J Emot Educ 2012;4:56-78.
- Tancred T, Paparini S, Melendez-Torres GJ, Fletcher A, Thomas J, Campbell R, et al. Interventions integrating health and academic interventions to prevent substance use and violence: a systematic review and synthesis of process evaluations. Syst Rev 2018;7. https://doi.org/10.1186/s13643-018-0886-3.
- Herlitz L, MacIntyre H, Osborn T, Bonell C. The sustainability of public health interventions in schools: a systematic review. Implement Sci 2020;15. https://doi.org/10.1186/s13012-019-0961-8.
- Langford R, Bonell CP, Jones HE, Pouliou T, Murphy SM, Waters E, et al. The WHO Health Promoting School framework for improving the health and well-being of students and staff. Cochrane Database Syst Rev 2014;4. https://doi.org/10.1002/14651858.CD008958.pub2.
- Akers J, Aguiar-Ibáñez R, Baba-Akbari Sari A, Beynon S, Booth A, Burch J, et al. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. York: Centre for Reviews and Dissemination, University of York; 2009.
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Med 2009;6. https://doi.org/10.1371/journal.pmed.1000097.
- Allara E, Angelini P, Gorini G, Bosi S, Carreras G, Gozzi C, et al. A prevention program for multiple health-compromising behaviors in adolescence: baseline results from a cluster randomized controlled trial. Prev Med 2015;71:20-6. https://doi.org/10.1016/j.ypmed.2014.12.002.
- Andersen S, Rod MH, Ersbøll AK, Stock C, Johansen C, Holmberg T, et al. Effects of a settings-based intervention to promote student wellbeing and reduce smoking in vocational schools: a non-randomized controlled study. Soc Sci Med 2016;161:195-203. https://doi.org/10.1016/j.socscimed.2016.06.012.
- Andersen S, Tolstrup JS, Rod MH, Ersbøll AK, Sørensen BB, Holmberg T, et al. Shaping the social: design of a settings-based intervention study to improve well-being and reduce smoking and dropout in Danish vocational schools. BMC Public Health 2015;15. https://doi.org/10.1186/s12889-015-1936-6.
- Bavarian N, Lewis KM, Dubois DL, Acock A, Vuchinich S, Silverthorn N, et al. Using social–emotional and character development to improve academic outcomes: a matched-pair, cluster-randomized controlled trial in low-income, urban schools. J Sch Health 2013;83:771-9. https://doi.org/10.1111/josh.12093.
- Beets MW. Factors Associated with the Implementation Fidelity of a School-based Social and Character Development Program: Findings from the Positive Action Program, Hawai’i. Corvallis, OR: Oregon State University; 2007.
- Beets MW, Flay BR, Vuchinich S, Acock AC, Li KK, Allred C. School climate and teachers’ beliefs and attitudes associated with implementation of the positive action program: a diffusion of innovations model. Prev Sci 2008;9:264-75. https://doi.org/10.1007/s11121-008-0100-2.
- Beets MW, Flay BR, Vuchinich S, Snyder FJ, Acock A, Li KK, et al. Use of a social and character development program to prevent substance use, violent behaviors, and sexual activity among elementary-school students in Hawaii. Am J Public Health 2009;99:1438-45. https://doi.org/10.2105/AJPH.2008.142919.
- Björklund K, Liski A, Samposalo H, Lindblom J, Hella J, Huhtinen H, et al. ‘Together at school’ – a school-based intervention program to promote socio-emotional skills and mental health in children: study protocol for a cluster randomized controlled trial. BMC Public Health 2014;14. https://doi.org/10.1186/1471-2458-14-1042.
- Bond L, Glover S, Godfrey C, Butler H, Patton GC. Building capacity for system-level change in schools: lessons from the Gatehouse Project. Health Educ Behav 2001;28:368-83. https://doi.org/10.1177/109019810102800310.
- Bond L, Patton G, Glover S, Carlin JB, Butler H, Thomas L, et al. The Gatehouse Project: can a multilevel school intervention affect emotional wellbeing and health risk behaviours?. J Epidemiol Community Health 2004;58:997-1003. https://doi.org/10.1136/jech.2003.009449.
- Bond L, Thomas L, Coffey C, Glover S, Butler H, Carlin JB, et al. Long-term impact of the Gatehouse Project on cannabis use of 16-year-olds in Australia. J Sch Health 2004;74:23-9. https://doi.org/10.1111/j.1746-1561.2004.tb06597.x.
- Bonell C, Sorhaindo A, Strange V, Wiggins M, Allen E, Fletcher A, et al. A pilot whole-school intervention to improve school ethos and reduce substance use. Health Educ 2010;110:252-72. https://doi.org/10.1108/09654281011052628.
- Bonell C, Allen E, Christie D, Elbourne D, Fletcher A, Grieve R, et al. Initiating change locally in bullying and aggression through the school environment (INCLUSIVE): study protocol for a cluster randomised controlled trial. Trials 2014;15. https://doi.org/10.1186/1745-6215-15-381.
- Bonell C, Fletcher A, Fitzgerald-Yau N, Hale D, Allen E, Elbourne D, et al. Initiating change locally in bullying and aggression through the school environment (INCLUSIVE): a pilot randomised controlled trial. Health Technol Assess 2015;19. https://doi.org/10.3310/hta19530.
- Bonell C, Mathiot A, Allen E, Bevilacqua L, Christie D, Elbourne D, et al. Initiating change locally in bullying and aggression through the school environment (INCLUSIVE) trial: update to cluster randomised controlled trial protocol. Trials 2017;18. https://doi.org/10.1186/s13063-017-1984-6.
- Bonell CP, Sorhaindo AM, Allen EE, Strange VJ, Wiggins M, Fletcher A, et al. Pilot multimethod trial of a school-ethos intervention to reduce substance use: building hypotheses about upstream pathways to prevention. J Adolesc Health 2010;47:555-63. https://doi.org/10.1016/j.jadohealth. 2010.04.011.
- Busch V, De Leeuw RJ, Schrijvers AJ. Results of a multibehavioral health-promoting school pilot intervention in a Dutch secondary school. J Adolesc Health 2013;52:400-6. https://doi.org/10.1016/j.jadohealth.2012.07.008.
- Carlson CE. HIPP: a comprehensive school-based substance abuse program with cooperative community involvement. J Prim Prev 1990;10:289-302. https://doi.org/10.1007/BF01326623.
- Chen L, Chen Y, Hao Y, Gu J, Guo Y, Ling W. Effectiveness of school-based smoking intervention in middle school students of Linzhi Tibetan and Guangzhou Han ethnicity in China. Addict Behav 2014;39:189-95. https://doi.org/10.1016/j.addbeh.2013.09.026.
- Cousson-Gélie F, Lareyre O, Margueritte M, Paillart J, Huteau ME, Djoufelkit K, et al. Preventing tobacco in vocational high schools: study protocol for a randomized controlled trial of P2P, a peer to peer and theory planned behavior-based program. BMC Public Health 2018;18. https://doi.org/10.1186/s12889-018-5226-y.
- Crooks CV, Scott K, Ellis W, Wolfe DA. Impact of a universal school-based violence prevention program on violent delinquency: distinctive benefits for youth with maltreatment histories. Child Abuse Negl 2011;35:393-400. https://doi.org/10.1016/j.chiabu.2011.03.002.
- Devries KM, Allen E, Child JC, Walakira E, Parkes J, Elbourne D, et al. The Good Schools Toolkit to prevent violence against children in Ugandan primary schools: study protocol for a cluster randomised controlled trial. Trials 2013;14. https://doi.org/10.1186/1745-6215-14-232.
- Devries KM, Knight L, Allen E, Parkes J, Kyegombe N, Naker D. Does the Good Schools Toolkit reduce physical, sexual and emotional violence, and injuries, in girls and boys equally? A cluster-randomised controlled trial. Prev Sci 2017;18:839-53. https://doi.org/10.1007/s11121-017-0775-3.
- Duncan R, Washburn IJ, Lewis KM, Bavarian N, DuBois DL, Acock AC, et al. Can universal SEL programs benefit universally? Effects of the Positive Action Program on multiple trajectories of social-emotional and misconduct behaviors. Prev Sci 2017;18:214-24. https://doi.org/10.1007/s11121-016-0745-1.
- Eischens A, Komro KA, Perry CL, Bosma LM, Farbakhsh K. The association of extracurricular activity participation with substance use among youth in the DARE Plus project. Am J Health Educ 2004;35:68-75. https://doi.org/10.1080/19325037.2004.10603612.
- Fagen MC, Flay BR. Sustaining a school-based prevention program: results from the Aban Aya Sustainability Project. Health Educ Behav 2009;36:9-23. https://doi.org/10.1177/1090198106291376.
- Flay BR, Allred CG. International Research Handbook on Values Education and Student Wellbeing. New York, NY: Springer; 2010.
- Flay BR, Allred CG, Ordway N. Effects of the Positive Action program on achievement and discipline: two matched-control comparisons. Prev Sci 2001;2:71-89. https://doi.org/10.1023/A:1011591613728.
- Flay BR, Berkowitz MW, Bier MC. Elementary school-based programs theorized to support social development, prevent violence, and promote positive school climate: description and hypothesized mechanisms of change. J Res Charact Educ 2009;7:21-49.
- Flay BR, Graumlich S, Segawa E, Burns JL, Holliday MY. Aban Aya Investigators. Effects of 2 prevention programs on high-risk behaviors among African American youth: a randomized trial. Arch Pediatr Adolesc Med 2004;158:377-84. https://doi.org/10.1001/archpedi.158.4.377.
- Fletcher A, Fitzgerald-Yau N, Wiggins M, Viner R. Involving young people in changing their school environment to make it safer: findings from a process evaluation in English secondary schools. Health Educ 2015;115:322-38. https://doi.org/10.1108/HE-04-2014-0063.
- Greco G, Knight L, Ssekadde W, Namy S, Naker D, Devries K. Economic evaluation of the Good School Toolkit: an intervention for reducing violence in primary schools in Uganda. BMJ Glob Health 2018;3. https://doi.org/10.1136/bmjgh-2017-000526.
- Guo S, Wu Q, Smokowski PR, Bacallao M, Evans CB, Cotter KL. A longitudinal evaluation of the Positive Action Program in a low-income, racially diverse, rural county: effects on self-esteem, school hassles, aggression, and internalizing symptoms. J Youth Adolesc 2015;44:2337-58. https://doi.org/10.1007/s10964-015-0358-1.
- Hawe P, Bond L, Ghali LM, Perry R, Davison CM, Casey DM, et al. Replication of a whole school ethos-changing intervention: different context, similar effects, additional insights. BMC Public Health 2015;15. https://doi.org/10.1186/s12889-015-1538-3.
- Henderson M, Jackson C, Levin K, McConnachie A, Wight D, Elliott L, et al. Social and Emotional Education and Development (SEED): A Stratified, Cluster Randomised Trial of a Multi-Component Primary School Intervention That Follows the Pupils’ Transition into Secondary School n.d. www.fundingawards.nihr.ac.uk/award/10/3006/13 (accessed 22 August 2022).
- Hodder RK, Freund M, Bowman J, Wolfenden L, Campbell E, Wye P, et al. A cluster randomised trial of a school-based resilience intervention to decrease tobacco, alcohol and illicit drug use in secondary school students: study protocol. BMC Public Health 2012;12. https://doi.org/10.1186/1471-2458-12-1009.
- Ingholt L, Sørensen BB, Andersen S, Zinckernagel L, Friis-Holmberg T, Frank VA, et al. How can we strengthen students’ social relations in order to reduce school dropout? An intervention development study within four Danish vocational schools. BMC Public Health 2015;15. https://doi.org/10.1186/s12889-015-1831-1.
- Jagers RJ, Morgan-Lopez AA, Howard TL, Browne DC, Flay BR. Aban Aya Coinvestigators. Mediators of the development and prevention of violent behavior. Prev Sci 2007;8:171-9. https://doi.org/10.1007/s11121-007-0067-4.
- Kayiwa J, Clarke K, Knight L, Allen E, Walakira E, Namy S, et al. Effect of the Good School Toolkit on school staff mental health, sense of job satisfaction and perceptions of school climate: secondary analysis of a cluster randomised trial. Prev Med 2017;101:84-90. https://doi.org/10.1016/j.ypmed.2017.05.022.
- Kelder SH, Orpinas P, McAlister A, Frankowski R, Parcel GS, Friday J. The students for peace project: a comprehensive violence-prevention program for middle school students. Am J Prev Med 1996;12:22-30. https://doi.org/10.1016/S0749-3797(18)30233-2.
- Knight L, Allen E, Mirembe A, Nakuti J, Namy S, Child JC, et al. Implementation of the Good School Toolkit in Uganda: a quantitative process evaluation of a successful violence prevention program. BMC Public Health 2018;18. https://doi.org/10.1186/s12889-018-5462-1.
- Komro KA, Perry CL, Murray DM, Veblen-Mortenson S, Williams CL, Anstine PS. Peer-planned social activities for preventing alcohol use among young adolescents. J Sch Health 1996;66:328-34. https://doi.org/10.1111/j.1746-1561.1996.tb03413.x.
- Komro KA, Perry CL, Munson KA, Stigler MH, Farbakhsh K. Reliability and validity of self-report measures to evaluate drug and violence prevention programs. J Child Adolesc Subst Abuse 2004;13:17-51. https://doi.org/10.1300/J029v13n03_02.
- Komro KA, Perry CL, Veblen-Mortenson S, Bosma LM, Dudovitz BS, Williams CL, et al. Brief report: the adaptation of Project Northland for urban youth. J Pediatr Psychol 2004;29:457-66. https://doi.org/10.1093/jpepsy/jsh049.
- Komro KA, Perry CL, Veblen-Mortenson S, Farbakhsh K, Toomey TL, Stigler MH, et al. Outcomes from a randomized controlled trial of a multi-component alcohol use preventive intervention for urban youth: project northland Chicago. Addiction 2008;103:606-18. https://doi.org/10.1111/j.1360-0443.2007.02110.x.
- Komro KA, Perry CL, Veblen-Mortenson S, Stigler MH, Bosma LM, Munson KA, et al. Violence-related outcomes of the D.A.R.E. Plus project. Health Educ Behav 2004;31:335-54. https://doi.org/10.1177/1090198104263337.
- Komro KA, Perry CL, Veblen-Mortenson S, Williams CL. Peer participation in Project Northland: a community-wide alcohol use prevention project. J Sch Health 1994;64:318-22. https://doi.org/10.1111/j.1746-1561.1994.tb03319.x.
- Komro KA, Perry CL, Veblen-Mortenson S, Willims CL, Roel JP. Peer leadership in school and community alcohol use prevention activities. J Health Educ 1999;30:202-8. https://doi.org/10.1080/10556699.1999.10604640.
- Komro KA, Perry CL, Williams CL, Stigler MH, Farbakhsh K, Veblen-Mortenson S. How did Project Northland reduce alcohol use among young adolescents? Analysis of mediating variables. Health Educ Res 2001;16:59-70. https://doi.org/10.1093/her/16.1.59.
- Lewis KM. Evaluation of a Social–Emotional and Character Development Program: Methods and Outcomes. Corvallis, OR: Oregon State University; 2012.
- Lewis KM, Schure MB, Bavarian N, DuBois DL, Day J, Ji P, et al. Problem behavior and urban, low-income youth: a randomized controlled trial of positive action in Chicago. Am J Prev Med 2013;44:622-30. https://doi.org/10.1016/j.amepre.2013.01.030.
- Li KK, Washburn I, DuBois DL, Vuchinich S, Ji P, Brechling V, et al. Effects of the Positive Action programme on problem behaviours in elementary school students: a matched-pair randomised control trial in Chicago. Psychol Health 2011;26:187-204. https://doi.org/10.1080/ 08870446.2011.531574.
- Malloy M, Acock A, DuBois DL, Vuchinich S, Silverthorn N, Ji P, et al. Teachers’ perceptions of school organizational climate as predictors of dosage and quality of implementation of a social–emotional and character development program. Prev Sci 2015;16:1086-95. https://doi.org/10.1007/s11121-014-0534-7.
- Malmberg M, Overbeek G, Kleinjan M, Vermulst A, Monshouwer K, Lammers J, et al. Effectiveness of the universal prevention program ‘Healthy School and Drugs’: study protocol of a randomized clustered trial. BMC Public Health 2010;10. https://doi.org/10.1186/ 1471-2458-10-541.
- Miller CF, Kochel KP, Wheeler LA, Updegraff KA, Fabes RA, Martin CL, et al. The efficacy of a relationship building intervention in 5th grade. J Sch Psychol 2017;61:75-88. https://doi.org/10.1016/j.jsp.2017.01.002.
- O’Hare L, Stark P, Orr K, Biggart A, Bonell C. Positive Action: Pilot Report and Executive Summary. London: Education Endowment Foundation; 2018.
- Orpinas P, Kelder S, Frankowski R, Murray N, Zhang Q, McAlister A. Outcome evaluation of a multi-component violence-prevention program for middle schools: the Students for Peace project. Health Educ Res 2000;15:45-58. https://doi.org/10.1093/her/15.1.45.
- Patton GC, Bond L, Carlin JB, Thomas L, Butler H, Glover S, et al. Promoting social inclusion in schools: a group-randomized trial of effects on student health risk behavior and well-being. Am J Public Health 2006;96:1582-7. https://doi.org/10.2105/AJPH.2004.047399.
- Perry CL, Komro KA, Veblen-Mortenson S, Bosma L, Munson K, Stigler M, et al. The Minnesota DARE PLUS Project: creating community partnerships to prevent drug use and violence. J Sch Health 2000;70:84-8. https://doi.org/10.1111/j.1746-1561.2000.tb06451.x.
- Perry CL, Komro KA, Veblen-Mortenson S, Bosma LM, Farbakhsh K, Munson KA, et al. A randomized controlled trial of the middle and junior high school D.A.R.E. and D.A.R.E. Plus programs. Arch Pediatr Adolesc Med 2003;157:178-84. https://doi.org/10.1001/archpedi.157.2.178.
- Perry CL, Lee S, Stigler MH, Farbakhsh K, Komro KA, Gewirtz AH, et al. The impact of Project Northland on selected MMPI-A problem behavior scales. J Prim Prev 2007;28:449-65. https://doi.org/10.1007/s10935-007-0105-9.
- Perry CL, Williams CL, Forster JL, Wolfson M, Wagenaar AC, Finnegan JR, et al. Background, conceptualization and design of a community-wide research program on adolescent alcohol use: Project Northland. Health Educ Res 1993;8:125-36. https://doi.org/10.1093/her/8.1.125.
- Perry CL, Williams CL, Komro KA, Veblen-Mortenson S, Forster JL, Bernstein-Lachter R, et al. Project Northland high school interventions: community action to reduce adolescent alcohol use. Health Educ Behav 2000;27:29-4. https://doi.org/10.1177/109019810002700105.
- Perry CL, Williams CL, Komro KA, Veblen-Mortenson S, Stigler MH, Munson KA, et al. Project Northland: long-term outcomes of community action to reduce adolescent alcohol use. Health Educ Res 2002;17:117-32. https://doi.org/10.1093/her/17.1.117.
- Perry CL, Williams CL, Veblen-Mortenson S, Toomey TL, Komro KA, Anstine PS, et al. Project Northland: outcomes of a communitywide alcohol use prevention program during early adolescence. Am J Public Health 1996;86:956-65. https://doi.org/10.2105/AJPH.86.7.956.
- Snyder F, Flay B, Vuchinich S, Acock A, Washburn I, Beets M, et al. Impact of a social– emotional and character development program on school-level indicators of academic achievement, absenteeism, and disciplinary outcomes: a matched-pair, cluster randomized, controlled trial. J Res Educ Eff 2010;3:26-55. https://doi.org/10.1080/19345740903353436.
- Snyder FJ, Acock AC, Vuchinich S, Beets MW, Washburn IJ, Flay BR. Preventing negative behaviors among elementary-school students through enhancing students’ social–emotional and character development. Am J Health Promot 2013;28:50-8. https://doi.org/10.4278/ajhp.120419-QUAN-207.2.
- Van Ryzin MJ, Roseth CJ. Peer influence processes as mediators of effects of a middle school substance use prevention program. Addict Behav 2018;85:180-5. https://doi.org/10.1016/j.addbeh.2018.06.016.
- West B, Abatemarco D, Ohman-Strickland PA, Zec V, Russo A, Milic R. Project Northland in Croatia: results and lessons learned. J Drug Educ 2008;38:55-70. https://doi.org/10.2190/DE.38.1.e.
- Williams CL, Perry CL. Lessons from Project Northland: preventing alcohol problems during adolescence. Alcohol Health Res World 1998;22:107-16.
- Williams CL, Perry CL, Farbakhsh K, Veblen-Mortenson S. Project Northland: comprehensive alcohol use prevention for young adolescents, their parents, schools, peers and communities. J Stud Alcohol Suppl 1999;13:112-24. https://doi.org/10.15288/jsas.1999.s13.112.
- Winkleby MA, Feighery E, Dunn M, Kole S, Ahn D, Killen JD. Effects of an advocacy intervention to reduce smoking among teenagers. Arch Pediatr Adolesc Med 2004;158:269-75. https://doi.org/10.1001/archpedi.158.3.269.
- Wolfe DA, Crooks C, Jaffe P, Chiodo D, Hughes R, Ellis W, et al. A school-based program to prevent adolescent dating violence: a cluster randomized trial. Arch Pediatr Adolesc Med 2009;163:692-9. https://doi.org/10.1001/archpediatrics.2009.69.
- Wolfe DA, Crooks CV, Chiodo D, Hughes R, Ellis W. Observations of adolescent peer resistance skills following a classroom-based healthy relationship program: a post-intervention comparison. Prev Sci 2012;13:196-205. https://doi.org/10.1007/s11121-011-0256-z.
- Wong DS, Cheng CH, Ngan RM, Ma SK. Program effectiveness of a restorative whole-school approach for tackling school bullying in Hong Kong. Int J Offender Ther Comp Criminol 2011;55:846-62. https://doi.org/10.1177/0306624X10374638.
- Bramer WM, de Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc 2018;106:531-41. https://doi.org/10.5195/jmla.2018.283.
- McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol 2016;75:40-6. https://doi.org/10.1016/j.jclinepi.2016.01.021.
- Falconer J, Bonell C. Search Strategies for: Whole-school Interventions Promoting Student Commitment to School to Prevent Substance Abuse and Violence and Improve Educational Attainment. London: London School of Hygiene & Tropical Medicine; 2021.
- Wright J. Checking for Duplicates Guidance. Leeds: University of Leeds; 2016.
- Falconer J. Removing Duplicates from an EndNote Library n.d. https://blogs.lshtm.ac.uk/library/2018/12/07/removing-duplicates-from-an-endnote-library/ (accessed 6 May 2021).
- Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Updated March 2011]. London: The Cochrane Collaboration; 2011.
- Shepherd J, Kavanagh J, Picot J, Cooper K, Harden A, Barnett-Page E, et al. The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13–19: a systematic review and economic evaluation. Health Technol Assess 2010;14. https://doi.org/10.3310/hta14070.
- Shemilt I, Mugford M, Byford S, Drummond M, Eisenstein E, Knapp M, et al. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. London: The Cochrane Collaboration; 2008.
- Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348. https://doi.org/10.1136/bmj.g1687.
- Sterne J, Egger M, Moher D. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons, Inc.; 2008.
- Cho BK, Rosenfeldt F, Turina MI, Karp RB, Ferguson TB, Bodnar E, et al. Joint statement on redundant (duplicate) publication by the editors of the undersigned cardiothoracic journals. Ann Thorac Surg 2000;69. https://doi.org/10.1016/S0003-4975(00)01088-2.
- Bonell C, Hinds K, Dickson K, Thomas J, Fletcher A, Murphy S, et al. What is positive youth development and how might it reduce substance use and violence? A systematic review and synthesis of theoretical literature. BMC Public Health 2016;16. https://doi.org/10.1186/s12889-016-2817-3.
- Tancred T, Paparini S, Melendez-Torres GJ, Thomas J, Fletcher A, Campbell R, et al. A systematic review and synthesis of theories of change of school-based interventions integrating health and academic education as a novel means of preventing violence and substance use among students. Syst Rev 2018;7. https://doi.org/10.1186/s13643-018-0862-y.
- Meiksin R, Melendez-Torres GJ, Falconer J, Witzel TC, Weatherburn P, Bonell C. Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis. Syst Rev 2021;10. https://doi.org/10.1186/s13643-020-01523-2.
- Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med 2012;75:2299-306. https://doi.org/10.1016/j.socscimed.2012.08.032.
- Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, et al. Young People and Healthy Eating: A Systematic Review of Barriers and Facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2001.
- Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355. https://doi.org/10.1136/bmj.i4919.
- Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. Oxford: Oxford University Press; 1997.
- Sutcliffe K, Thomas J, Stokes G, Hinds K, Bangpan M. Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions. Syst Rev 2015;4. https://doi.org/10.1186/s13643-015-0126-z.
- Carroll C, Booth A, Leaviss J, Rick J. ‘Best fit’ framework synthesis: refining the method. BMC Med Res Methodol 2013;13. https://doi.org/10.1186/1471-2288-13-37.
- Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: SAGE Publications, Inc.; 1994.
- Tancred T, Melendez-Torres GJ, Paparini S, Fletcher A, Stansfield C, Thomas J, et al. Interventions integrating health and academic education in schools to prevent substance misuse and violence: systematic review and evidence synthesis. Public Health Res 2019;7. https://doi.org/10.3310/phr07170.
- May C. Towards a general theory of implementation. Implement Sci 2013;8. https://doi.org/10.1186/1748-5908-8-18.
- Cheung MW. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol Methods 2014;19:211-29. https://doi.org/10.1037/a0032968.
- Thompson S, Sharp S. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med 1999;18:693-708. https://doi.org/10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.
- Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-analysis. Chichester: John Wiley & Sons, Inc.; 2009.
- Ragin CC. Redesigning Social Inquiry: Fuzzy Sets and Beyond. Chicago, IL: University of Chicago Press; 2008.
- Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev 2014;3. https://doi.org/10.1186/2046-4053-3-67.
- Acosta J, Chinman M, Ebener P, Malone PS, Phillips A, Wilks A. Evaluation of a whole-school change intervention: findings from a two-year cluster-randomized trial of the Restorative Practices Intervention. J Youth Adolesc 2019;48:876-90. https://doi.org/10.1007/s10964-019-01013-2.
- Battistich V, Schaps E, Watson M, Solomon D, Lewis C. Effects of the Child Development Project on students’ drug use and other problem behaviors. J Prim Prev 2000;21:75-99. https://doi.org/10.1023/A:1007057414994.
- Battistich V, Schaps E, Wilson N. Effects of an elementary school intervention on students’ ‘connectedness’ to school and social adjustment during middle school. J Prim Prev 2004;24:243-62. https://doi.org/10.1023/B:JOPP.0000018048.38517.cd.
- Battistich V, Schaps E, Watson M, Solomon D. Prevention effects of the Child Development Project: early findings from an ongoing multi-site demonstration trial. J Adolesc Res 1996;11:12-35. https://doi.org/10.1177/0743554896111003.
- Bonell C, Allen E, Opondo C, Warren E, Elbourne DR, Sturgess J, et al. Examining intervention mechanisms of action using mediation analysis within a randomised trial of a whole-school health intervention. J Epidemiol Community Health 2019;73:455-64. https://doi.org/10.1136/jech-2018-211443.
- Bonell C, Allen E, Warren E, McGowan J, Bevilacqua L, Jamal F, et al. Effects of the Learning Together intervention on bullying and aggression in English secondary schools (INCLUSIVE): a cluster randomised controlled trial. Lancet 2018;392:2452-64. https://doi.org/10.1016/S0140-6736(18)31782-3.
- Bonell C, Allen E, Warren E, McGowan J, Bevilacqua L, Jamal F, et al. Modifying the secondary school environment to reduce bullying and aggression: the INCLUSIVE cluster RCT. Public Health Res 2019;7. https://doi.org/10.3310/phr07180.
- Bonell C, Dodd M, Allen E, Bevilacqua L, McGowan J, Opondo C, et al. Broader impacts of an intervention to transform school environments on student behaviour and school functioning: post hoc analyses from the INCLUSIVE cluster randomised controlled trial. BMJ Open 2020;10. https://doi.org/10.1136/bmjopen-2019-031589.
- Cross D, Monks H, Hall M, Shaw T, Pintabona Y, Erceg E, et al. Three-year results of the Friendly Schools whole-of-school intervention on children’s bullying behaviour. Br Educ Res J 2011;37:105-29. https://doi.org/10.1080/01411920903420024.
- Cross D, Shaw T, Epstein M, Pearce N, Barnes A, Burns S, et al. Impact of the Friendly Schools whole-school intervention on transition to secondary school and adolescent bullying behaviour. Eur J Educ 2018;53:495-513. https://doi.org/10.1111/ejed.12307.
- Cross D, Waters S, Pearce N, Shaw T, Hall M, Erceg E, et al. The Friendly Schools Friendly Families programme: three-year bullying behaviour outcomes in primary school children. Int J Educ Res 2012;53:394-406. https://doi.org/10.1016/j.ijer.2012.05.004.
- Devries KM, Knight L, Child JC, Mirembe A, Nakuti J, Jones R, et al. The Good School Toolkit for reducing physical violence from school staff to primary school students: a cluster-randomised controlled trial in Uganda. Lancet Glob Health 2015;3:e378-86. https://doi.org/10.1016/S2214-109X(15)00060-1.
- Flay BR, Allred CG. Long-term effects of the Positive Action program. Am J Health Behav 2003;27:S6-21. https://doi.org/10.5993/AJHB.27.1.s1.2.
- Gottfredson DC. An empirical test of school-based environmental and individual interventions to reduce the risk of delinquent behaviour. Criminology 1986;24:705-31. https://doi.org/10.1111/j.1745-9125.1986.tb01508.x.
- Hodder RK, Freund M, Bowman J, Wolfenden L, Campbell E, Dray J, et al. Effectiveness of a pragmatic school-based universal resilience intervention in reducing tobacco, alcohol and illicit substance use in a population of adolescents: cluster-randomised controlled trial. BMJ Open 2017;7. https://doi.org/10.1136/bmjopen-2017-016060.
- Hodder RK, Freund M, Bowman J, Wolfenden L, Campbell E, Dray J, et al. Differential intervention effectiveness of a universal school-based resilience intervention in reducing adolescent substance use within student subgroups: exploratory assessment within a cluster-randomised controlled trial. BMJ Open 2018;8. https://doi.org/10.1136/bmjopen-2017-021047.
- Jagers RJ, Morgan-Lopez AA, Flay BR. Aban Aya Investigators. The impact of age and type of intervention on youth violent behaviors. J Prim Prev 2009;30:642-58. https://doi.org/10.1007/s10935-009-0200-1.
- Kyriakides L, Creemers B, Papastylianou D, Papadatou-Pastou M. Improving the school learning environment to reduce bullying: an experimental study. Scand J Educ Res 2014;58:453-78. https://doi.org/10.1080/00313831.2013.773556.
- Lewis KM, Vuchinich S, Ji P, DuBois DL, Acock A, Bavarian N, et al. Effects of the Positive Action program on indicators of positive youth development among urban youth. Appl Dev Sci 2016;20:16-28. https://doi.org/10.1080/10888691.2015.1039123.
- Shinde S, Weiss HA, Khandeparkar P, Pereira B, Sharma A, Gupta R, et al. A multicomponent secondary school health promotion intervention and adolescent health: an extension of the SEHER cluster randomised controlled trial in Bihar, India. PLOS Med 2020;17. https://doi.org/10.1371/journal.pmed.1003021.
- Shinde S, Weiss HA, Varghese B, Khandeparkar P, Pereira B, Sharma A, et al. Promoting school climate and health outcomes with the SEHER multi-component secondary school intervention in Bihar, India: a cluster-randomised controlled trial. Lancet 2018;392:2465-77. https://doi.org/10.1016/S0140-6736(18)31615-5.
- Simons-Morton B, Haynie D, Saylor K, Crump AD, Chen R. Impact analysis and mediation of outcomes: the Going Places program. Health Educ Behav 2005;32:227-41. https://doi.org/10.1177/1090198104272002.
- Simons-Morton B, Haynie D, Saylor K, Crump AD, Chen R. The effects of the Going Places program on early adolescent substance use and antisocial behavior. Prev Sci 2005;6:187-97. https://doi.org/10.1007/s11121-005-0005-2.
- Singla DR, Shinde S, Patton G, Patel V. The mediating effect of school climate on adolescent mental health: findings from a randomized controlled trial of a school-wide intervention. J Adolesc Health 2021;69:90-9. https://doi.org/10.1016/j.jadohealth.2020.09.030.
- Van Ryzin MJ, Roseth CJ. Enlisting peer cooperation in the service of alcohol use prevention in middle school. Child Dev 2018;89:e459-67. https://doi.org/10.1111/cdev.12981.
- Van Ryzin MJ, Roseth CJ. Cooperative learning in middle school: a means to improve peer relations and reduce victimization, bullying, and related outcomes. J Educ Psychol 2018;110:1192-201. https://doi.org/10.1037/edu0000265.
- Van Ryzin MJ, Roseth CJ. Cooperative learning effects on peer relations and alcohol use in middle school. J Appl Dev Psychol 2019;64. https://doi.org/10.1016/j.appdev.2019.101059.
- Van Ryzin MJ, Roseth CJ. Effects of cooperative learning on peer relations, empathy, and bullying in middle school. Aggress Behav 2019;45:643-51. https://doi.org/10.1002/ab.21858.
- Anyon Y, Nicotera N, Veeh CA. Contextual influences on the implementation of a schoolwide intervention to promote students’ social, emotional, and academic learning. Child Schools 2016;38:81-8. https://doi.org/10.1093/cs/cdw008.
- Bosma LM, Komro K, Perry C, Veblen-Mortenson S, Farbakhsh K. Community organizing to prevent youth drug use and violence: the D.A.R.E. Plus Project. J Community Pract 2005;13:5-19. https://doi.org/10.1300/J125v13n02_02.
- Cross D, Barnes A, Cardoso P, Hadwen K, Shaw T, Campbell M, et al. Reducing Cyberbullying in Schools: International Evidence-based Best Practices. London: Academic Press, Elsevier, Inc.; 2018.
- Cross D, Lester L, Pearce N, Barnes A, Beatty S. A group randomized controlled trial evaluating parent involvement in whole-school actions to reduce bullying. J Educ Res 2018;111:255-67. https://doi.org/10.1080/00220671.2016.1246409.
- Mitchell S. Portland Peers Project. 1989–91 Final Evaluation Report 1991.
- Warren E, Bevilacqua L, Opondo C, Allen E, Mathiot A, West G, et al. Action groups as a participative strategy for leading whole-school health promotion: results on implementation from the INCLUSIVE trial in English secondary schools. Br Educ Res J 2019;45:979-1000. https://doi.org/10.1002/berj.3547.
- Warren E, Melendez-Torres GJ, Viner R, Bonell C. Using qualitative research to explore intervention mechanisms: findings from the trial of the Learning Together whole-school health intervention. Trials 2020;21. https://doi.org/10.1186/s13063-020-04688-2.
- Legood R, Opondo C, Warren E, Jamal F, Bonell C, Viner R, et al. Cost–utility analysis of a complex intervention to reduce school-based bullying and aggression: an analysis of the inclusive RCT. Value Health 2021;24:129-35. https://doi.org/10.1016/j.jval.2020.04.1839.
- Rapee RM, Shaw T, Hunt C, Bussey K, Hudson JL, Mihalopoulos C, et al. Combining whole-school and targeted programs for the reduction of bullying victimization: a randomized, effectiveness trial. Aggress Behav 2020;46:193-209. https://doi.org/10.1002/ab.21881.
- Solomon D, Battistich V, Watson M, Schaps E, Lewis C. A six-district study of educational change: direct and mediated effects of the Child Development Project. Soc Psychol Educ 2000;4:3-51. https://doi.org/10.1023/A:1009609606692.
- Cross D, Shaw T, Hadwen K, Cardoso P, Slee P, Roberts C, et al. Longitudinal impact of the Cyber Friendly Schools program on adolescents’ cyberbullying behavior. Aggress Behav 2016;42:166-80. https://doi.org/10.1002/ab.21609.
- Kyriakides L, Creemers B, Muijs D, Rekers-Mombarg L, Papastylianou D, Van Petegem P, et al. Using the dynamic model of educational effectiveness to design strategies and actions to face bullying. Sch Eff Sch Improv 2013;25:83-104. https://doi.org/10.1080/09243453.2013.771686.
- Jamal F, Fletcher A, Harden A, Wells H, Thomas J, Bonell C. The school environment and student health: a systematic review and meta-ethnography of qualitative research. BMC Public Health 2013;13. https://doi.org/10.1186/1471-2458-13-798.
- Flay BR, Petraitis J. Advances in Medical Sociology, Vol. IV: A Reconsideration of Models of Health Behavior Change. Greenwich: JAI Press; 1994.
- Hawe P, Shiell A, Riley T. Complex interventions: how ‘out of control’ can a randomised controlled trial be?. BMJ 2004;328:1561-3. https://doi.org/10.1136/bmj.328.7455.1561.
- National Institute for Health and Care Excellence . Methods for the Development of NICE Public Health Guidance (Third Edition): Process and Methods n.d. www.nice.org.uk/process/pmg4/resources/methods-for-the-development-of-nice-public-health-guidance-third-edition-pdf-2007967445701 (accessed 15 September 2021).
- Melendez-Torres GJ, Warren E, Viner R, Allen E, Bonell C. Moderated mediation analyses to assess intervention mechanisms for impacts on victimisation, psycho-social problems and mental wellbeing: evidence from the INCLUSIVE realist randomized trial. Soc Sci Med 2021;279. https://doi.org/10.1016/j.socscimed.2021.113984.
- Johnson DW, Johnson RT, Holubec EJ. Cooperation in the Classroom. Edina, MN: Interaction Book Company; 2013.
- Perry CL. Thousand Oaks, CA: SAGE Publications Ltd; 1999.
Appendix 1 Protocol deviations and clarifications
Date deviation implemented | Change | Rationale |
---|---|---|
16 January 2020 | Additional databases were searched: Australian Education Index, CINAHL Plus, ClinicalTrials.gov, Education Abstracts, Education Database, Educational Administration Abstracts, ERIC, Embase, Global Health, Scopus, Teacher Reference Center | A number of education-specific databases were added to make sure that school-focused literature was included. A wider range of medical, nursing and public health databases were also included to make sure that the multidisciplinary nature of this topic was adequately reflected in the sources of literature retrieved |
16 January 2020 | The following databases were not searched: Database of Abstracts of Reviews of Effects, Dissertation Abstracts (now incorporated in ProQuest Dissertations & Theses Global), NHS Economic Evaluation Database and the Health Technology Assessment database | These databases are no longer updated |
16 January 2020 | The following databases were not searched: ProQuest Dissertations & Theses Global; ProQuest International Bibliography of the Social Sciences | We could not access these because of problems with the database interface. This may result in some results from social science journals not being retrieved. However this has been mitigated by the addition of several education-specific sources and Scopus. There may also be some theses that have not been included |
16 January 2020 | The following database was not searched: eResearch Index Citations | We could not access the database because we did not have a subscription |
2 March 2021 | Follow-up periods were regarded as separate outcomes, changed from ‘less than three months, three months to one year and more than one-year post-intervention’ in the original protocol to ‘up to one year and more than one-year post-baseline’ | Interventions included in the study tended to be enduring, with very few follow-ups being < 1 school year, meaning that the ‘less than three months’ category was sparse |
1 April 2021 | The policy and practice stakeholder group and ALPHA groups were convened only once to review the complete findings from the study, instead of meeting twice as specified in the protocol | This was because of the impact of COVID (furlough of research staff and the principal investigator focusing on providing scientific advice to government on the COVID-19 pandemic) on the project timeline and a desire not to increase pressure on overstretched participants in the public health and education sectors |
19 April 2021 | In assessing the quality of economic evaluations using an adapted version of the Drummond et al.150 checklist, we expanded a number of questions | This was to ensure that information that was particularly relevant to this review was extracted |
1 May 2021 | We did not present meta-analyses for observed violence | Meta-analyses for observed violence were not possible because of heterogeneity of measures (a mix of informants for this outcome) |
1 May 2021 | We did not present meta-analyses for academic attainment | Meta-analyses for academic attainment were not carried out because it became apparent in the course of the review that there is an additional body of evidence reporting the effects on educational attainment of the sorts of whole-school interventions as defined in this review; these studies were not included in the present review because they did not also report effects on violence or substance use |
1 May 2021 | Meta-analyses included only randomised trials | The analytic methods that were used in included non-randomised evaluations meant that effect sizes from these evaluations could not systematically be computed for inclusion in a meta-analysis |
1 May 2021 | We did not attempt to synthesise evidence by age group | Many interventions spanned multiple age groups and included multiyear longitudinal follow-up |
1 May 2021 | We undertook random-effects models without sensitivity analyses using fixed-effects models | This was because of the statistical heterogeneity of included studies. We did not undertake sensitivity checks with fixed-effects models, as this would have been redundant given the evidence identified; this is because the clinical heterogeneity of included interventions, and outcome measures and follow-up times, meant that it would be imprudent to ignore any between-study variance, however slight |
1 May 2021 | We did not explore moderation of effects between studies using meta-regression or qualitative comparative analysis | This was because of the insufficient number of comparable studies. Study-level differences in effectiveness were too inconsistent and number of studies too small to calibrate studies into effective and ineffective sets; hence, qualitative comparative analysis was not possible |
Appendix 2 Descriptions of interventions
Intervention name | Country/area | Study reports | Schools targeted | Intervention timing | Intervention activities | Providers | Training and support for providers |
---|---|---|---|---|---|---|---|
AAYP school/ community intervention | USA/Chicago | Elementary schools (students aged 6–11 years) and middle schools (students aged 11–13 years) | 4 years | This intervention consisted of the Aban Aya Social Development Curriculum plus the school-wide climate and parent and community components. The curriculum was delivered usually in social studies classes. It consisted of 16–21 lessons per year in grades 5 through 8, teaching cognitive–behavioural skills to build self-esteem and empathy; manage stress and anxiety; develop interpersonal relationships; resist peer pressure; and develop decision-making, problem-solving, conflict resolution and goal-setting skills. It aimed also to promote interactive and culturally appropriate teaching methods. The parent component promoted child–parent and school–parent communication. The community programme forged linkages among parents, schools and local businesses. The school climate element involved each school forming a local school task force consisting of school personnel, students, parents, community advocates and project staff to implement the programme, propose changes in school policy, develop other school-community liaisons supportive of school-based efforts and solicit community organisations to conduct activities to support its efforts. Activities included school-wide fairs, annual field trips for parents and children and seeking additional funding from local businesses. The goal was to ‘rebuild the village’ and create a ‘sense of ownership’ by all stakeholders to promote sustainability of these efforts on completion of the project | University-based health educators delivered the curriculum | Health educators received two training sessions before each lesson in which they role-played activities and senior staff provided feedback. Weekly debriefings were held to discuss issues that may have affected implementation. Senior staff also observed sessions to ensure fidelity and help target training needs. School classroom teachers received a 4-hour workshop to provide an overview of programme philosophy, curriculum content and clarification of their support roles | |
CDP | USA/multiple sites | Elementary schools (students aged 6–11 years) | 3 years | The CDP programme consists of an intensive classroom programme, a school-wide component and a family involvement component. The intervention aims to help schools become caring communities and to provide opportunities for students to participate in decision- making and actively engage in the social and intellectual life of the school. It aims to influence school curriculum, pedagogy, organisation, management and climate. Teachers are encouraged to develop warm, supportive relationships with students and encourage them to develop similar relationships with each other. Teachers integrate social, ethical and intellectual learning, and connect new knowledge to previous understanding, promoting exploration and problem-solving. This enables diverse students to engage the curriculum in ways most appropriate to their own learning needs and interests. Students gain access to literature and are encouraged to read aloud and discuss. Teachers promote collaborative and co-operative learning. Students work in small groups on projects that require collaboration and have group products. Teachers minimise use of extrinsic incentives and coercive punishment, instead focusing on teaching discipline. Students are involved in creating and maintaining class rules and norms, and given appropriate responsibility for classroom management and decision-making. There are also classroom and school community-building activities, and activities to bridge family and school learning, for example interviewing parents on family heritages including school-wide and school/family events and a school-wide buddy scheme | Training and support by DSC staff to enable district teams to train school staff to deliver the intervention | DSC staff worked with groups of 10–16 site staff in each school district in a 1-year training of trainers, followed by ongoing training in the implementation years. These district implementation teams worked with the DSC staff developers to introduce the programme into schools, with the team members (who included the school principals) taking increasing responsibility for conducting workshops and providing support to programme teachers in their district in successive years. Teachers at the programme schools worked collaboratively in small groups to plan, implement, and reflect on implementation | |
Cooperative Learning | Pacific Northwest/ USA | Middle schools (students aged 11–13 years) | 2 years | The programme followed Johnson et al.’s approach to co-operative learning,206 which can include forms of reciprocal teaching, peer tutoring, jigsaws, collaborative reading and other methods whereby peers help each other learn in small groups. Activities are designed based on the principles of positive interdependence, whereby goals are structured such that individuals can attain their own goals only if their group reaches its goals. This might involve students in projects that require a single deliverable for a group (goal interdependence), students receiving rewards only if everyone reaches a certain threshold on a quiz or test (reward interdependence), each member of a group having different materials they must share to complete an activity (resource interdependence), or each member of a team having a specific role to play (role interdependence) or task to complete (task interdependence) as part of a team. Students may also have their own group identity (identity interdependence). Such activities aim to promote mutual support and assistance instead of competition and isolation among students. The approach also emphasises individual accountability to ensure that each student contributes to outcomes and involves explicit coaching for students in collaborative social skills, a high degree of face-to-face interaction to enable collaboration and guided assessment of group performance to ensure that students improve their group skills over time. Teachers are trained to reinforce the use of positive social skills by observing student interactions during learning activities and recording the number of times students exhibit particular kinds of positive behaviour. Teachers are also expected to discourage any hints of ingroup vs. outgroup bias or prejudice | Developers of intervention and research staff provided training. School staff implemented | Three half-day in-person sessions delivered at different points across the first year of the intervention with periodic check-ins via video conference and access to resources such as newsletters. A 1-day training session was provided for school administrators and a half-day follow-up training session was conducted in the second year of the programme | |
CFS | Australia/Perth | Secondary schools (students aged 12–16 years) | 2 years | The programme enabled staff to implement school organisational strategies to develop the following: student relationships and peer support, revised and implemented policy involving the school community, school ethos, student social and emotional development, positive behaviour management with fewer punitive solutions, and more school–home–community links. Implementation was led by a school project team comprising student leaders and school staff. Schools used a policy and practice audit tool to determine gaps and implement whole-school actions to enhance policies and practices. Students helped to review school policies, teach staff about technologies used by students and provided student and parent cyberbullying prevention training. Six-hour classroom teaching targeted students in grades 8/9, aiming to reduce student harm via addressing online contexts, online contacts, managing privacy, online skills and content accessed via activities such as problem-solving and quizzes | School staff member identified as project co-ordinator, pastoral care staff and grade-10 student cyber leaders implemented the intervention | Grade-8 and -9 teachers received 3-hour training to support classroom activities. Pastoral care teams received 6-hour training to implement whole-school policy and practices, and use the audit tool. Student cyber leaders were trained for 10 hours to lead whole-school activities | |
DASI | Greece and Cyprus | Primary schools | 1 year | The intervention used a whole-school approach to reducing bullying by addressing the school policy for teaching, the school learning environment and school evaluation. Research staff first explained to school stakeholders the approach and rationale. Then data were collected on these school factors and improvement priorities identified for each school. Schools then received support to develop their own strategies and actions to reduce bullying. The research team then worked closely with school stakeholders to help define their strategies and action plans informed by research on bullying and educational effectiveness, and adapt the guidelines that have emerged from the literature in relation to their school context. Then school stakeholders and the research team monitored implementation, using data to refine strategies and plans. Schools were advised that decisions should involve a committee with representatives of parents, students and teachers. Implementation then lasted for approximately 8 months, with support from the research team |
School staff, students and parents, with support from the research team | The research team provided support, but not formal training, to school staff | |
DARE Plus programme | USA/ Minnesota | Middle schools (students aged 11–13 years) and junior high schools (students aged 11–14 years) | 2 years | The intervention included a 10-session curriculum on character- building and citizenship, as well as ‘On the Verge,’ a four-session parental involvement programme with four interactive homework assignments. Parents were sent information every 4–6 weeks. Student-led youth action teams planned and could apply for funds for after-school activities that included campaigns to address drug use and violence. Neighbourhood action teams worked to create safer and healthier school and neighbourhood environments for young adolescents | The curriculum was taught by police officers. Peers delivered the On the Verge curriculum. Newly employed organisers delivered after- school activities. Neighbourhood action teams were composed of adult volunteers | Organisers were trained in facilitating youth action teams, extracurricular activities and neighbourhood actions. Police offers were trained to deliver the curriculum. Officers who were teaching DARE Plus received an extra 2-hour training session by the research team on interactive teaching methods | |
Friendly Schools | Australia/Perth | Primary schools (students aged 6–11 years) | 2 years | The intervention involved whole- school actions to build commitment and capacity to address bullying, family awareness-raising and skills-based self-efficacy activities, and classroom activities for grade-4 and grade-5 students. The whole-school actions involved four or five staff forming a school team that used survey data on student needs and reviewed its school’s current practices, and planned and implemented its school’s policy and other bullying prevention and management activities Family actions involved home activities (linked to and reinforcing classroom learning and raising parents’ awareness, skills and self-efficacy to talk with their children about bullying) plus newsletter items and family involvement in development and communication of the school’s bullying policy. Classroom activities were implemented for around 3 hours at the start of term in each of the first 2 school years to build prosocial skills, including peer discouragement of bullying, social support for individuals being bullied, non-violent conflict resolution and other interpersonal problem-solving skills, and to build empathy for individuals being bullied. These activities also targeted students’ ability to discuss bullying with each other and adults; how to respond appropriately to bullying, including reporting bullying; help-seeking; and responding assertively | School staff supported by unspecified training provider | School teams received 4-hour training in each of the first and second years of the study. Committee members were also trained to manage bullying incidents using the method of shared concern. In the first and second years, intervention classroom teachers received 6 hours of interactive professional development to enhance their comfort, knowledge, skills and confidence to teach the curriculum, and to help them manage student bullying | |
Friendly Schools, Cool Kids Taking Control | Australia/New South Wales and Western Australia | Rapee 2020197 | Primary schools (students aged 2–12 years) | 2 years | Combined Friendly Schools as described above with an indicated intervention for children who were experiencing victimisation and had high levels of anxiety symptoms. Cool Kids Taking Control involved eight lessons released online over 8 weeks consisting of animated videos teaching a new skill or providing information, which were to be completed by the child with a coach, usually the parent. Skills included social skills, challenging unrealistic thoughts, dealing with exposure to feared events, strategies to respond to bullying and coping with being bullied. Coaches were also provided with supplementary information about bullying perpetration, victimisation and anxiety | Online | Leadership teams were invited to attend a 1-day whole-school training session, and the relevant classroom Teachers were invited to a 1-day training session in the curriculum |
FSEF | Australia/Perth | Primary schools (students aged 6–11 years) | 3 years | High- and moderate-intensity versions of the intervention comprised whole-school, classroom, family and individual elements. Whole-school activities aimed to build a positive climate and relations and connectedness between students, school staff and parents; provide effective policies and practices to prevent and manage bullying; and build school capacity support for implementation. Staff modified school social, organisational and physical environments (e.g. availability of student activities during breaktimes) and involved families. Classroom learning targeted all grade levels from 1 to 7 and developed understanding of bullying, building empathy and social skills. Family activities worked in partnership with parents, building awareness, attitudes and self-efficacy to help their children to prevent or respond to bullying. These activities also encouraged parent engagement with the school. Individual-level activities included support for victims. Schools selected five staff and a parent to form a project team that received summaries of aggregated student and parent survey results to support and monitor impacts of implementation | School staff were supported by unspecified training provider to deliver the intervention | All high-/moderate-intensity intervention schools received 2 hours of whole-staff training to prevent and manage bullying, develop students’ social skills and teach the learning activities. Key school staff were trained to use restorative approaches to prevent and manage bullying, including the method of shared concern. High-intensity intervention schools received an additional 3 hours of training each year to increase their capacity to engage parents. In high- and moderate-intensity intervention schools, school teams received 6 hours of training in the first and second years, which included a capacity needs assessment and feedback process to systematically map their current practices and to identify gaps and needs | |
FSTP | Australia/Perth | Cross 2018170 | Primary (students aged 6–11 years) and secondary schools (students aged 12–16 years) | 2 years | The programme included pre-transition booklets for parents and students; capacity-building engagement with primary and secondary school staff; training and resources to support staff (review and implement bullying-related policies and procedures involving the school community, enhance school culture and physical environment, provide positive behaviour management strategies and less punitive-based responses to bullying and strengthen school–home–community links); classroom resources for grade-8 students and newsletter tips for parents; and post-transition resources for school staff to focus on grade-9 student social competence, bystander and advocacy support | School pastoral care staff, classroom teachers | The intervention included a 2-hour training session for all school staff to raise awareness and establish consistent approaches to bullying prevention and management. Six hours of group training for school implementation teams was attended by approximately six senior pastoral care staff from each school to support the implementation of the whole-school, student and parent components. There was also a 1-hour coaching session four times per year with each school’s implementation team, and a 2-hour training session for grade-8 and grade-9 teaching staff delivering the classroom curriculum |
Gatehouse Project | Australia/Victoria | Secondary schools (students aged 12–16 years) | 3 years | A whole-school component involved the establishment and support of a school adolescent health team; the identification of risk and protective factors in each school’s social and learning environment from student surveys; and, using these data, the identification of locally decided strategies to address these issues. The team involved the principal or assistant principal, a curriculum leader, a student welfare co-ordinator or equivalent, relevant year-level teachers and a community representative. This component sought to make changes in the school’s environment to enhance student security, social connectedness and positive regard through valued participation. Intervention elements ranged from establishing an inclusive classroom environment to creating opportunities for student participation in school beyond the classroom. A curriculum component taught social and emotional skills, and began in the second term of the school year, focused initially on students in the second year of secondary school. Lessons were taught over a 10-week period in English and health and/or personal development classes | Centre for Adolescent Health liaison team trained and supported school staff to deliver the intervention | The school liaison team provided professional development and ongoing support for schools during implementation. Team members each worked intensively with two to four schools. Most professional development was done in schools during weekly meetings. An average of 40 hours of professional development was provided by the intervention team to each school during the first and second years, with the time split between a focus on curriculum and whole-school strategies | |
Going Places programme | USA/Maryland | Middle schools (students aged 11–13 years) | 3 years | The programme includes curriculum, parent and school components designed to increase academic engagement and prevent substance use and aggressive behaviours. The curriculum provides skills instruction in problem-solving, self-management, self-control, school involvement, communication and conflict resolution with application to school engagement, smoking, drinking and aggression. The school environment component uses social marketing strategies to improve school climate, establish prosocial norms and a positive school image, reinforce student achievement and apply curriculum skills to the wider school environment. Activities include student rewards, including for applying study skills beyond Going Places lessons, such as special activities, that students earn the right to attend. Parent education aimed to increase parental involvement and expectations regarding academic engagement and problem behaviour via parenting advice, homework requiring parental involvement and newsletters | Teachers with support from research staff | Teachers received a 6-hour training session for each curriculum unit, including 3 days of training in sixth and seventh grade and 1 day in eighth grade. Continuing support was provided to teachers by a master teacher on the research staff. Teacher training emphasised authoritative classroom management techniques in which teachers were encouraged to be demanding and responsive, interactive, use co-operative learning methods and give constructive feedback | |
GST | Uganda/Luwero District | Primary schools (students aged 6–14 years) | 18 months | The toolkit consists of six steps implemented in sequence with > 60 different activities for staff, students and administration, focused on topics such as improving the school compound and creating a better learning environment, respecting and understanding power relationships, improving teaching techniques, creating accountability and learning non-violent methods of discipline. The intervention engages multiple groups within a school (teachers, administration, students and parents) to change ideas and attitudes. Staff, students and administration set school-wide goals, develop action plans, encourage empathy by facilitating reflection on experiences of violence, provide new knowledge on alternative non-violent discipline and provide opportunities to practise new behavioural skills. Staff and student protagonists lead activities with other staff and students in their school, encouraging them to form, lead and join groups to do various intervention activities. Children participate actively and form committees and groups related to different activities. Schools reward successful implementation by creating celebrations. Some student and staff activities improve the school environment by painting murals on school walls, and hanging codes of conduct in visible places | Raising Voices voluntary organisation supports delivery by school staff | There is an inception 2-hour visit to the school by Raising Voices staff to introduce the toolkit, followed by at least two staff ‘protagonists’ being identified and attending a 3-day workshop. Here they develop an action plan for their school. Raising Voices then provides support to these staff protagonists, as well as to at least two key student ‘protagonists’, to implement the plan | |
HSE | UK/SE England | Secondary schools (students aged 11–16 years) | 1 year | Each school undertook a survey of students and received training and financial resources. Schools instituted an action team (including representatives from senior leadership team; head of year or equivalent; teaching, student support and curriculum staff; parents; external staff, e.g. school nurse; and at least two students from different year groups) to review data on student needs derived from surveys, and plan and deliver various actions (some mandatory, some locally determined) to improve student relationships with teachers and other students, security, social support, self-regard and engagement. Mandatory actions included developing agreed rules for appropriate conduct, reviewing policies on bullying and feedback to students, providing one-to-one pastoral care, and facilitating events and displays. Core funding was provided per school plus responsive funding, with bids being judged by the research team | External facilitator supported school staff to convene action team meetings. Training provided by specialist education provider | 10 hours of staff training was delivered to teachers/ classroom assistants on developing inclusive classroom management and improving teacher–student communication | |
Learning Together/ INCLUSIVE | UK/SE England | Secondary schools (students aged 11–16 years) | 3 years | Intervention comprised staff training, survey of student needs, school action groups and a student social and emotional skills curriculum. Schools convened action groups of at least six staff and six students, held twice per term, to revise relevant school policies and co-ordinate the intervention informed by survey data on student needs. School staff delivered primary restorative practices using respectful language to challenge or support behaviour and circle time to build relationships, and secondary restorative practices involved some staff implementing restorative conferences to address more serious behaviour problems. Teachers delivered 5–10 hours per year of lessons on social and emotional skills for students in years 8–10 (ages 12–15 years), aiming to ensure that schools teach not only academic knowledge, but also attend to students’ broader social development | School staff supported in first 2 years by an external facilitator, who was a freelance education consultant with previous secondary school leadership experience | In the first year, all school staff were trained in restorative practices with in-depth training for selected staff from accredited providers over 3 days. For the first 2 years, the groups that attended action group meetings were encouraged to discuss and take action by an external facilitator with school management experience, but in the third year, the group was facilitated by school staff only | |
PPP | USA/Portland | Mitchell 1991193 | Middle schools (students aged 11–13 years) | 2 years | Approximately 25 students are identified in each school as influential, helpful and representative of students. With the support of the school’s co-ordinator, students deliver peer activities across the following areas: orienting new students to the school, providing one-to-one help, peer tutoring, acting as study buddies, social skills tutors, discussion and advisory group leaders, special projects, conflict management, community service and cross-age facilitation. Peer helpers meet regularly to receive training and support. Schools develop action plans to address problems the students identify. Cross-grade element involves middle-school students visiting fifth-grade students each spring to build trust, answer questions and discuss concerns about entering middle school. Programme also involves referral to alcohol and drug programme when needed | The project is run by project directors and project co-ordinators attached to each school, with school staff and students | Five staff members are trained in the programme model and selected grade-6/-7 students attend two half-day training sessions on substance abuse prevention, communication, helping skills, information referral, trust-building and decision-making |
Positive Action | USA/Nevada, Chicago and Hawaii | Elementary schools (students aged 6–11 years) and middle schools (students aged 11–13 years) | Open-ended duration | Curriculum element comprises units on self-concept; positive actions for mind and body; positive social-emotional actions focusing on getting along with others and managing; and being honest with, and continually improving, oneself. A school-wide climate development element undertaken by the principal and a Positive Action co-ordinator/committee (including parent and student representatives) focuses on using curriculum lessons in school-wide teaching practices and other activities to promote and reward further positive actions among students, the school, families and the community. The programme for school-climate change also includes guidance for engaging students in community service, and parent and community-involvement activities. It supports the school counsellor’s role in creating a positive school climate and ways for involving parents and other community members in school and programme activities | Positive Action team train and support school staff to deliver the intervention | Teachers, administrators and support staff attended training sessions with the programme developer, which lasted approximately 3–4 hours for the initial year and 1–2 hours for successive years. Booster sessions by the project co-ordinator were provided at least once per year, which lasted approximately 30–50 minutes. Mini-conferences were held each year for five or six staff per school. These allowed participants to share ideas and experiences and get answers to any questions regarding programme implementation | |
Project PATHE | USA/Charleston | Gottfredson 1986174 | Middle schools (students aged 11–13 years) and high schools (students aged 14–18 years) | 3 years | Schools initiated an organisational structure to enable shared decision-making among community agencies, students, teachers, school administrators and parents in planning for school improvement. The primary aim of the team structure was to involve these groups in designing, planning and carrying out school improvement programmes. Curriculum changes aimed to increase teacher competencies and improve curriculum development and delivery. Achievement test results were used to diagnose weaknesses. School staff used the resulting information to plan and carry out activities to strengthen the academic programme. The programme delivered interventions aimed at improvements in academic performance such as courses on study skills, practice tests, test-taking tips and free reading periods. Student team learning was also used, involving teams of students differing in ability levels, earning rewards for improvement. Interventions to enhance school climate included a school pride campaign, expanded extracurricular activities and peer counselling. These aimed to promote prosocial norms, increase cohesiveness among students and teachers and improve morale. Schools also implemented career exploration programmes introducing students to technical careers, and a job-seeking skills programme, teaching skills for finding and keeping a job. Other activities targeted affective and academic services to marginal students. Students participated in the development of school and classroom rules, and a standard discipline referral procedure was established | School staff supported by specialists. School teams involving staff, students, parents and community members. Two project-funded staff in each school helped organise and monitor the teams | School teams were trained by specialists in assessing needs, researching problems, defining objectives, developing and implementing plans, assessing progress and redesigning strategies. Teachers received training in innovative teaching techniques including student team learning. The specialists served as resources for teachers in developing and implementing study skills units for their classes. Training also addressed teacher competencies in classroom management and school-wide discipline management |
Responsive Classroom | USA | Anyon 2016189 | Elementary schools (students aged 6–11 years) and middle schools (students aged 11–13 years) | 1 year | The intervention involved three practices: morning meeting (teachers and students participate in community-building exercises that also target academic skills; teacher language (reinforcing positive behaviours, questions and interactive modelling to remind students of rules and expectations developed by the students, redirecting students with direct, specific, and firm statements of instruction); and logical consequences (non-punitive responses to persistent misbehaviour usually involving reparation) | School staff supported by specialists | All staff received 2 days of professional development at a summer retreat and five booster sessions every other month to address challenges and support adoption. On-site coaching was provided by the behaviour specialist as needed. Regular consultation was also provided to school staff by other members of the implementation team at least twice a month via in-person meetings, e-mail and Google Docs (Google Inc.) |
Restorative Practices Intervention | USA/Maine | Acosta 2019161 | Middle schools (students aged 11–13 years) | 2 years | Staff use of affective statements, and restorative circles and conferences. Circles can establish ground rules or address inappropriate behaviour. Conferences are meetings that can address low-level or more serious or ongoing conflict. School staff use restorative practices to build relationships and resolve problems, and to interact with parents | Training and support provided by the IIRP coaches to enable school staff to deliver intervention | All school staff trained (4 days over 2 years) to deliver restorative practices ranging from using ‘affective’ statements to hosting a restorative ‘circle’ or conference’ Staff join monthly participatory learning groups of 8–12 individuals on restorative practices. Monthly consultation telephone calls to support implementation. Optional training enables school administrators to train new staff on the intervention. IIRP coaches visit campuses twice per year to troubleshoot on site. Guidelines and sample plans provided to steer circles and conferences |
SEHER programme | India/Bihar | Middle schools (students aged 11–13 years) | 2 years | Whole-school activities addressed the following: hygiene, bullying, mental health, substance use, reproductive and sexual health, gender and violence, rights and responsibilities, and study skills. Each school formed a School Health Promotion Committee, comprising management, parents, teachers and students, with support from an intervention lead (lay counsellor or teacher, depending on trial arm). This committee managed and monitored the programme and discussed issues raised by students via a speak-out letterbox. The lead facilitated assemblies and other activities, a monthly wall magazine, competitions for students, new policies on zero tolerance to bullying and antisubstance use developed with staff and students in the School Health Promotion Committee meeting. Students from each class were elected to form a peer group meeting monthly facilitated by the intervention lead to discuss the topic of the month and student concerns, develop a plan of action and assist in organising the abovementioned activities. The intervention lead ran a workshop on effective study skills for students and supportive and non-exclusionary disciplinary practices for teachers. The intervention lead also provided counselling or referrals for students in need | School staff supported by a graduate-level lay counsellor or existing teacher, depending on trial arm | Intervention leads received 1 week of training, followed up with in-service training through monthly group meetings and three planned individual visits per month | |
Whole-of-school intervention | Australia/New South Wales | Secondary schools (students aged 12–16 years) | 2 years | Schools identified a project co-ordinator and a school intervention team who engaged in a structured planning process to prioritise and select appropriate resources/programmes based on a needs assessment survey of students and school community planning workshops and one strategy review workshop. Schools were provided with details of existing resources and programmes addressing curriculum, teaching and learning, ethos and environment (rewards and recognition, peer support/ mentoring, antibullying, student empowerment/leadership and aboriginal culture/leadership, partnerships and services). Schools could choose which to implement in each of these areas. Schools delivered 9 hours of lessons on individual protective factors across school subjects. Schools engaged Aboriginal and other local community organisations in school, and developed referral pathways to health, community and youth services. Schools developed strategies to increase parental involvement in school (e.g. school events) and provided information on student protective factors to parents via school newsletters | School staff supported by an embedded psychology or education-trained implementation support officer | Support from the implementation support officer employed 1 day per week. Staff also received 1 hour of mental health training |
Appendix 3 Descriptions of theories of change
Intervention | Study | Summary of theory of change | Summary of contextual contingencies | Existing theories drawn from |
---|---|---|---|---|
AAYP school/ community intervention | The curriculum aimed to teach cognitive-behavioural skills to build self-esteem and empathy, manage stress and anxiety, develop interpersonal relationships, resist peer pressure, and develop decision-making, problem-solving, conflict resolution, self-efficacy and goal-setting skills, and application of these skills, to resist peer pressure and negotiate interpersonal relationships to avoid violence, provocative behaviour, school delinquency, drug use and unsafe sexual behaviours. The interventions aimed to promote unity, self-determination and responsibility using culturally based teaching methods, such as use of storytelling and proverbs, and reference to African and African American history and literature. The homework assignments involve parents to encourage parent–child communication and reinforcement of skills learnt, and to expand the target of the intervention to parents. The climate element aimed to generalise learnt skills into the broader school environment, while the community element aimed to develop linkages among parents, schools and local businesses. The school climate and community components aimed to ‘rebuild the village’ and create a ‘sense of ownership’ by all stakeholders | The intervention theory of change was intended to align with African American cultural values of unity, self-determination and collective responsibility, and so be particularly effective for this population | The theory of triadic influence | |
CDP | The intervention aimed to build a caring school environment by building stable, warm and supportive relationships (via various community-building activities in the classroom and the wider school, such as social events and interyear buddy schemes); an explicit shared commitment to values of caring, justice, responsibility and learning (via teaching and modelling); ensuring student autonomy, influence and self-direction; using collaborative learning and shared decision-making (e.g. developing rules and learning developmental discipline rather than merely being subject to coercive discipline); providing an accessible, relevant and engaging curriculum; and using a ‘constructivist’ approach to teaching and learning. This provides; attention to social, ethical and intellectual learning; teaching for understanding (connecting to existing student knowledge via exploration and problem-solving); meaningful, challenging, learner-centred curriculum (with access to good literature) in which students experience success academically and socially; and fostering intrinsic motivation (minimising extrinsic rewards and punishments) and co-operative learning (setting social as well as academic learning goals and exploring each other’s ‘constructions’ of knowledge). These are achieved through pedagogical strategies and curriculum materials. In addition, home activities aim to promote communication and understanding between child and parent. Through these elements, the intervention was theorised to facilitate students’ academic, intellectual and sociomoral development, including knowledge of subject matter, conceptual understanding, reasoning and thinking skills, social competencies and interpersonal understanding, as well as to satisfy students’ basic psychological needs for autonomy, competence and belonging to a social group whose members are mutually supportive and concerned. Satisfaction of students’ basic needs was theorised to increase their attachment or bonding to the school community, which, in turn, promotes commitment to and internalisation of the community’s salient norms and values (including those proscribing drug use and violence and those endorsing personal responsibility and concern for others), and behaviour consistent with them, and therefore reduced involvement in drug use and other such behaviours | Constructivist learning, connecting new to existing knowledge, and learner-centred teaching are both theorised to be particularly appropriate to diverse student groups | Attachment theory | |
Cooperative Learning | In contrast to didactic and individualistic approaches to instruction, Cooperative Learning uses group-based peer learning activities that encourage interpersonal contact and mutual support between diverse (preferably randomly assigned) groups of students. The approach is underpinned by the principles of positive interdependence, whereby individual goals are aligned with the goals of the group such that individual success can be realised only via group success. Under positive interdependence, peers come to promote the success of one another through mutual assistance and support and the sharing of information and resources. Mutual disclosure to assist in discovering areas of commonality is encouraged and those in positions of authority (i.e. teachers) are expected to discourage any hints of ingroup or outgroup bias or prejudice and reinforce positive, helpful behaviour. Informed by contact theory, interaction in such collaborative, group peer-based learning activities is theorised to lead to the reduction of competition, bias, prejudices and exclusion among students. As such these processes are theorised to promote improvements in social integration and interpersonal relationships, leading to enhanced empathy (both cognitive and affective) among peers, thus preventing bullying behaviour. The positive feelings that arise from collaborative and supportive interactions to achieve goals also results in a ‘benign spiral’ that further increases positive social interactions and enhances peer relations. Bringing a broad cross-section of students together in these group-based peer learning activities also provides opportunities to break down the process of homophily among bullies and for socially marginalised at-risk youth to develop relationships and socially integrate with more prosocial peers, thus also interrupting the process of deviant peer clustering and reducing bullying behaviour and substance use | Theorised to be particularly effective for more marginalised youth at risk of substance use | Contact theory | |
CFS | Rather than promoting only punitive-based solutions to cyberbullying, the intervention emphasised raising awareness among students, parents and educators of the harms associated with the misuse of technology, providing strategies to minimise this harm, and delivering opportunities for students to experience, learn and benefit from new communications technology. The intervention aimed to reduce cyberbullying by fostering positive behaviour among adolescents using a harm minimisation approach, recognising that being online is a ubiquitous phenomenon among adolescents, and, rather than aiming to eradicate its use, focusing on educating young people about the potential risks, thereby reducing the potential hazards associated with such technology. The intervention enabled staff to implement strategies relating to their school’s organisational context to build a consistent understanding of cyberbullying, with strategies to develop students’ social relationships and peer support; policy and its implementation involving the school community; school ethos; student social and emotional development; positive behaviour management with fewer punitive solutions; and more school-home-community links. Classroom teaching aimed to enhance online social skills, with strong emphasis on positive communication, resilience, self-management, conflict resolution and social responsibility. The intervention addressed student-level mediators such as involvement in offline bullying perpetration, empathic responsiveness, moral disengagement, attitudes, normative beliefs, attachment to school and friends, peer support, emotional regulation, social competence and perceived efficacy to deal with cyberbullying. It also addressed school organisational factors, school ethos and social climate, and policy and practice, as well as parent–child–staff relationships, and parental monitoring and responses to cyberbullying | Ecological systems theory | ||
DARE Plus programme | Violence-specific goals of the DARE Plus intervention included decreasing intentions to be violent, providing positive role models to create and support school and community violence-free norms, supporting reasons for being violence free (outcome expectancies) and negative outcome expectations related to violence, developing a safe and supportive home environment for young adolescents by increasing family rules and communication about violence, and increasing social skills such as communication and decision-making skills. The community-organising element aimed to develop and implement community-determined strategies to address environmental conditions that affect students’ access and exposure to alcohol, tobacco, marijuana and violence. The youth-organising component aimed to plan and implement safe, supervised and drug-and violence-free extracurricular activities. These organising elements aimed to develop empowerment and ownership of the intervention within communities | – |
|
|
DASI | The whole school aimed to reduce bullying by developing a school policy for teaching, building a safe and positive school learning environment and developing school self-evaluation. These are theorised to improve learning outcomes and reduce bullying. School policy may encourage teachers to be visible and vigilant in common areas. Increased monitoring of student behaviour during recesses and before the beginning of lessons can help school stakeholders to identify and intervene when bullying occurs. Including staff, students and parents in the creation and implementation of antibullying policies, the school management team may receive valuable input from all those directly affected, and the active involvement of these stakeholders in defining the policy, strategies and action plans to face bullying may encourage their active participation in implementing these action plans for improvement purposes. Improving the school learning environment may affect the development of positive and desirable behaviours, which can be characterised as respectful, reliable and responsible, which are negatively associated with bullying. Rewarding good behaviour inside and outside the classroom can improve the school’s social environment by emphasising maintenance of the behaviour code and the promotion of appropriate and positive behaviours outside the classroom, thus reducing bullying. Classroom teaching may involve teachers introducing relevant cognitive and affective aims targeted at reducing bullying (e.g. development of social cognition, understanding of social values, emotional recognition and development of positive attitudes towards peers). Effective schools develop evaluation mechanisms to investigate whether or not their strategies and actions for reducing bullying are effective | – | – | |
Friendly Schools | The intervention is theorised to modify student attitudes (ensuring that these are pro-victim and antibully with a negative outcome expectancy of bullying), perceptions of social support (increasing peer and teacher support, empathy, social competence and number of reciprocated friends), knowledge of bullying and school connectedness and adjustment (increasing happiness at school, decreasing absenteeism and loneliness at school, and increasing liking school, perceived safety at school and expectation that teachers will take action on bullying). In turn these are theorised to reduce experienced and observed bullying behaviours (victimisation, perpetration) and improve psychological health (reduced depression, reduced anxiety, improved peer relations and enhanced general self-concept). The intervention aims to do this by taking a multicomponent, systems-based approach, involving the development of whole-school strategies addressing the school’s social climate, group mechanisms of bullying, normative social influence, social support, empathy and outcome expectancies. Home activities aimed to reinforce and practise classroom learning and to raise parents’ awareness, utility knowledge, skills and self-efficacy to talk with their children about bullying, and help their children prevent and manage bullying. Parent involvement was also encouraged via review and dissemination of the school bullying policy. Classroom learning activities aimed to build prosocial skills, including peer discouragement of bullying, social support for individuals being bullied, non-violent conflict resolution and other interpersonal problem-solving skills, and to build empathy for individuals being bullied. They also aimed to enhance students’ understanding of what constitutes bullying and how to respond to bullying, and why bullying is unacceptable behaviour. The activities also targeted students’ ability to talk about bullying with each other and adults; how to respond adaptively to bullying, including reporting bullying; seeking support; and responding assertively | – |
|
|
Friendly Schools and Cool Kids Taking Control | Rapee 2020197 | This intervention combined Friendly Schools, described above, with an indicated intervention, Cool Kids Taking Control, which was designed for children experiencing victimisation and simultaneously high levels of anxiety symptoms. Cool Kids Taking Control aimed to teach social and emotional skills that would reduce the likelihood that potential victims will be targeted by bullies, and improve their mechanisms for coping with victimisation. A set of eight online lessons are completed by students and a coach (usually a parent), who receives supplementary information on bullying perpetration, victimisation and anxiety to help them support the student. As a combined intervention, it aims to address both external (e.g. school normative environment) and internal (e.g. social isolation, unassertive response to victimisation) risk factors for bullying |
|
|
FSFF | The intervention used a whole-school, system-based approach addressing ecological, cognitive and psychosocial risk and protective factors that are potentially amenable to change, and that can be regulated or mediated at the school, classroom, family and/or individual levels to reduce bullying. The whole-school-level activities aimed to build a positive social climate, build positive relations and connectedness between students, school staff and parents; provide effective policies and common understanding and practices to prevent and effectively manage and reduce current bullying; and build school capacity support for implementation through assessment of organisational structures, resources, skills and commitment levels. These outcomes were addressed using detailed whole-school support materials and assessment tools and staff training that suggested modifications to the school’s social, organisational and physical environment (e.g. enhancing supervision levels, availability of student activities during breaktimes, tracking bullying ‘hotspots’) and through the involvement of students’ families. School-level mediators are theorised to be school leadership, school policy and associated practices, school capacity to implement strategies to reduce bullying, physical environment, social environment, staff management of incidents and school contact with parents. Classroom activities aimed to complement students’ other social and emotional learning, focusing on the reciprocal relationship between students who observe bullying, those who are bullied or bully others and their social environment. The learning activities developed common understandings about the nature of bullying, its effects and how it can be discouraged, while also addressing empathy and social skill building. The activities helped teachers to enhance the positive interactions they have with students using role play, stories, role modelling, skills training and observational learning. Student-level mediators are theorised to be understanding about bullying, attitudes to bullying, self-efficacy to deal with bullying, social competence, peer support, reciprocated friendships, social norms and normative expectations (peer attitudes to bullying and peer responses to bullying) and connectedness to school. Staff-level mediators are theorised to be understanding about bullying, attitudes to bullying and responses to bullying. Classroom-level mediators are theorised to be behaviour management, interactive group-based teaching methods and teaching and learning content about bullying. The individual-level activities included selective and indicated activities to support victimised students; to help modify the behaviour of students who bully others; and to facilitate school links with local health professionals with specialist skills. Family-level activities worked in partnership with parents by building their awareness, attitudes and self-efficacy to role model and help their children to develop social competence and to prevent or respond to bullying. These activities also encouraged school and parent communication and parents’ engagement with the school to reduce student bullying. Family-level mediators were parents’ understandings about bullying, attitudes to bullying and self-efficacy to help children with bullying, and reduced sibling bullying. In turn, these mechanisms were theorised to increase telling when bullied, and reduce bullying victimisation and perpetration. They are also theorised to improve student mental and physical health outcomes in terms of depression, anxiety, general self-concept and somatic symptoms, and to improve student attendance and academic achievement |
It is theorised that student age and gender and school size and socioeconomic status will moderate effects | – | |
FSTP | Cross 2018170 | The intervention aimed to build student advocacy, coping with bullying, emotional regulation, assertive responses to bullying and help-seeking, and to provide support to others. The whole-school element aimed to help schools enhance student peer support and social relationships, review bullying-related policies and processes, and their implementation (involving the school community), enhance physical environment and school culture, provide positive-behaviour strategies, reduce punitive-based responses to bullying and strengthen school–parent–community links. The parent component provided parenting tips to promote children’s social competence and minimise harm from bullying, help parents understand their children’s new school, prepare children for these changes and help children maintain friendships and make new friends | – |
|
Gatehouse Project | The intervention aimed to promote change in the social and learning environments of the school to promote a sense of social inclusion and connection, as well as promoting change at an individual level to build students’ sense of attachment, security and trust, increase student skills and opportunities for good communication and build students’ sense of positive regard through valued participation in aspects of school. The conceptual framework of the project emphasises the importance of healthy attachments or a sense of positive connection with teachers and peers. The strategy seeks to make changes in the schools’ social and learning environments, to introduce relevant and important skills through the curriculum and to strengthen the structures within the school that promote links between the school and its community. The individual component focused on cognitive and interpersonal skills underlying emotional well-being relevant to normal developmental experiences of teenagers, such as dealing with common challenges and stressors | – | Attachment theory | |
Going Places programme | Simons-Morton 2005182,183 | The programme sought to provide positive and reciprocal person-environment influences during the critical period of early adolescent development known as the adolescent transition. The focus of the curriculum was problem-solving, self-control, communication and conflict resolution skills. It aimed to influence perceptions, attitudes and expectations about substance use and antisocial behaviour and to improve self-efficacy, social skills and competence at a time when adolescents are rapidly developing problem-solving and social skills in a context of increasingly important peer influences. Social influences, attitudes, skills and competence are dynamic and reciprocal, with peers influencing attitudes and behaviour, and behaviour influencing peer selection and attitudes. Adolescents with well-developed social skills may be better able to select friends, negotiate difficult social situations, balance social demands with social responsibilities, and maintain control over their behaviour. Efficacy expectations reflect confidence in one’s ability to perform certain tasks or behaviours. Outcome expectations are the expected social and personal consequences of anticipated behaviour. Enhancing social skills and competence can improve decision-making, self-efficacy, prosocial attitudes and affiliation with prosocial peers. It can also improve school engagement and commitment, increase the potential for prosocial friendship development, alter attitudes and perceptions about problem behaviour, prevent substance abuse, and reduce adjustment and mental health problems. The school environment aimed to improve school climate, establish realistic, prosocial norms, establish a positive image for the school, reinforce student achievement and extend curriculum concepts to the larger school environment. The parent component aimed to increase parental monitoring, involvement and expectations regarding academic engagement and problem behaviour | – |
|
GST | The intervention took a whole-school approach to violence prevention and aims to change culture at the school level to promote respect, participation and reflection on concepts and exercise of power. The intervention aimed to enable schools to set goals, make action plans and provide social support to students. It encouraged staff and students to develop empathy by facilitating reflection on experiences of violence, provided staff with new knowledge on alternative non-violent discipline, improved teaching techniques and provided opportunities to staff and students to practise new behavioural skills. Schools were encouraged to self-monitor their progress according to their action plans. Reinforcement of new information and ideas, feedback on progress, and modelling of new techniques and behaviours was provided by visits from the agency supporting the intervention and within school by trained staff. Schools rewarded successful achievement of group goals and action plan deliverables by creating celebrations. Social support for behavioural change occurred through the intervention engaging multiple groups within a school to change ideas and attitudes | – | Transtheoretical model of behavioural change | |
HSE | The intervention aimed to enhance student social ties and pride, security, communication and self-regard The intervention aimed to increase student and parent engagement and pride in school to reduce the number of disengaged students who use substances as alternative status markers. The intervention aimed to render schools safer to reduce the number of students using substances to facilitate protective friendships with substance-using peers. The intervention aimed to ensure that schools promote student communication and self-regard and provide more support to students to reduce the number of students who use substances to self-medicate for anxiety | The theory of change implies that the intervention mechanisms will particularly benefit schools where more students report disengagement, lack of safety and anxiety | – | |
Learning Together/ INCLUSIVE | These intervention components were theorised to work synergistically: distribute decision-making authority across the school; strengthen communication and trusting, empathetic and warm relationships between and among staff and students; reorient learning and teaching, discipline, social support, management and organisation to centre on student needs and provide life skills (and for discipline to focus on opportunities for learning through restorative practices); and integrate students’ academic learning and broader development, including social and emotional learning. These were theorised to transform the whole-school climate and improve staff-student relationships, student engagement with the instructional order (learning) and regulatory order (sense of belonging and participation in the school community and acceptance of school prosocial norms) and reduce engagement with antischool peer groups. Along with improved social and emotional skills, these would enable young people to choose healthier behaviours and develop better quality of life, psychological functioning and mental well-being by promoting their autonomy, motivation and reasoning ability, and disengagement from risk behaviours (violence and bullying, substance use and sexual risk behaviours, truancy and school exclusion), thereby reducing use of NHS and contact with the police | The mechanisms will generate more impact among students of low socioeconomic status or engagement with schools who are less likely to engage with school as a default | Theory of human functioning and school organisation | |
PPP | Mitchell 1991193 | The intervention aims to build: positive peer influence, peer helper and communication strategies; cross-age teaching activities on drug-specific and other topics to link middle school peer helpers to fourth and fifth grade students; and increased parent involvement. Peer management involving mediation, conflict management and social skills tutoring is expected to have a positive influence on social behaviour. Cross-age tutoring involving academic tutors and study buddies is expected to have a positive influence on academic engagement. Intervention activities aim to promote student empowerment through involvement in meaningful activities, acquisition of information and skills, and recognition of student responsibilities. By utilising multiple peer influences, communication strategies and cross-age peer teaching, the project addressed a number of mediators of substance abuse such as academic failure, lack of bonding to school, alienation, impaired confidence and self-esteem | – | – |
Positive Action | The intervention encourages youth to engage in prosocial behaviours, to think in a positive manner and to appreciate themselves for their positive actions; in this regard, the programme aims to increase adolescents’ self-esteem. Engaging in prosocial actions is expected to positively affect one’s thoughts and feelings, especially with respect to self-appraisal. By explicitly linking thoughts, feelings and actions, the programme is theorised to enhance the development and integration of affective and cognitive brain functions. Youth who have high levels of self-esteem are theorised to continue behaving in a positive and prosocial manner and to refrain from engaging in negative behaviours, such as aggression. Furthermore, as a youth’s self-esteem is continuously bolstered through the ongoing intervention and continued engagement in positive actions, this high self-esteem is theorised to combat poor mental health, such as symptoms of anxiety and depression. The curriculum was designed to first introduce the overarching goal of positive self-appraisal (i.e. feeling good about self), and then increase specific skills needed to achieve this goal. The theory of change guiding the classroom curriculum component can be divided into immediate outcomes, short-term effects (mediators), and long-term impact. The classroom curriculum is hypothesised to improve teacher–student relations, student–student relations, student–parent relations and student engagement with the community. In addition to improving the learning environment, short-term programme goals include improved self-concept; enhanced study/thinking skills; increased motivation to learn; improved health habits; effective self-management; increased interpersonal/social/emotional skills; increased self-honesty; and improved skill levels in goal-setting, problem-solving and decision-making. These short-term programme goals are hypothesised to translate into improved school attendance, improved grades, higher test scores, fewer disciplinary problems, reduced substance use, and less aggression and violence, resulting in overall improvement in the school culture. If all youth in a middle school are exposed to the intervention, the entire culture of the school is theorised to improve, causing youth to perceive the overall school environment more positively (e.g. decreased perception of physical and verbal aggression). Thus, the intervention aims to improve the culture of a school by making positive behaviours the norm, which is theorised to decrease bullying. The intervention also addresses more upstream influences on behaviour with a holistic approach to school reorganisation, teacher-student relations, parent involvement, instructional practices, and development of the self-concept of students, teachers and parents. | – |
|
|
Project PATHE | Gottfredson 1986174 | The intervention targeted several aspects of the school environment, attempting to ameliorate academic and social deficits of delinquency-prone students and to alter environmental cues, rewards and punishments, and structural arrangements in the school in ways intended to make undesired behaviour less likely to occur. The intervention aimed to develop student involvement in constructive learning activities and provide consistent rewards for successful participation. It sought to create a climate of mutual respect and co-operation and a sense of belonging among teachers, administrators and students; to increase effective communication; to increase student and faculty involvement in planning for and implementing school change efforts; to increase the clarity, fairness and consistency of school rule enforcement; and to increase teachers’ classroom management skills. The programme sought to reduce school disorder by decreasing academic failure experiences among students in the participating schools; increasing social bonding, including attachment to the people in the school and caring about what others think about oneself; increasing involvement in conventional activities, and commitment of time and energy to the enterprise of schooling; and improving students’ self-concepts. It was theorised that this would foster attachment to and sense of belonging to school and prosocial peers, increasing social competence and sense of usefulness, and giving students something to lose if they misbehave. This is theorised to restrain youths from engaging in delinquent activities and increase academic achievement | Mechanism works particularly for delinquency-prone students |
|
Responsive Classroom | Anyon 2016189 | The intervention aimed to manage student problem behaviours using student-centred strategies. Morning Meetings aim to build sense of community and academic skills. Modifying teacher language aimed to reinforce positive behaviours. Teachers used questions and modelling to reinforce behavioural norms. Discipline was reoriented from punitive to restorative approaches focused on reflecting on harms and reparations | – | – |
Restorative Practices Intervention | Acosta 2019161 | The intervention is theorised to work through three psychological mechanisms: maximising positive affect through proactive practices such as restorative circles, which are aimed at developing closer bonds and relationships among youth; minimising negative affect by providing responsive practices that ensure that offenders can take public responsibility for their behaviour and reintegrate into normal community life; and encouraging free expression of emotion through practices such as affective statements and questions. The intervention primarily focuses on the peer and school systems at the micro- and meso-system levels. At the student level, it helps to build relationships with peers and teachers and for students to have a more active voice in responding to specific school incidents. The intervention is theorised to sustain relationships with adults by creating positive and sustained adult–youth relationships through teacher–student dialogue that occurs in circles. The intervention builds skills by using teachers and other school staff to coach students, ensuring that these skills are applied in running the circles for students. At the school level, when students have positive interactions with their peers, this improves their peer relationships and can lead to more active participation in school, which, in turn, promotes the positive environment of their school. These changes in the peer and school environments can have a reciprocal positive influence on student development and that, in turn, can promote the three individual psychological mechanisms listed previously. This results in continually reinforcing and reciprocal positive individual, peer and school cultures that effectively regulate and consistently build and repair social bonds, connectedness and relationships. These are key mechanisms to prevent high-risk and harmful behaviours | – |
|
SEHER programme | The intervention aims to build a positive school climate. The school climate is theorised to comprise student perceptions or experiences of the school’s social environment, which encompasses safety (social, emotional and physical), teaching and learning (academic climate) and relationships (community climate). Physical safety refers to the degree to which violence, aggression and physical bullying are present; emotional safety includes availability of school-based mental health services and support; and social safety refers to the sense of feeling safe with other people. The teaching and learning climate refers to actual methods and instructional practices used by the teachers in the classrooms and how teachers communicate their expectations and give feedback to students, and student commitment to academic values. The community aspect of school climate refers to the quality of relationships within a school (i.e. relationships between teachers, students and administrators) and includes sense of school connectedness, participation and belonging, respect for diversity, and partnership with other members of the community. The intervention addresses the following areas to achieve this: promoting social skills among adolescents; engaging adolescents, teachers and parents in school-level decision-making processes; providing access to factual knowledge about health and risk behaviours to the school community; and enhancing problem-solving skills among adolescents. The intervention strategies were organised at three levels: whole school, group and individual | – | – | |
Whole-of-school intervention | The intervention involved various strategies across the domains of: school ethos and environment (e.g. rewards and recognition, peer support, empowerment/leadership programmes); curriculum, teaching and learning (age-appropriate health lessons); and community partnerships and services (e.g. engagement with community organisations and services, increased parental information and involvement). Strategies addressed individual factors (self-efficacy, problem-solving, co-operation/communication, self-awareness, empathy, goals/aspirations) or school environmental factors (school support, school meaningful participation) or community factors (community support, community meaningful participation, home support, home meaningful participation) and factors in multiple domains (peer caring relationships, prosocial peers). Addressing such factors was theorised to reduce adolescent substance use and increase resilience | – | – |
Appendix 4 Quality assessment of theories of change
Intervention name | Study | Describes pathway from intervention to outcomes | Concepts/constructs are clear | Inter-relations between concepts/constructs are clear | Mechanisms are explained | Describes how mechanisms vary by place or person | Overall quality after reconciling/5 | Quality assessment | |||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CB | RP | CB | RP | CB | RP | CB | RP | CB | RP | ||||
AAYP school/community | Flay 200487 | Yes | Yes | Noa | Yes | Noa | Yes | No | No | Yes | Nob | 4 | High |
intervention | Jagers 2009177 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
CDP | Battistich 1996164 | Yes | Yes | No | No | No | No | No | No | No | No | 1 | Low |
Battistich 2000162 | No | No | Noa | Yes | Noa | Yes | Noa | Yes | No | No | 3 | Medium | |
Solomon 2000198 | No | No | Yes | Yes | Yes | Nob | No | No | No | No | 2 | Medium | |
Battistich 2004163 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Cooperative Learning | Van Ryzin 2018185 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High |
Van Ryzin 2018186 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
Van Ryzin 2019187 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
Van Ryzin 2019188 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
CFS | Cross 2016199 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | No | 3 | Medium |
Cross 2018191 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | No | 3 | Medium | |
DASI | Kyriakides 2013178 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High |
Kyriakides 2014200 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
DARE Plus Project | Perry 2003117 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
Komro 2004103 | Noa | Yes | Noa | Yes | No | No | No | No | No | No | 2 | Medium | |
Bosma 2005190 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Friendly Schools | Cross 2011169 | Yes | Yes | Yes | Yes | No | No | No | No | No | No | 2 | Medium |
Rapee 2020197 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Friendly Schools, Cool Kids Taking Control | Rapee 2020197 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
FSFF | Cross 2012171 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | No | 3 | Medium |
Cross 2018192 | Yes | Yes | Yes | Yes | Yes | Nob | Yes | Nob | No | No | 4 | High | |
FSTP | Cross 2018170 | Yes | Nob | Yes | Yes | Yes | Nob | No | No | No | No | 3 | Medium |
Gatehouse Project | Bond 200166 | No | No | Noa | Yes | No | Yesb | Noa | Yes | No | No | 2 | Medium |
Bond 200467 | No | No | Yes | Yes | No | No | No | No | No | No | 1 | Low | |
Bond 200468 | No | No | Noa | Yes | No | Yesb | No | No | No | No | 1 | Low | |
Patton 2006115 | Yes | Yes | No | No | No | No | No | No | No | No | 1 | Low | |
Going Places programme | Simmons-Morten 2005182 | Yes | Nob | Yes | Nob | Yes | Nob | No | No | No | No | 3 | Medium |
Simmons-Morten 2005183 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
GST | Devries 2015172 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
Devries 201780 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Knight 201898 | No | Yesb | No | No | No | No | No | No | No | No | 0 | Low | |
HSE | Bonell 201073 | No | Yesb | No | Yesb | No | Yesb | No | Yesb | No | No | 0 | Low |
Bonell 201069 | No | Yesb | No | Yesb | No | Yesb | No | No | No | No | 0 | Low | |
Learning Together | Bonell 201571 | Yes | Yes | Yes | Yes | Yes | Yes | Noa | Yes | No | No | 4 | High |
Fletcher 201588 | No | Yesb | No | Yesb | No | Yesb | No | Yesb | No | No | 0 | Low | |
Bonell 2018166 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 5 | High | |
Bonell 2019165 | Yes | Yes | Yes | Yes | Yes | Yes | Noa | Yes | No | No | 4 | High | |
Bonell 2019167 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
Warren 2019194 | Yes | Yes | Yes | Yes | Yes | Yes | Noa | Yes | No | No | 4 | High | |
BoneII 2020168 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
Warren 2020195 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High | |
PPP | Mitchell 1991193 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
Positive Action | Flay 200185 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High |
Flay 2003173 | No | No | Noa | Yes | No | No | No | Yesb | No | No | 1 | Low | |
Beets 200964 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Beets 200863 | No | No | Noa | Yes | No | Yesb | No | No | No | No | 1 | Low | |
O’Hare 2010113 | Yes | Yes | Yes | Yes | Yes | Yes | No | Yesb | No | No | 3 | Medium | |
Snyder 2010123 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Li 2011109 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Lewis 201252 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Bavarian 201361 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Lewis 2013108 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Snyder 2013124 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Malloy 2015110 | No | No | No | No | No | No | No | No | No | No | 0 | Low | |
Lewis 2016179 | No | No | No | No | No | No | No | Yesb | No | No | 0 | Low | |
Project PATHE | Gottfredson 1996174 | Yes | Nob | Yes | Nob | Yes | Nob | No | No | No | No | 3 | Medium |
Responsive Classroom | Anyon 2016189 | Yes | Yes | No | No | No | No | No | No | No | No | 1 | Low |
Restorative Practices Intervention | Acosta 2019161 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | 4 | High |
SEHER programme in the state of Bihar, India | Shinde 2018181 | Yes | Nob | Yes | Nob | No | No | No | No | No | No | 2 | Medium |
Shinde 2020180 | No | No | Yes | Yes | No | No | No | No | No | No | 1 | Low | |
Singla 2021184 | No | No | Yes | Yes | No | No | No | No | No | No | 1 | Low | |
Whole-of-school | Hodder 2017175 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
intervention based universal (school resilience – intervention) | Hodder 2018176 | No | No | No | No | No | No | No | No | No | No | 0 | Low |
Agreement between CB and RP, n/N (%) | 54/63 (86) | 50/63 (79) | 50/63 (79) | 52/63 (83) | 62/63 (98) | Overall: 85% agreement | |||||||
Reconciliation, n/N (%) | |||||||||||||
Reconciled in favour CB | 8/9 (89) | 6/13 (46) | 11/13 (85) | 6/11 (56) | 1/1 (100) | 32/47 (68) | |||||||
Reconciled in favour of RP | 1/9 (11) | 7/13 (54) | 2/13 (15) | 5/11 (45) | – | 15/47 (32) |
Appendix 5 Characteristics and quality assessment of process evaluations
Characteristic/question | Answer |
---|---|
Intervention name | CFS |
Study report | Cross 2018191 |
Study location | Australia/Perth |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | Students acting as cyberleaders were surveyed each year of the 2-year intervention about their proposed activities and confidence in acting as a cyberleader. Cyberleaders also completed survey evaluations of the training activities at the end of the workshop and a brief telephone interview to measure their progress and any barriers to activity implementation at the end of each year |
School project teams completed a baseline survey during the training workshop they attended and evaluations of training activities. Teaching staff completed baseline surveys prior to curriculum training and follow-up surveys a year later | |
Project co-ordinators in each school were interviewed about their whole-school planning and activities at baseline and at post tests in 2011 and 2012 | |
Methods of data analysis | Not stated |
Evaluation participants | Students and school staff |
Sample size of participants | Cyberleaders completing surveys during training, n = 87; n = 138 at post-test 1 and n = 48 at post-test 2. Teachers completing baseline surveys, n = 51. Teachers completing follow-up surveys at post-test 1, n = 78. Project co-ordinators completing interviews at baseline, n = 28. Project co-ordinator interviews at post-test 1, n = 25, and at post-test 2, n = 33 |
Sociodemographic characteristics of participants | Not stated |
Were steps taken to minimise bias and error/increase rigour in sampling? | No, none specified. Recruitment and sampling of participants for process evaluation not discussed |
Were steps taken to minimise bias and error/increase rigour in data collection? | No, none clearly specified. Reference to use of a standardised protocol for project co-ordinator interviews |
Were steps taken to minimise bias and error/increase rigour in data analysis? | No, none specified |
Were the findings of the study grounded in/supported by data? | No. Discussion of findings is limited and does not flow clearly from methods as described. Full range of data is not discussed and there is a lack of clarity concerning where some findings are drawn from. Data tables or quotations from qualitative data are not provided to support findings |
Was there good breadth and depth achieved in the findings? | No. Surface-level data on training satisfaction for cyberleaders discussed, but full range of findings from different data sources and participants not presented. Touches on a broad range of potential implementation issues identified in student data, but these are covered in very limited depth. Qualitative data are not well reported or utilised |
Were the perspectives of young people privileged? | No. Presentation of student responses to surveys is limited and young people’s accounts from interviews are reduced to lists |
Overall reliability of findings based on the above (low, medium or high) | Low. Limited information on methodological rigour is provided. Unclear how bias and error were controlled for in sampling, data collection and analysis. Limited data presented to support findings |
Overall usefulness of findings to this review (low, medium or high) | Low. Some useful insights in relation to factors facilitating and impeding implementation of the intervention, but focus is mainly on cyberleader component and discussion of findings lacks depth |
Intervention name | DARE Plus |
Study report | Bosma 2005190 |
Study location | USA/Minnesota |
Intervention subtype | Student participation in policy decisions (other approaches) |
Study design | Quantitative |
Evaluation examined |
|
Methods of data collection | Data collected included number and types of activities, numbers and names of participants, issues addressed and leadership roles of team members. Seven forms were used to collect quantitative data on (1) adult one-on-ones, (2) youth one-on-ones, (3) presentations at community meetings to recruit support, (4) adult action team meetings, (5) youth action team meetings, (6) adult activities and (7) youth extracurricular activities |
Methods of data analysis | Not stated |
Evaluation participants | Students and community members |
Sample size of participants | Not stated |
Sociodemographic characteristics of participants | Not stated specifically for participants in research activities. School data indicated that the 24 schools participating in the study were similar to Minnesota schools overall, with slightly more non-white students and more students receiving free or reduced-price lunch in the study schools. Student racial/ethnic composition in the 24 study schools was 79.2% white, 11% Asian, 6.3% black, 2.6% Hispanic and 0.9% American Indian. Almost 30% of students received free or reduced-price lunches |
Were steps taken to minimise bias and error/increase rigour in sampling? | No, none described |
Were steps taken to minimise bias and error/increase rigour in data collection? | No, none described |
Were steps taken to minimise bias and error/increase rigour in data analysis? | No. Methods of analysis are not described |
Were the findings of the study grounded in/supported by data? | No. Limited data provided to support findings. Data sources for many of the findings are not clear and limited data examples are used to support findings |
Was there good breadth and depth achieved in the findings? | No. Findings are largely limited to describing levels of participation and frequencies of events/activities implemented. No data on participant views are presented |
Were the perspectives of young people privileged? | No. Data on young people’s perspectives are not included |
Overall reliability of findings based on the above (low, medium or high) | Low. Methods are poorly described and it is unclear from what data many of the findings and conclusions are drawn |
Overall usefulness of findings to this review (low, medium or high) | Low. Findings on factors affecting implementation are limited |
Intervention name | FSFF |
Study report | Cross 2018192 |
Study location | Australia/Perth |
Intervention subtype | Student relationships with teachers, but not student participation in decisions or delivery (other approaches) |
Study design | Quantitative |
Evaluation examined |
|
Methods of data collection | Parent questionnaire covering knowledge about bullying; self-efficacy to talk to children about bullying; parents’ attitudes to bullying behaviour; frequency of parent–child communication about bullying; parents’ perception of their ability to help their children respond to bullying; parents’ perception that bullying is a priority at the child’s school |
Data collected at baseline and at 10 and 22 months | |
Methods of data analysis | Multilevel models used, taking account of clustering |
Evaluation participants | Parents |
Sample size of participants | 3211 parents completed the questionnaire at baseline (1077 grade-2 parents, 1094 grade-4 parents and 1040 grade-6 parents); 2152 parents completed it at post-test 1 (10 months) and 1784 (56%) parents completed it at post-test 2 (22 months); 45% (n = 1444) of the original cohort of parents completed surveys at baseline, at post-test 1 and at post-test 2 |
Sociodemographic characteristics of participants | 83% of respondents were mothers, 14% were fathers and the remainder were step-parents and guardians. A significantly higher proportion of parents in the high-dose (19%) and moderate-dose (17%) intervention groups had a university education than parents in the low-dose (11%) intervention group. 51% of the respondents had male children |
Were steps taken to minimise bias and error/increase rigour in sampling? | No. Self-selected sample |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Appropriate statistical tests used |
Were the findings of the study grounded in/supported by data? | Yes. Detailed data tables are provided to support findings |
Was there good breadth and depth achieved in the findings? | No. Limited data from parent questionnaire examining reach and dose only |
Were the perspectives of young people privileged? | No |
Overall reliability of findings based on the above (low, medium or high) | Medium. Sampling open to self-selection, but otherwise well-conducted study |
Overall usefulness of findings to this review (low, medium or high) | Low. Limited information on factors influencing implementation of parent component |
Intervention name | Gatehouse Project |
Study report | Bond 200166 |
Study location | Australia/Victoria |
Intervention subtype | Student relationships with teachers, but not student participation in decisions or delivery (other approaches) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | Field notes and records documenting meetings, the changes that occurred over time and the professional development provided to the school collected by the ‘critical friends’ from the Centre for Adolescent Health |
Semistructured interviews with ‘key informants’ (curriculum co-ordinators, student welfare, and administration) for each intervention school in the last year of implementation | |
At the end of each year, school background information was obtained on all schools via a structured interview with senior personnel. Audits related to school structures, policies, programmes in place, strategies used to promote emotional well-being of students and demographic information, and could capture whole-school-level changes related to policies and programmes | |
Methods of data analysis | Thematic analysis of field notes, key informant interviews and school background audits. Conceptual tools from literature on capacity-building used as a lens through which to analyse the data |
Evaluation participants | School staff |
Sample size of participants | Not stated |
Sociodemographic characteristics of participants | Not stated |
Were steps taken to minimise bias and error/increase rigour in sampling? | No. Sampling methods for individual participants unclear |
Were steps taken to minimise bias and error/increase rigour in data collection? | No. None stated |
Were steps taken to minimise bias and error/increase rigour in data analysis? | No. None stated |
Were the findings of the study grounded in/supported by data? | No. Quotations used in places, but not all findings are supported with data or linked clearly to data sources |
Was there good breadth and depth achieved in the findings? | Yes. Multiple aspects of and perspectives on implementation explored in good depth using qualitative data |
Were the perspectives of young people privileged? | No |
Overall reliability of findings based on the above (low, medium or high) | Low. Limited information on sampling of schools and participants provided and findings not always supported by data presented. Limited information to assess quality and robustness of methods and analysis provided |
Overall usefulness of findings to this review (low, medium or high) | Medium. Provides useful information to understand how features of the intervention (and to a lesser extent) context may facilitate/ support implementation. Range of data collection tools and multiple perspectives used to explore implementation, but data from young people not included |
Intervention name | GST |
Study report | Knight 201898 |
Study location | Uganda/Luwero district |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Quantitative |
Evaluation examined |
|
Methods of data collection | Cross-sectional survey at end point to measure student and staff exposure to the toolkit |
Data on delivery of intervention to schools were collected routinely by intervention programme officers | |
All interactions with the schools, including technical support visits, group trainings and telephone calls, were systematically documented by each programme officer termly. School-led implementation was measured using termly ‘action plans’ routinely completed by schools | |
Adoption of toolkit elements by schools was tracked by an independent ‘study process monitor’ | |
Head teachers were asked a standard set of questions about toolkit structures in each school termly, a subset of which were verified by direct observation | |
Methods of data analysis | To describe the overall implementation of the toolkit in schools, school-level mean or median values of each measure were calculated, along with standard deviations or interquartile ranges |
To explore whether or not student’s end-point toolkit exposure was associated with attending a school with ‘low’, ‘medium’ or ‘high’ level of toolkit implementation, two unadjusted linear regression models are fitted, accounting for school clustering by fitting school as a random effect. To explore which student characteristics were associated with students’ toolkit exposure, a linear regression model was fitted, adjusting for school clustering | |
Evaluation participants | Students and school staff |
Sample size of participants | 1921 students and 286 staff completed end-point surveys |
Sociodemographic characteristics of participants | Staff surveyed were 87% teaching staff and 13% administrators, cooks and other staff. No further demographic information provided |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. All students and staff invited to participate, with high student and staff response rates reported. All head teachers included in assessment of implementation |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Standardised data collection tools used. All measures pilot-tested before use |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Triangulation (observation and teacher report) used in measurement of adoption. Appropriate statistical models and tests for reliability used |
Were the findings of the study grounded in/supported by data? | Yes. Data described in detail and full data tables provided to support conclusions |
Was there good breadth and depth achieved in the findings? | No. Range of measures, data collection methods and participants included to explore implementation, exposure/reach, but focus is on a limited set of questions on how level of implementation affected reach and outcomes, with some data provided on how reach varied by participants. In-depth perspectives on implementation and factors shaping delivery or receipt are not explored |
Were the perspectives of young people privileged? | No. Student survey data on reach are included and form part of the main analysis, but wider perspectives on implementation are not included |
Overall reliability of findings based on the above (low, medium or high) | High. Standardised data collection tools used and triangulation with direct observation to verify findings on implementation. Survey measures piloted and were subjected to appropriate tests for reliability. Comprehensive data tables are provided to support conclusions |
Overall usefulness of findings to this review (low, medium or high) | Low. Some useful data on how receipt (reach) varies by gender, year group and educational needs, but main focus is on how level of implementation affects reach and outcomes. Does not contribute useful information on how features of interventions, context or providers influence implementation |
Intervention name | HSE |
Study report | Bonell 201069 |
Study location | UK/south-east |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | Pre- and post-intervention survey of year-7 students conducted in private classrooms with support from two fieldworkers to examine reach |
Semistructured interviews with students and staff in intervention schools and intervention providers to examine feasibility, acceptability, awareness and contextual factors. Unstructured observations of various meetings to examine processes of participation and enable triangulation with interview accounts were also undertaken | |
Field notes written during observations and sometimes augmented later the same day from memory | |
Methods of data analysis | Survey data were analysed to provide descriptive statistics on the proportion of students reporting awareness of various policies and actions in intervention and comparison schools. Crude and adjusted ORs were used to assess these differences, overall and among subgroups (gender, baseline attitude to school). All analyses adjusted for clustering except when small samples in some subgroup analysis did not allow, and multivariate adjusted for ethnicity and socioeconomic status, plus gender and baseline attitude to school when not stratifying for these |
Thematic content analysis of qualitative data was carried out by two researchers. Data, codes and memos were compared to identify and resolve disagreement in coding and interpretation | |
Evaluation participants | School staff, students and intervention providers |
Sample size of participants | Across all four schools, 721 (90.4%) took part in the follow-up survey. Semistructured interviews were carried out with one head teacher per intervention school (n = 2), the external facilitator, the two trainers and a subset of action team members. In one school, three senior staff, one junior staff and one student involved in action groups took part (n = 5). In the other school, one senior staff, two junior staff and two students involved in action groups participated (n = 5). Interviews were also carried out with two other staff per school (one experienced and one less experienced), four in total, who participated in the training, as well as with three students in one school and five in the other who participated in other intervention activities, and 17 students in each school not participating in specific actions |
Sociodemographic characteristics of participants | Detailed breakdown not provided Semistructured interviews were undertaken with teaching staff in |
a range of roles. Students in one school were drawn from a range of ethnic groups whereas those in the other school were white, reflecting the composition of each school | |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. Large subset of participants used for qualitative data collection, purposively selected to ensure diversity, capturing a range of roles and levels of involvement with the intervention. Student sample selected to broadly reflect student body |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Questionnaire was piloted with similar-age students in another school and conducted, privately, in classrooms with fieldworkers. Interviews were conducted by researchers in private rooms using standardised interview guides |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Interviews were recorded and transcribed in full. Two researchers coded the data (both inductively and deductively) and compared and contrasted their analysis and interpretation before coding a second time to arrive at a final set of themes. Unstructured observations of meetings were triangulated with interview accounts |
Were the findings of the study grounded in/supported by data? | Yes. Supported with appropriately detailed data tables and illustrative quotations from a range of participants |
Was there good breadth and depth achieved in the findings? | Yes. Range of aspects of implementation explored in detail via multiple participant perspectives and data sources, including in-depth qualitative data |
Were the perspectives of young people privileged? | Yes. Student accounts used to address relevant RQs |
Overall reliability of findings based on the above (low, medium or high) | High. Steps taken to avoid bias and improve rigour in sampling, data collection and analysis. Data, including sample of appropriately illustrative quotations, presented to support findings |
Overall usefulness of findings to this review (low, medium or high) | High. Provides highly useful information drawing largely on in-depth qualitative data from a range of participants to illustrate key factors facilitating and impeding implementation |
Intervention name | Learning Together (Pilot) |
Study report | |
Study location | UK/south-east England |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | To examine fidelity of implementation of action groups, documentary evidence was collected via intervention facilitator checklists, action group meeting minutes and school policies |
To assess delivery of student needs assessment, response rates for the baseline survey were examined to ascertain if they were > or < 80% of all year-8 students at the school | |
To examine reach of staff training and the uptake of restorative practices, evidence was collected from training provider and facilitator checklists | |
To examine the delivery of the student curriculum, evidence from intervention facilitators checklists was collected. Observations of training and action group meetings were carried out | |
Focus groups were carried out with students and school staff and semistructured interviews were undertaken with school leadership at each participating school | |
A subsample of action group members and all intervention providers were interviewed to explore their views on the intervention | |
Action group members at each intervention school were surveyed to examine their views on acceptability | |
Intervention reach was assessed via student survey | |
A teacher survey also included questions on implementation | |
Methods of data analysis | Qualitative data were transcribed. Analysis drew on techniques derived from thematic content analysis and grounded theory and was undertaken by two researchers. Documentary evidence and records of observations were used to support cross-checking and data triangulation |
Descriptive statistics were compiled to analyse checklists, minutes and changes to policy documents, and were triangulated with qualitative data from interviews and focus groups | |
Survey data were adjusted for clustering by school and, when possible, appropriate confounders: the analyses of students’ reports adjusted for sex, ethnicity and housing tenure at baseline, and the analyses of teachers’ responses adjusted for sex, ethnicity and teaching role at baseline. Adjustment for baseline differences in school practices was not possible, as these were not measured | |
Evaluation participants | Students and school staff |
Sample size of participants | Students (n = 1114) and teachers (n = 336) took part in the follow-up survey. 34 action group members plus 16 other students and 4 staff took part in interviews. 112 students took part in focus groups |
Sociodemographic characteristics of participants | 54.2% of students were male and 45.8% were female. The average age was 12.1 years at baseline. The majority of students were White British (44.3%), followed by black and Black British (18.4%), Asian/Asian British (15%), other (12.5%), mixed ethnicity (9%) and Chinese (0.8%). The most common religion was Christianity (53.7%), although the proportion of students reporting being Christian varied markedly between the intervention (34%) and comparison groups (74%) |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. Schools purposively selected for diversity. For qualitative data collection, individuals were selected purposively to represent diversity of students and staff |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Student surveys completed individually in classroom settings monitored by researchers, not teachers. Staff surveys completed anonymously and confidentially. Interviews/focus groups conducted by researchers in private rooms using standardised and pre-piloted interview schedules. Standardised data collection tools (checklists) also used to monitor implementation |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Detailed and layered qualitative analysis methods described, drawing on well-recognised methodological approaches. Data thoroughly triangulated (verified) using range of data sources. Thorough approach to qualitative analysis including double researcher coding/analysis and regular discussion and refinement of coding and interpretation. Log book of decisions was used |
Were the findings of the study grounded in/supported by data? | Yes. Findings flow logically from methods and are supported by detailed data tables, descriptions of participant responses and a number and range of quotations from different participants |
Was there good breadth and depth achieved in the findings? | Yes. Perspectives from a range of participants on a number of different areas of implementation explored using both quantitative and qualitative data, providing good breadth and depth of enquiry |
Were the perspectives of young people privileged? | Yes. Student responses considered in detail and lengthy quotations used to support interpretations |
Overall reliability of findings based on the above (low, medium or high) | High. Well-conducted study with appropriate steps taken to increase rigour in sampling, data collection and analysis. Findings follow from methods and are well supported with detailed descriptions and data |
Overall usefulness of findings to this review (low, medium or high) | High. Range of implementation factors related to delivery and receipt considered using robust methodological approach. Provides high-quality, in-depth information from a range of participants on factors influencing implementation |
Intervention name | Learning Together |
Study report | |
Study location | UK/south-east England |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | In all schools, diaries were completed by trainers, researchers observed training and staff completed a satisfaction survey for training. Semistructured telephone interviews were also carried out with trainers. Diaries were kept by facilitators of action groups and minutes were reviewed. A survey was carried out with action group members each year of the intervention. Researchers carried out observations of action groups; semistructured telephone interviews with action group facilitators in years 1 and 2 of the intervention; and semistructured interviews with action group members (two per year) were also completed. A survey was carried out with staff leading curriculum implementation annually and semistructured interviews were conducted each year with staff delivering the curriculum. In all schools, a survey was also carried out termly with staff implementing restorative practice and interviews were conducted with other school staff in years 1 and 3. In six case study schools, focus groups were conducted with staff each year of the intervention and with students who were directly involved in intervention activities, as well as with those who were less so. Semistructured interviews were also carried out with students involved in restoratives practice |
Methods of data analysis | Quantitative analysis used descriptive statistics to assess intervention fidelity, satisfaction and empowerment. Qualitative analysis was undertaken by two researchers using an initial coding framework based on the RQs and theory guiding the intervention, with these codes augmented, refined and subcategorised informed by an initial wave of coding inductively from the data. Approaches popularised in grounded theory, such as constant comparisons and examination of deviant cases, were used to refine the emerging analysis |
Evaluation participants | Students and staff |
Sample size of participants | Not stated |
Sociodemographic characteristics of participants | Not stated |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. Either all relevant sample included in research activities or participants were purposively sampled for diversity |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Surveys were anonymous, completed independently by students in classrooms with a researcher present and returned in envelopes sent to researchers. Interviews and focus groups were conducted in private rooms with only researcher present |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Multiple data sources used to triangulate data. Analysis of qualitative data carried out by two researchers using standardised coding framework and recognised methods of qualitative analysis drawn from grounded theory, such and constant comparison and examination of deviant cases |
Were the findings of the study grounded in/supported by data? | Yes. Findings are described in detail and follow logically from methods. Full data tables and appropriate, lengthy quotations from a number of participants are provided to support conclusions |
Was there good breadth and depth achieved in the findings? | Yes. Range of data collection methods and participants included across all schools providing very comprehensive picture of implementation |
Were the perspectives of young people privileged? | Yes. Survey data and qualitative data from young people are drawn on and discussed in detail as part of the main findings |
Overall reliability of findings based on the above (low, medium or high) | High. Well-conducted study that includes broad range of measures, methods and diversity of participants, with data collected over a 3-year intervention period, creating a very comprehensive and reliable picture of implementation |
Overall usefulness of findings to this review (low, medium or high) | High. Well-conducted study using range of methods to capture diverse perspectives on breadth of implementation issues |
Intervention name | PPP |
Study report | Mitchell 1991193 |
Study location | USA/Portland |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection |
|
Methods of data analysis | Not stated |
Evaluation participants | Students, parents project and school staff |
Sample size of participants | Not stated |
Sociodemographic characteristics of participants | Sample of research participants not described. The peer helper sample group was approximately 45% male and 55% female. The authors report that diverse ethnic groups were represented in the samples of this study |
Were steps taken to minimise bias and error/increase rigour in sampling? | No. Methods of sampling not described |
Were steps taken to minimise bias and error/increase rigour in data collection? | No. None stated. Data collection methods are poorly described |
Were steps taken to minimise bias and error/increase rigour in data analysis? | No. None stated. Data analysis methods are not described |
Were the findings of the study grounded in/supported by data? | No. Unclear from what data sources findings have been derived. Data are not presented to support findings |
Was there good breadth and depth achieved in the findings? | No. Limited detail/depth to findings on implementation and qualitative data are poorly reported |
Were the perspectives of young people privileged? | No. Interviews were carried out with students, but these are reduced to case studies written by researchers |
Overall reliability of findings based on the above (low, medium or high) | Low. Methods are poorly described, so it is difficult to assess rigour and quality of study. Discussion of findings is limited and sufficient data are not provided to support conclusions |
Overall usefulness of findings to this review (low, medium or high) | Low. Findings on implementation are limited and it is difficult to assess the rigour and quality of the study. Small number of useful data provided on intervention acceptability and features of intervention that impeded implementation of parent component |
Intervention name | Positive Action |
Study report | Beets 200863 |
Study location | USA/Nevada and Hawaii |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Quantitative (cross-sectional survey) |
Evaluation examined |
|
Methods of data collection | Year-end repeat cross-sectional self-administered staff survey carried out in 10 elementary schools at years 2 and 3 of the multiyear evaluation. The survey assessed teacher perceptions of support from their administrators and connectedness with their school; their belief in their responsibility to teach social and character development concepts; their attitude towards Positive Action; the amount of the Positive Action curriculum delivered; and their usage of programme-specific materials in the classroom and school-wide context. School climate was assessed via two series of questions assessing perceived administrative support and school connectedness |
Methods of data analysis | Structural equation modelling, using a conceptual model based on hypothesised relationships among latent constructs |
Evaluation participants | Teachers |
Sample size of participants | In total, 171 teachers in the 10 elementary schools completed year-end process evaluations in year 2 and 191 in year 3 |
Sociodemographic characteristics of participants | For year 2: 41% of participants identified as Japanese, 21% as other Asian decent and 18% white non-Hispanic. For year 3: 36% of participants identified as Japanese, 25% as white non-Hispanic and 12% full or part Hawaiian. Approximately 17% and 10% of the teachers were male in years 2 and 3, respectively |
Were steps taken to minimise bias and error/increase rigour in sampling? | No. All staff invited to participate, but response rates were low in some schools and sample may be subject to self-selection bias |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Surveys anonymised to promote more ‘truthful’ answers (teachers asked only to identify year and grade level taught). Validated and pre-piloted scales used to measure key constructs |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Appropriate statistical analysis and testing used accordingly |
Were the findings of the study grounded in/supported by data? | Yes. Data to support interpretations clearly presented in tables |
Was there good breadth and depth achieved in the findings? | No. Focus is on teacher survey data relating to few key concepts related to implementation |
Were the perspectives of young people privileged? | No. Study does not include student data |
Overall reliability of findings based on the above (low, medium or high) | Medium. Appropriate steps taken to minimise bias data collection and analysis, but not sampling. Data to support findings are presented |
Overall usefulness of findings to this review (low, medium or high) | Low. Very useful information on role of teacher beliefs and attitudes and perception of school climate in shaping implementation of curriculum and whole-school materials, informed by appropriate theory, although analysis is limited to quantitative data from teachers and small range of concepts and variables used |
Intervention name | Positive Action |
Study report | Malloy 2015110 |
Study location | USA/Chicago |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Quantitative |
Evaluation examined |
|
Methods of data collection | Teacher unit implementation reports from first year of implementation (2004–5) administered at six time points |
Teacher work climate survey administered at baseline and after training | |
Methods of data analysis | Analysis involved production of descriptive statistics for demographic characteristics and model variables, bivariate correlations, missing data reports, scale analysis of the three school organisational climate scales and model estimation. Model estimation was conducted using ordinary least squares regression with the vee (cluster variable) command in Stata® (StataCorp LP, College Station, TX, USA) |
Evaluation participants | Teachers |
Sample size of participants | 46 teachers who completed both the unit implementation reports (for ≥ 1 of the units) and the teacher work climate survey were included in the analysis |
Sociodemographic characteristics of participants | 78.7% female; 42.6% self-identified as black, 44.7% as white and 12.8% as Hispanic. The mean number of years of teaching experience was 14.7 (range 1–34 years). 46.8% had a graduate degree |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. All teachers invited to participate, with 73% response rate. Data collected for the teachers who did not take part showed that they were not significantly different from those who did, suggesting that the sample was representative |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Existing predictor variable measures were used and piloted and refined using principle component factors analysis prior to their use. Standardised data collection tools and measures used for teacher-reported implementation |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Although unit implementation reports were self-reported and some of the implementation data were missing, weekly implementation data were triangulated with end-of-term summaries and with student-reported levels of engagement with the programme, which supported the validity of these data. Appropriate statistical tests used |
Were the findings of the study grounded in/supported by data? | Yes. Findings follow logically from methods and full data tables are used to support conclusions |
Was there good breadth and depth achieved in the findings? | No. Limited range of concepts related to implementation explored using staff survey data |
Were the perspectives of young people privileged? | No. Focus on teachers |
Overall reliability of findings based on the above (low, medium or high) | High. Well-conducted study with bias and error in sampling, data collection and analysis accounted for. Sufficient data to support findings are provided |
Overall usefulness of findings to this review (low, medium or high) | Low. Provides useful and reliable data on association between teacher-perceived organisational climate and implementation, but sample and breadth and depth of analysis are limited, with other implementation issues relating to intervention, context or population not explored |
Intervention name | Positive Action |
Study report | O’Hare 2018113 |
Study location | UK/south-east England |
Intervention subtype | Student participation in policy decisions (students sat on a decision-making group with staff) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | Teachers completed an implementation survey at the end of each unit of the programme for lessons and at the end of every week for classroom activities used and whole-school activity to assess dose |
Head teachers completed a school climate survey | |
Students completed a satisfaction questionnaire measuring their engagement and pupil–teacher relationships | |
Qualitative data were collected through classroom observations to assess fidelity (quality) | |
Pupil focus groups and teacher and head teacher interviews were conducted to explore implementation | |
Methods of data analysis | Not stated |
Evaluation participants | Students and staff |
Sample size of participants | 19 teachers completed the implementation survey. |
For survey (engagement and relationships measures), 473 students moving through years 4 and 5 across 15 schools | |
25 students from five schools randomly selected from the 15 took part in focus groups | |
One teacher and one head teacher were interviewed from each of the five randomly selected schools | |
Sociodemographic characteristics of participants | Not stated |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. All students and teachers in the study were invited to participate in survey (although response rates not reported. Unclear if sample was representative). Schools selected at random for qualitative work. Students for focus groups selected at random from five selected schools. Unclear how teachers for interviews were selected |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Survey measures were developed based on existing pre-tested scales. Observation, focus group and interview schedules were piloted in in each of the 15 schools in an earlier phase and refined prior to implementation |
Were steps taken to minimise bias and error/increase rigour in data analysis? | No. Methods of data analysis not described |
Were the findings of the study grounded in/supported by data? | No. Full data tables are provided to support quantitative findings. Description and presentation of qualitative data are limited, however |
Was there good breadth and depth achieved in the findings? | No. Range of methods used to capture information on different aspects of implementation from different perspectives. However, depth of data on participant perspectives limited |
Were the perspectives of young people privileged? | No. Use of student data from focus groups is limited |
Overall reliability of findings based on the above (low, medium or high) | Low. Steps taken to reduce bias and error in sampling and data collection, but methods of analysis not described and description and presentation of qualitative data are poor |
Overall usefulness of findings to this review (low, medium or high) | Low. Although depth of data on implementation is limited, provides some useful data on student engagement and characteristics of interventions that might affect this (and, consequently, implementation and outcomes), as well as on feasibility for schools in terms of curriculum dose and challenges to implementing whole-school elements |
Intervention name | Responsive Classroom |
Study report | Anyon 2016189 |
Study location | USA |
Intervention subtype | Student participation in decision-making (other approaches) |
Study design | Mixed methods |
Evaluation examined |
|
Methods of data collection | Observation by trained research staff of teacher implementation of Responsive Classroom rated on a three-point scale |
Classroom practices frequency survey to capture teachers’ self-reported use of intervention strategies | |
Focus groups carried out with a range of school staff to examine factors that constrain or enable implementation | |
Methods of data analysis | Minimum, maximum and mean calculated for quantitative data. Independent sample t-tests were used on the quantitative data to explore difference by instructor grade level |
Focus group responses were transcribed verbatim. Template analysis was carried out by three members of the research team. Researcher codes were assessed for inter-rater reliability using Cohen’s kappa (κ = 0.82) | |
Mixed-methods analysis was used in which conceptual relationships between t-test results and qualitative themes were identified | |
Evaluation participants | School staff |
Sample size of participants | Sample included 30 school staff. 24 teachers participated in classroom observations at two time points, and 19 teachers completed the classroom practices survey. 15 participants completed a first round of focus groups in autumn 2013, and 19 individuals participated in spring 2014 |
Sociodemographic characteristics of participants | Sample included 24 teachers, three administrators and three social work interns who ‘were representative of all school staff members in terms of grade-level focus and gender, though there was variation by data source’189 |
Of the teachers observed, 54% were female; 60% were elementary school. Of those completing the classroom practices frequency survey, 58% were female and 73% were elementary school staff. Of those completing the first round of focus groups, 46% were female, 60% were elementary school staff. Of those completing the second round of focus groups, 58% were female and 63% were elementary school staff | |
Were steps taken to minimise bias and error/increase rigour in sampling? | Yes. All staff members invited to participate. Potential for self-selection bias, but researchers claim sample was representative of total population |
Were steps taken to minimise bias and error/increase rigour in data collection? | Yes. Different instruments used to triangulate data on implementation. Standardised protocol used for qualitative data collection and previously validated instruments used for quantitative data collection |
Were steps taken to minimise bias and error/increase rigour in data analysis? | Yes. Quantitative and qualitative data triangulated. Qualitative data analysis carried out by three independent coders. Appropriate statistical tests used for quantitative data |
Were the findings of the study grounded in/supported by data? | Yes. Data presented to support quantitative findings. A number of appropriate participant quotations used in text to support qualitative findings |
Was there good breadth and depth achieved in the findings? | Yes. Mixed methods used to capture both breadth (level of implementation across all classrooms) and depth (factors shaping this). Comprehensive data collection on implementation and factors shaping this. Range of teaching staff included to capture different perspectives, but lacks data on student perspectives |
Were the perspectives of young people privileged? | No. No student data reported in study |
Overall reliability of findings based on the above (low, medium or high) | High. Steps taken to minimise bias in all areas |
Overall usefulness of findings to this review (low, medium or high) | Medium. Good-quality, detailed information on implementation factors provided, but no student data reported |
Appendix 6 Characteristics and quality assessment of outcome evaluations
This appendix reproduces material with permission from the Cochrane Handbook for Systematic Reviews of Interventions. 138
Intervention name | AAYP school/community intervention | |
Study reports | ||
Study location | USA/Chicago | |
Intervention subtype | ||
Methods | Design | RCT |
Unit of allocation | School | |
Comparator | Delivered health enhancement curriculum including lessons on life skills with a focus on nutrition, physical activity and general health care, and the importance of cultural pride and communalism | |
Participants | School type | Elementary schools |
School years included | Commencing grade 5 at baseline followed over 3.5 years to end of grade 8 | |
Sample at baseline | Intervention: four schools and 185 students; control: four schools and 184 students (student analytic sample) | |
Sex | Male: 49.5% analytic sample | |
Ethnicity | African American: study schools 91% | |
SES | Received federally subsidised school lunches: approximately 77% analytic sample | |
Outcomes |
|
|
Risk of bias (RCT) | Sequence generation: was the allocation sequence adequately generated? | Yes |
Allocation concealment: was the allocation adequately concealed? | Not stated | |
Blinding: was knowledge of the allocation intervention adequately prevented during the study? | Not stated | |
Complete outcome data: were complete data for each outcome reported, and, if not, were adequate reasons for incomplete outcome data provided? | Yes | |
No selective outcome reporting: were the findings of the study free of selective outcome reporting? | Not stated | |
Key confounders: were differences in non-random studies in key confounders (e.g. SES, sex, age) adequately controlled for? | Not applicable | |
Clustering: was clustering of participants accounted for in the analysis? | Yes | |